How can it never hallucinates when retrieval accuracy is at 95% instead of 100% ? #244
-
|
I'm curious on how you did it. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
We grade hallucinations and abstentions separately. So for example: saying "I don't know" is not a hallucination since it doesn't give the user the wrong answer. However, ideally the system would just respond with the correct answer. As a result, for some questions, "I don't know" gets a 0/1, even though it is not a hallucination. Let me know if this makes sense @vikyw89! |
Beta Was this translation helpful? Give feedback.
-
|
what about the |
Beta Was this translation helpful? Give feedback.
We grade hallucinations and abstentions separately. So for example: saying "I don't know" is not a hallucination since it doesn't give the user the wrong answer. However, ideally the system would just respond with the correct answer.
As a result, for some questions, "I don't know" gets a 0/1, even though it is not a hallucination.
Let me know if this makes sense @vikyw89!