How many Rs are in strawberry?
Sounds like a simple question that should be easy for any LLM to answer.
It is not, though. It's a known challenge for LLMs to answer this question.
So we decided to test how well different LLMs perform on this task.
And we had found a lot of interesting insights doing so.
Check out the results and the testbed.