Skip to content

fix: update rag_eval_ragas notebook for ragas>=0.4.3 API - #298

Open
SyedShahmeerAli12 wants to merge 1 commit into
deepset-ai:mainfrom
SyedShahmeerAli12:fix/ragas-new-api
Open

fix: update rag_eval_ragas notebook for ragas>=0.4.3 API#298
SyedShahmeerAli12 wants to merge 1 commit into
deepset-ai:mainfrom
SyedShahmeerAli12:fix/ragas-new-api

Conversation

@SyedShahmeerAli12

@SyedShahmeerAli12 SyedShahmeerAli12 commented Jun 22, 2026

Copy link
Copy Markdown

Summary

Fixes #285 - updates the rag_eval_ragas cookbook notebook to work with ragas>=0.4.3, which introduced breaking API changes.

What changed:

  • Replaced HaystackLLMWrapper (removed in ragas 0.4.x) with llm_factory and embedding_factory from the native ragas API
  • Each metric now receives its LLM at construction time, e.g. AnswerRelevancy(llm=llm, embeddings=embeddings)
  • Updated imports to use ragas.metrics.collections (non-deprecated path in 0.4.x)
  • Replaced the standalone ragas.evaluate() section with RagasEvaluator.run() per sample
  • Cleared all stale cell outputs

Tested:

Closes deepset-ai#285

- Replace HaystackLLMWrapper with llm_factory/embedding_factory
- Each metric now receives its LLM at construction time
- Import metrics from ragas.metrics.collections (non-deprecated path)
- Replace ragas.evaluate() standalone section with RagasEvaluator.run()
  per sample (ragas.evaluate() does not support SimpleBaseMetric in v0.4.3)
- Clear stale cell outputs
@SyedShahmeerAli12
SyedShahmeerAli12 requested a review from a team as a code owner June 22, 2026 11:27
@review-notebook-app

Copy link
Copy Markdown

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@kacperlukawski kacperlukawski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @SyedShahmeerAli12 I had a look at the suggested changes, and started wondering whether the integration itself shouldn't be extended to reflect the latest changes. The previous approach was just a bit more elegant. Let me have a look at ragas-haystack first and then I'll come back to this PR.

@kacperlukawski kacperlukawski self-assigned this Jul 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

update the RAGAS cookbook to work with ragas-haystack>=4.0.0

2 participants