Name: PRO Session (DataWeek): Open-Rag-Eval: Fast, Automated Evaluation for Reliable Rag Systems
Start: 2025-09-04T14:00:00-0700
End: 2025-09-04T14:25:00-0700

Thursday September 4, 2025 2:00pm - 2:25pm PDT

DataWeek -- Main Stage

Ofer Mendelevitch, Vectara, Leads Developer Relations

Manual evaluation often creates a bottleneck in developing effective RAG systems. Enter `open-rag-eval`—an open-source framework that revolutionizes RAG assessment by harnessing the power of LLM judges for scalable, automated evaluation. Building on pioneering research from the University of Waterloo, this framework integrates innovative tools like UMBRELA for reference-free relevance scoring and AutoNuggetizer for automated fact-checking. Designed with a flexible connectors architecture, it seamlessly plugs into any RAG pipeline while delivering fast, transparent, and interpretable metrics on retrieval, generation, and hallucination in RAG. Join our technical deep dive and live demo to discover how `open-rag-eval` can drastically reduce manual overhead, enhance system reliability, and accelerate your RAG development efforts.

Speakers

Ofer Mendelevitch

Leads Developer Relations, Vectara

Ofer Mendelevitch leads developer relations at Vectara. He has extensive hands-on experience in machine learning, data science and big data systems across multiple industries, and has focused on developing products using large language models since 2019. Prior to Vectara he built... Read More →

Thursday September 4, 2025 2:00pm - 2:25pm PDT
DataWeek -- Main Stage

DataWeek

Talk Type PRO Session
Tracks DataWeek, AI Models and Management (DataWeek), Data Science and Machine Learning (DataWeek)
In-Person/Virtual In Person

API World + CloudX + DataWeek 2025

Ofer Mendelevitch

Get help with the event

API World + CloudX + DataWeek 2025

Ofer Mendelevitch

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Get help with the event