RAG Evaluation with Ragas: Metrics That Actually Tell You If Your Pipeline Works

A practical, current-API guide to measuring retrieval and generation quality in RAG systems

If you cannot measure your RAG pipeline, you cannot improve it. Yet most teams ship retrieval-augmented systems on vibes: someone types a few questions, the answers look plausible, and it goes to production. Then a chunking change quietly tanks retrieval, a prompt tweak introduces hallucinations, and nobody notices until a user complains. RAG evaluation exists to turn those silent regressions into red test runs.

Ragas is the most widely used open-source library for this. It gives you a set of reference-free and reference-based metrics purpose-built for RAG, plus tooling to generate test data and run evaluations at scale. This guide covers why eval matters, the core Ragas metrics, how to build a dataset, and the current API - which has changed enough that older tutorials will actively mislead you. Everything here is written against Ragas 0.4.x.

Why RAG needs its own evaluation approach

A RAG pipeline has two failure surfaces, and generic LLM eval collapses them into one. The retriever can fetch the wrong chunks; the generator can ignore good chunks or invent facts. A single "is the answer correct?" score cannot tell you which half broke. If retrieval is the problem, you tune chunking, embeddings, or reranking. If generation is the problem, you tune the prompt or swap models. You need metrics that isolate each stage.

Ragas is built around this split. Some metrics score retrieval quality (did you fetch the right context?), others score generation quality (did the model stay faithful to that context and actually answer the question?). Most are computed by an LLM acting as a judge, so you need an evaluator model - which itself has cost and reliability implications we will come back to.

The core Ragas metrics

Ragas splits its RAG metrics into retrieval-focused and generation-focused. Here are the ones you will actually use, with what each measures and whether it needs a ground-truth reference.

How to read them together

The metrics are diagnostic in combination, not isolation. High faithfulness but low response relevancy means the model is grounded but not answering the question. Low context recall means no amount of prompt engineering will help - the answer simply is not in the retrieved chunks, so fix retrieval first. Low faithfulness with high context recall means the context was there and the model ignored it, which is a generation problem.

Faithfulness and Response Relevancy are reference-free, so you can run them on production traffic without ground-truth answers. Context Recall and Factual Correctness need a reference, so they belong in an offline golden dataset.

Building an evaluation dataset

Every Ragas sample has up to four fields: user_input (the question), retrieved_contexts (the chunks your retriever returned), response (what your pipeline generated), and reference (the ground-truth answer, for reference-based metrics). You assemble these by running your actual pipeline over a set of questions and capturing what it retrieved and produced.

from ragas import EvaluationDataset
 
# Each row is one question run through YOUR pipeline.
records = []
for question, ground_truth in eval_questions:
    contexts = retriever.get_relevant_documents(question)
    answer = rag_chain.invoke(question)
    records.append({
        "user_input": question,
        "retrieved_contexts": [c.page_content for c in contexts],
        "response": answer,
        "reference": ground_truth,
    })
 
dataset = EvaluationDataset.from_list(records)

Where do the questions and ground-truth answers come from? Three sources, in rough order of value: real user queries from your logs (most representative), hand-written questions from domain experts (highest quality, lowest volume), and synthetic generation. Ragas ships a test set generator that builds diverse question/answer pairs from your own documents, which is the fastest way to get from zero to a few hundred samples.

from ragas.testset import TestsetGenerator
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
 
generator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
generator_emb = LangchainEmbeddingsWrapper(OpenAIEmbeddings())
 
generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_emb)
testset = generator.generate_with_langchain_docs(docs, testset_size=30)
eval_dataset = testset.to_evaluation_dataset()

Synthetic questions are a starting point, not a substitute for real data. They inherit the blind spots of the generator model and will not surface the weird phrasings real users produce. Treat a generated set as scaffolding you curate, not a finished golden set.

Running an evaluation with the current API

This is where old tutorials break. Ragas moved off the Hugging Face Dataset objects and ragas.metrics.* string names that earlier versions used. In 0.4.x you import metric classes, wrap your judge LLM (and embeddings, for metrics that need them), and pass everything to evaluate().

from ragas import EvaluationDataset, evaluate
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.metrics import (
    Faithfulness,
    ResponseRelevancy,
    LLMContextPrecisionWithReference,
    LLMContextRecall,
    FactualCorrectness,
)
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
 
# The judge model and embeddings, wrapped for Ragas.
evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
evaluator_emb = LangchainEmbeddingsWrapper(OpenAIEmbeddings())
 
result = evaluate(
    dataset=dataset,
    metrics=[
        Faithfulness(),
        ResponseRelevancy(),
        LLMContextPrecisionWithReference(),
        LLMContextRecall(),
        FactualCorrectness(),
    ],
    llm=evaluator_llm,
    embeddings=evaluator_emb,
)
 
print(result)          # aggregate scores per metric
df = result.to_pandas()  # per-sample breakdown for drill-down

A few things worth knowing. You pass the judge llm and embeddings once to evaluate() and they flow to every metric; you can also inject a different model per metric by constructing it as Faithfulness(llm=other_llm). ResponseRelevancy needs embeddings - if you forget them it will error. The result object aggregates scores across the dataset, and to_pandas() gives you the per-row scores you need to find the specific questions that failed.

Scoring a single sample

For unit-style checks or debugging one bad answer, score a single sample directly. Metric classes expose an async single_turn_ascore method that takes a SingleTurnSample.

import asyncio
from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import Faithfulness
 
sample = SingleTurnSample(
    user_input="When was the first Super Bowl?",
    response="The first Super Bowl was held on Jan 15, 1967.",
    retrieved_contexts=[
        "The First AFL-NFL World Championship Game was played on "
        "January 15, 1967 at the Los Angeles Memorial Coliseum."
    ],
)
scorer = Faithfulness(llm=evaluator_llm)
score = asyncio.run(scorer.single_turn_ascore(sample))

Ragas is mid-migration to a new collections API (ragas.metrics.collections) with a simpler ascore(...) signature. The class-based metrics shown above are marked for deprecation in 0.4 and removal in 1.0, but they remain the mainstream, best-documented path in 0.4.x and are what most of the ecosystem uses today. Pin your ragas version in requirements and re-check the metric imports when you upgrade toward 1.0.

Interpreting the results

Scores run 0 to 1. Resist the urge to chase 1.0 everywhere - LLM-judged metrics are noisy, and the absolute number matters less than the trend and the outliers. Use the aggregate to track direction over time and the per-sample dataframe to investigate the tail.

• Faithfulness below ~0.8 signals hallucination. Pull the low-scoring rows and read the answer against retrieved_contexts - you will usually see the model asserting things the context does not support.
• Context Recall low but Faithfulness high: retrieval is starving the generator. Improve chunking, embeddings, top-k, or add a reranker before touching the prompt.
• Context Precision low: you are retrieving relevant chunks but burying them below noise. A reranker or tighter top-k usually helps.
• Response Relevancy low with everything else high: the prompt is letting the model waffle. Tighten instructions to answer the question directly.

Never trust a single run in isolation. Because the judge is an LLM, small score wiggles are noise. What you care about is a metric moving materially between two versions of your pipeline on the same fixed dataset.

Wiring Ragas into CI

Eval only prevents regressions if it runs automatically. The pattern is: keep a fixed golden dataset in the repo, run evaluate() in CI, and fail the build if any metric drops below a threshold you have agreed on. Because LLM judges cost money and add latency, most teams run the full set nightly or on pull requests that touch retrieval or prompts, rather than on every commit.

# test_rag_eval.py  -- run under pytest in CI
def test_rag_quality():
    result = evaluate(
        dataset=golden_dataset,
        metrics=[Faithfulness(), ResponseRelevancy(), LLMContextRecall()],
        llm=evaluator_llm,
        embeddings=evaluator_emb,
    )
    scores = result.to_pandas().mean(numeric_only=True)
 
    assert scores["faithfulness"]       >= 0.85
    assert scores["answer_relevancy"]   >= 0.80
    assert scores["context_recall"]     >= 0.80

Set thresholds slightly below your current baseline so normal judge noise does not produce flaky failures, then ratchet them up as the pipeline improves. Log the full result table as a CI artifact so that when a run fails you can immediately see which questions regressed rather than re-running locally. Provide the judge model's API key as a CI secret, and be aware that a 200-sample run against a frontier judge model is a real (if small) recurring cost - budget for it.

Where Ragas stops

Ragas is excellent at scoring a RAG pipeline's retrieval and generation quality on a dataset. It is not an observability platform, it does not evaluate multi-step agents well, and its LLM-judged metrics are only as good as the judge. For online evaluation on live traffic, human review workflows, agent trajectory scoring, and richer experiment tracking, you pair Ragas with other tools - which is the subject of the companion article, "Beyond Ragas." For a standard RAG pipeline where you want fast, principled, component-level metrics, though, Ragas is the right first tool to reach for.