Beyond Ragas: A Practical Guide to Evaluating RAG and Agent Pipelines

LLM-as-judge, offline vs online eval, golden datasets, and how to choose an eval stack

Ragas gives you sharp, component-level metrics for a RAG pipeline. But a real system is more than a retriever and a generator scored on a fixed dataset. You have live traffic with no ground truth, agents that take multiple steps, subjective quality that no automated metric captures, and a nagging question about whether your LLM judge can be trusted at all. This article maps the broader evaluation landscape - the techniques, the tools, and how to choose among them - so you can build an eval strategy rather than bolt on a single library.

The one technique underneath everything: LLM-as-judge

Almost every modern eval tool, Ragas included, leans on the same core method: prompt a capable model to score another model's output against criteria you define. It scales where human review cannot and captures nuance that string-matching metrics like BLEU or ROUGE miss. It is also the single biggest source of error in your eval stack, so understand its failure modes before you build on it.

• Judges have biases: they favor longer answers, prefer their own model family's outputs, and are sensitive to answer position in pairwise comparisons.
• Judges are non-deterministic. The same input can score differently across runs, which is why single-run numbers are misleading.
• Judges need validation. Before you trust a judge in production, check its agreement with human labels on a sample. If it does not correlate with humans, its scores are theater.

Give the judge a rubric and ask for chain-of-thought reasoning before the score. A judge forced to explain itself against explicit criteria is measurably more consistent than one asked for a bare number. DeepEval's G-Eval and most platform judges are built on exactly this pattern.

Offline vs online evaluation

The most important distinction in eval strategy is not which tool you use but when you evaluate. The two modes answer different questions and you need both.

Offline eval is your regression gate: a golden dataset scored in CI that blocks changes which degrade quality. Online eval is your smoke alarm: reference-free metrics and user signals computed on sampled production traces, catching the drift and edge cases your golden set never anticipated. Teams that only do offline eval ship regressions the dataset did not cover; teams that only do online eval cannot tell whether a change helped before it hits users.

Golden datasets: the asset that outlives every tool

Whatever framework you pick, your golden dataset - the curated set of inputs with known-good outputs - is the most valuable and most durable thing you build. Tools come and go; a well-maintained golden set is portable across all of them. A few principles:

1. Source from reality. Seed it with real user queries from your logs, especially the ones that went wrong. Synthetic generation (Ragas, or a plain generation script) is fine for volume but must be curated by a human.
2. Cover the distribution, including the tail. Include easy questions, ambiguous ones, adversarial ones, and out-of-scope ones the system should refuse.
3. Version it. Treat the dataset like code - in the repo, reviewed, with a changelog. When scores move, you need to know whether the pipeline or the dataset changed.
4. Keep it small enough to run often but large enough to be stable. A hundred well-chosen cases beat a thousand redundant ones, and they cost less per CI run.

The tool landscape

Below are the tools you will actually run into, grouped by what they are for. There is heavy overlap; you are choosing a combination, not a winner.

RAG-specific scoring libraries

• Ragas - the default open-source RAG metric library. Reference-free and reference-based metrics that cleanly separate retrieval from generation. Best for offline, dataset-based RAG scoring. Covered in depth in the companion article.
• ARES - a research framework that fine-tunes lightweight LLM judges on synthetic data for context relevance, answer faithfulness, and answer relevance. Its distinctive feature is prediction-powered inference: using a small set of human annotations to produce statistical confidence intervals on its scores. More setup than Ragas, but rigorous where you need defensible numbers.
• TruLens - organizes RAG evaluation around the "RAG triad": context relevance, groundedness, and answer relevance. Strong passing scores on all three give you confidence the app is free of hallucination. It instruments your app with feedback functions and is well suited to both experimentation and monitoring.

General LLM testing frameworks

• DeepEval - a Pytest-native testing framework for LLM apps with 30+ metrics, including G-Eval (define any criterion in natural language and it builds an LLM judge with chain-of-thought). You write LLMTestCase objects and assert against metrics, so evals feel like unit tests. Best when you want eval to live in your existing test suite and CI.

Observability and experiment platforms

• LangSmith - tracing, dataset management, and experiments, with built-in LLM-as-judge evaluators (boolean, categorical, continuous) and templates for safety and quality. Natural fit if you already build on LangChain or LangGraph, though it now works beyond that ecosystem.
• Langfuse - open-source and self-hostable, with LLM-as-judge and deterministic code evaluators that run inside the platform on both live observations and dataset experiments, and can gate CI/CD on experiment results. Choose it when you need data control, self-hosting, or a stack outside LangChain.

A useful mental model: the scoring libraries (Ragas, ARES, DeepEval) define and compute metrics; the platforms (LangSmith, Langfuse, TruLens) capture traces, store datasets, run evaluators on production traffic, and give you dashboards. Most mature setups use one of each - for example, Ragas metrics computed on datasets tracked in Langfuse or LangSmith.

Human-in-the-loop review

Automated metrics are a filter, not a verdict. For subjective quality - tone, helpfulness, domain correctness a judge cannot assess - you still need humans, but you use them surgically rather than at scale.

• Have automated metrics triage. Let the judge flag low-scoring or low-confidence traces and route only those to human reviewers, instead of reviewing everything.
• Use human labels to validate your judges. Periodically score a sample by hand and measure agreement with the automated judge; if it drifts, re-tune the judge prompt.
• Capture implicit and explicit user feedback - thumbs up/down, corrections, regenerations - as online eval signal. It is the cheapest ground truth you will ever get.
• Feed reviewed failures back into the golden dataset so every caught bug becomes a permanent regression test.

Common eval pitfalls

These are the mistakes that quietly make an eval suite worthless. Each one produces green dashboards while quality degrades.

• Trusting an unvalidated judge. If you have never checked your judge against human labels, you do not know whether its scores mean anything.
• Overfitting to the eval set. If you tune the pipeline against the same fixed questions long enough, you optimize for the test, not the task. Refresh and expand the dataset.
• Chasing absolute scores. LLM-judged metrics are noisy; a jump from 0.86 to 0.88 is likely noise. Track deltas between versions on a fixed dataset, not vanity numbers.
• Evaluating only end-to-end. A single correctness score cannot tell you whether retrieval or generation failed. Keep component-level metrics so failures are diagnosable.
• Offline-only or online-only. Each misses what the other catches; you need both the regression gate and the production smoke alarm.
• Ignoring cost and latency. Every LLM-judged metric is an extra model call. Sample production traffic rather than scoring 100% of it, and reserve expensive judges for the cases that matter.

Evaluating agents, not just RAG

If your system takes multiple steps - calling tools, planning, looping - single-turn RAG metrics are insufficient. Agent evaluation adds trajectory-level questions that Ragas is not designed to answer: did the agent choose the right tools, in a sensible order, recover from errors, and stop at the right time? You evaluate both the final output and the path taken to reach it. DeepEval, LangSmith, and Langfuse all offer multi-step and tool-use evaluators for this; expect to combine outcome metrics (was the task completed?) with process metrics (was the tool selection and step count reasonable?).

How to choose

There is no single right stack, but the decision reduces to a few questions.

Start smaller than you think. Build a versioned golden dataset from real queries, pick one scoring library and one platform, validate your judge against a handful of human labels, and wire an offline regression gate into CI. Add online eval on sampled production traffic once the offline loop is trustworthy. The tools matter less than the discipline: measure the same thing the same way over time, keep humans in the loop for what machines cannot judge, and treat your dataset as the asset that outlasts every framework on this list.