RAG 評估完整指南——RAGAS、TruLens 與 LLM-as-Judge（2026 版）

文章資訊

作者：Basukori　　日期：2026-06-21

原文標題：RAG Evaluation: The Complete Guide to RAGAS, TruLens & LLM-as-Judge (2026 Edition)

Medium 原連結：https://medium.com/@basukori8463/rag-evaluation-the-complete-guide-to-ragas-trulens-llm-as-judge-2026-edition-068b6e9dc5d0

⭐ 收藏

🎧 摘要語音

🔊 朗讀全文（收藏後自動生成）

---

📝 重點摘要

TL;DR

RAG 評估應視為基礎設施，分層使用 RAGAS、TruLens、LLM-as-Judge 三工具持續監測品質。

核心問題

RAG 系統 demo 表現亮眼，上線後卻悄悄引用錯誤文件而無人察覺，因為沒有任何度量。傳統 ML 評估無法套用於 RAG——它是「檢索＋生成」的管線、沒有單一標準答案、且品質是多維度的（流暢但不忠實、忠實但不相關）。本文教你建立可落地的自動化評估迴圈。

關鍵發現 / 數據

• 評估須拆成兩軸：檢索面（Context Precision/Recall/Relevance）與生成面（Faithfulness、Answer Relevancy、Correctness），才能定位是檢索還生成出錯。
• 範例中答案稱「30 天退款」但 context 寫「14 天」，Faithfulness 立即標出此幻覺——最危險的失敗模式因為看似有依據。
• 真實案例：法律工具用自訂「管轄權正確性」rubric ＋ pairwise 比較，驗證新 reranker 將管轄正確引用提升 22%。
• fintech bot 在每次 PR 以 Faithfulness 0.9 為門檻，兩個月內攔下 3 次 原會靜默上線的 prompt regression。
• LLM Judge 有四種已知偏誤：位置、冗長、自我偏好、不一致——須隨機化順序、懲罰冗長、用不同模型族、temperature=0 多次取平均。

方法亮點

• RAGAS：多數指標免人工標註；附 TestsetGenerator 自動從語料生成 50 題（simple/reasoning/multi_context 分布），補手寫測試集易漏的多跳題。
• TruLens：以 feedback functions 做 trace 級即時觀測，提出 RAG Triad（Context Relevance／Groundedness／Answer Relevance），可跨版本（app_id）比較並對抽樣線上流量告警。
• LLM-as-Judge：自訂 rubric 評分捕捉領域特定標準；Pairwise 比較比絕對分數穩定（同 Chatbot Arena 原理），並須對 50–100 筆人工標註校準（Cohen's kappa）。
• 持續評估管線：離線 RAGAS＋自訂 judge 當 CI 門檻 → 線上 TruLens 觀測漂移 → 低分 trace 回流擴充測試集。

對我的研究有用嗎？

RAG Triad 的「檢索 vs 生成」分解思維可直接移植到 GraphRAG——可拆出「子圖檢索品質」與「基於圖的生成忠實度」分別評估。Pairwise 評估與 judge 偏誤校準方法論，對評比不同圖建構／檢索策略很實用。但本文針對 vanilla RAG，未觸及圖結構特有指標（如多跳路徑正確性、實體連結品質），這些需自行延伸。

評語

實作導向、整合三工具的優秀工程實踐指南，程式碼可直接套用；但屬經驗整理而非研究貢獻，無新 benchmark，案例數據（22%、3 次）僅軼事性、不可驗證。適合工程落地參考，研究深度有限。

---

🌐 中英對照

Author: Basukori
Published:
Source: https://medium.com/@basukori8463/rag-evaluation-the-complete-guide-to-ragas-trulens-llm-as-judge-2026-edition-068b6e9dc5d0
Fetched: 2026-06-21T11:08:31.309609

作者：Basukori
發布日期：
來源：https://medium.com/@basukori8463/rag-evaluation-the-complete-guide-to-ragas-trulens-llm-as-judge-2026-edition-068b6e9dc5d0
擷取時間：2026-06-21T11:08:31.309609

---

RAG Evaluation: The Complete Guide to RAGAS, TruLens & LLM-as-Judge (2026 Edition) / RAG 評估：RAGAS、TruLens 與 LLM-as-Judge 完整指南（2026 版）

Your RAG pipeline looks great in the demo. It answers five questions perfectly, the retrieved chunks look relevant, and everyone in the room nods. Then it ships — and three weeks later, support tickets start piling up because the bot is confidently citing the wrong policy document. Nobody noticed, because nobody was measuring anything. This is the most common failure mode in production RAG, and it has nothing to do with the model.

你的 RAG 管線在示範時看起來很棒。它完美回答了五個問題，檢索到的文字片段 (chunks) 看起來相關，在場所有人都點頭認可。然後它上線了——三週後，客服工單開始堆積，因為這個機器人正信心滿滿地引用了錯誤的政策文件。沒有人發現，因為沒有人在衡量任何東西。這是生產環境 RAG 中最常見的失敗模式，而且和模型一點關係都沒有。

Press enter or click to view image in full size

按下 Enter 或點擊以全尺寸檢視圖片

🧩 The Problem Nobody Wants to Own / 🧩 沒人想承擔的問題

Every RAG tutorial — including the last one — ends with the same checklist item: “evaluate with RAGAS.” It’s mentioned once, in passing, like a footnote. Then everyone moves on to the exciting parts: agentic loops, GraphRAG, reranking.

每一篇 RAG 教學——包括上一篇——都以同樣的待辦事項收尾：「用 RAGAS 來評估」。它只被順帶提及一次，像個註腳。然後大家就轉向那些令人興奮的部分：代理迴圈 (agentic loops)、GraphRAG、重新排序 (reranking)。

Here’s the uncomfortable truth: evaluation isn’t a checklist item. It’s the feedback loop that makes every other optimization possible.

這裡有個令人不安的真相：評估不是一個待辦事項。它是讓其他所有優化成為可能的回饋迴路 (feedback loop)。

Without it, you’re tuning chunk sizes, swapping embedding models, and adding rerankers based on vibes. You ship a change, eyeball five test queries, and call it “better.” Three weeks later, a different set of queries breaks in ways you can’t explain — because you never had a baseline to compare against.

沒有它，你只是憑感覺在調整文字片段大小、替換嵌入模型 (embedding models)、加上重新排序器。你推出一個變更，瞄一眼五個測試查詢，就稱之為「更好了」。三週後，另一組查詢以你無法解釋的方式出錯——因為你從來沒有一個可供比較的基準 (baseline)。

The teams that ship reliable RAG in 2026 treat evaluation the way mature software teams treat testing: as infrastructure, not an afterthought. This guide shows you exactly how to build that infrastructure — using RAGAS, TruLens, and LLM-as-Judge — with working code you can drop into your pipeline today.

在 2026 年能交付可靠 RAG 的團隊，對待評估的方式就如同成熟的軟體團隊對待測試：把它當作基礎設施，而非事後補救。本指南會精確地展示如何建構這套基礎設施——使用 RAGAS、TruLens 與 LLM-as-Judge——並附上你今天就能放進管線裡的可運作程式碼。

🎯 Why RAG Evaluation Is Hard (And Different From ML Evaluation) / 🎯 為何 RAG 評估很難（且與機器學習評估不同）

Traditional ML evaluation is comparatively simple: you have a labeled test set, you compute accuracy or F1, done. RAG breaks this model in three ways.

傳統的機器學習 (ML) 評估相對簡單：你有一個標註好的測試集，計算準確率 (accuracy) 或 F1 分數，就完成了。RAG 在三個方面打破了這個模式。

1. It’s a pipeline, not a model. A RAG system has at least two independently-failing components — the retriever and the generator. A bad answer could mean the retriever returned irrelevant chunks, or the LLM ignored good chunks and hallucinated anyway. If you only measure the final answer, you can’t tell which.

1. 它是一條管線，而非單一模型。 一個 RAG 系統至少有兩個會獨立出錯的元件——檢索器 (retriever) 與生成器 (generator)。一個糟糕的答案可能意味著檢索器回傳了不相關的片段，或者 LLM 忽略了好的片段卻仍然產生幻覺 (hallucinated)。如果你只衡量最終答案，你無法分辨是哪一個出了問題。

2. There’s no single “ground truth.” Two completely different sentences can both be “correct” answers to the same question. Exact-match and BLEU/ROUGE scores — borrowed from machine translation — are notoriously poor proxies for answer quality in open-ended QA.

2. 沒有單一的「標準答案」(ground truth)。 兩個完全不同的句子可以同時是同一個問題的「正確」答案。完全比對 (exact-match) 與 BLEU/ROUGE 分數——這些是從機器翻譯借用來的——在開放式問答 (open-ended QA) 中是出了名地差勁的答案品質替代指標。

3. Quality is multi-dimensional. An answer can be fluent but unfaithful to the source (it sounds right but contradicts the retrieved context). It can be faithful but irrelevant (it accurately summarizes a document that doesn’t answer the question). You need metrics that separate these dimensions, or you’re optimizing the wrong thing.

3. 品質是多維度的。 一個答案可以流暢但對來源不忠實 (unfaithful)（聽起來對，卻與檢索到的脈絡相矛盾）。它也可以忠實但不相關（準確地摘要了一份並未回答問題的文件）。你需要能區分這些維度的指標，否則你就是在優化錯誤的東西。

This is why RAG evaluation frameworks emerged as their own category — and why three of them (RAGAS, TruLens, and LLM-as-Judge approaches) now dominate production pipelines.

這就是為什麼 RAG 評估框架會成為一個獨立的類別——也是為什麼其中三者（RAGAS、TruLens 與 LLM-as-Judge 方法）如今主導著生產環境的管線。

🗺️ The RAG Evaluation Taxonomy / 🗺️ RAG 評估的分類體系

Before touching any tool, it helps to mentally separate evaluation into two axes:

在接觸任何工具之前，先在腦中將評估區分成兩個軸線會很有幫助：

RETRIEVAL                    GENERATION  
                ┌─────────────────┐      ┌─────────────────────┐  
Question  ───▶  │  Did we fetch    │ ───▶ │  Did the LLM use    │ ───▶ Answer  
                │  the RIGHT       │      │  the context        │  
                │  chunks?         │      │  CORRECTLY?         │  
                └─────────────────┘      └─────────────────────┘  
                  Context Precision         Faithfulness  
                  Context Recall            Answer Relevancy  
                  Context Relevance         Answer Correctness

Every metric you’ll encounter across RAGAS, TruLens, and custom judges maps to one of these two buckets — or to the end-to-end answer quality that depends on both being right. Keep this mental model handy; it’s the key to debugging why a RAG system is failing, not just that it’s failing.

你在 RAGAS、TruLens 與自訂評審器 (custom judges) 中遇到的每一個指標，都對應到這兩個桶子之一——或者對應到那個同時取決於兩者都正確的端到端 (end-to-end) 答案品質。把這個心智模型放在手邊；它是除錯的關鍵，能讓你弄清 RAG 系統為何失敗，而不只是知道它失敗了。

📐 Part 1: RAGAS — The Reference Framework / 📐 第一部分：RAGAS——標竿框架

RAGAS (Retrieval Augmented Generation Assessment) is the most widely adopted open-source evaluation framework for RAG, and for good reason: it was purpose-built to decompose RAG quality into the retrieval/generation axes above, and most of its core metrics don’t require human-labeled ground truth answers.

RAGAS（檢索增強生成評估，Retrieval Augmented Generation Assessment）是 RAG 領域最被廣泛採用的開源評估框架，這是有充分理由的：它是專為將 RAG 品質拆解到上述檢索／生成軸線而打造的，而且它大部分的核心指標都不需要人工標註的標準答案。

The Core RAGAS Metrics / 核心 RAGAS 指標

Faithfulness — Does the generated answer only contain claims that can be inferred from the retrieved context? RAGAS breaks the answer into individual statements, then checks each one against the context using an LLM. A low faithfulness score means your model is hallucinating despite having relevant context — often the most damaging failure mode because it looks grounded.

忠實度 (Faithfulness) — 生成的答案是否只包含可以從檢索到的脈絡中推斷出的論述？RAGAS 會把答案拆解成個別的陳述，然後用 LLM 逐一對照脈絡進行檢查。低忠實度分數意味著你的模型即使擁有相關脈絡仍在產生幻覺——這往往是傷害最大的失敗模式，因為它看起來有所本 (grounded)。

Answer Relevancy — Does the answer actually address the question, or does it wander off-topic, pad with caveats, or answer a related-but-different question? RAGAS measures this by generating synthetic questions from the answer and comparing their similarity to the original question.

答案相關性 (Answer Relevancy) — 答案是否真正回應了問題，還是偏離主題、用各種但書 (caveats) 灌水，或是回答了一個相關但不同的問題？RAGAS 透過從答案生成合成問題 (synthetic questions)，並比較它們與原始問題的相似度來衡量這一點。

Context Precision — Of the chunks you retrieved, how many were actually relevant to the question, and were the relevant ones ranked near the top? This is a retrieval-quality metric — it tells you if your reranker is doing its job.

脈絡精確率 (Context Precision) — 在你檢索到的片段中，有多少實際上與問題相關，而且相關的那些是否被排在前面？這是一個檢索品質指標——它告訴你重新排序器是否有在做好它的工作。

Context Recall — Of the information needed to answer the question, how much was actually present in the retrieved context? This one does require a reference answer, since you need to know what “complete” looks like.

脈絡召回率 (Context Recall) — 在回答問題所需的資訊中，有多少實際上存在於檢索到的脈絡裡？這一個確實需要一個參考答案 (reference answer)，因為你需要知道「完整」是什麼樣子。

Installing and Running RAGAS / 安裝與執行 RAGAS

pip install ragas datasets langchain-openai

from ragas import evaluate  
from ragas.metrics import (  
    faithfulness,  
    answer_relevancy,  
    context_precision,  
    context_recall,  
)  
from datasets import Dataset  
# Your evaluation dataset: questions, retrieved contexts,  
# generated answers, and (optionally) ground-truth answers  
eval_data = {  
    "question": [  
        "What is our refund policy for annual subscriptions?",  
        "How long does shipping take to the EU?",  
    ],  
    "answer": [  
        "Annual subscriptions can be refunded within 30 days of purchase, prorated.",  
        "Shipping to the EU takes 5-7 business days.",  
    ],  
    "contexts": [  
        ["Refund Policy: Annual plans are eligible for a full refund "  
         "within 14 days. After 14 days, refunds are prorated based "  
         "on unused months."],  
        ["Standard shipping to EU countries: 5-7 business days. "  
         "Express shipping: 1-2 business days."],  
    ],  
    "ground_truth": [  
        "Annual subscriptions can be refunded in full within 14 days, "  
        "and prorated after that.",  
        "Standard EU shipping takes 5-7 business days.",  
    ],  
}  
dataset = Dataset.from_dict(eval_data)  
result = evaluate(  
    dataset=dataset,  
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],  
)  
print(result)  
df = result.to_pandas()  
print(df[["question", "faithfulness", "answer_relevancy",  
          "context_precision", "context_recall"]])

Run this on the example above, and notice what happens with the first row: the answer says “within 30 days,” but the context says “within 14 days.” Faithfulness will flag this immediately — the model contradicted its own retrieved source. This is exactly the kind of silent failure that ships to production unnoticed without automated evaluation.

在上面的例子上執行這段程式，注意第一列發生了什麼：答案說「30 天內」，但脈絡說「14 天內」。忠實度會立刻標記這一點——模型與它自己檢索到的來源相矛盾了。這正是那種在沒有自動化評估的情況下，會悄無聲息地上線到生產環境的失敗。

Building a RAGAS Evaluation Dataset From Your Pipeline / 從你的管線建立 RAGAS 評估資料集

In practice, you don’t hand-write these dictionaries — you generate them by running your actual pipeline against a query set:

實務上，你不會手寫這些字典——你會藉由用實際的管線去跑一組查詢來生成它們：

def build_eval_dataset(queries, ground_truths, rag_chain):  
    questions, answers, contexts = [], [], []  

for query in queries:  
        result = rag_chain.invoke({"query": query})  
        questions.append(query)  
        answers.append(result["result"])  
        contexts.append([doc.page_content for doc in result["source_documents"]])  
    return Dataset.from_dict({  
        "question": questions,  
        "answer": answers,  
        "contexts": contexts,  
        "ground_truth": ground_truths,  
    })  
# eval_dataset = build_eval_dataset(test_queries, test_ground_truths, qa_chain)  
# result = evaluate(eval_dataset, metrics=[faithfulness, answer_relevancy,  
#                                           context_precision, context_recall])

Synthetic Test Set Generation / 合成測試集生成

The biggest barrier to running RAGAS isn’t the metrics — it’s getting a representative question set with ground truths. RAGAS ships a TestsetGenerator that solves this by generating diverse questions directly from your document corpus, including multi-hop and reasoning questions that are easy to forget when writing test cases by hand:

執行 RAGAS 的最大障礙不是指標——而是取得一組具代表性、且帶有標準答案的問題集。RAGAS 內建了一個 TestsetGenerator，它透過直接從你的文件語料庫 (document corpus) 生成多樣化的問題來解決這個問題，包括那些手寫測試案例時容易遺漏的多跳 (multi-hop) 與推理 (reasoning) 問題：

from ragas.testset import TestsetGenerator  
from ragas.testset.evolutions import simple, reasoning, multi_context  
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

generator = TestsetGenerator.from_langchain(  
    generator_llm=ChatOpenAI(model="gpt-4o"),  
    critic_llm=ChatOpenAI(model="gpt-4o"),  
    embeddings=OpenAIEmbeddings(),  
)testset = generator.generate_with_langchain_docs(  
    documents=raw_docs,  
    test_size=50,  
    distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25},  
)

This gives you 50 questions spanning simple lookups, multi-step reasoning, and questions that require synthesizing multiple chunks — the same failure modes that hand-written test sets routinely miss.

這會給你 50 個問題，涵蓋簡單查找、多步推理，以及需要綜合多個片段的問題——這些正是手寫測試集經常漏掉的失敗模式。

🔍 Part 2: TruLens — Observability for RAG / 🔍 第二部分：TruLens——RAG 的可觀測性

If RAGAS answers “how good is my pipeline on this test set,” TruLens answers a different question: “what’s happening, chunk by chunk and call by call, inside my pipeline — and how is quality trending over time as real traffic flows through it?”

如果說 RAGAS 回答的是「我的管線在這個測試集上有多好」，那麼 TruLens 回答的是一個不同的問題：「在我的管線內部，一個片段接一個片段、一次呼叫接一次呼叫地，正在發生什麼——以及當真實流量流經它時，品質隨時間的趨勢如何？」

TruLens is built around feedback functions — programmable evaluators (often LLM-based) that score traces of your application as they happen, not just on a fixed offline test set.

TruLens 是圍繞回饋函式 (feedback functions) 建構的——這些是可程式化的評估器（通常以 LLM 為基礎），會在你的應用程式的軌跡 (traces) 發生當下為其評分，而不只是在一個固定的離線測試集上。

The RAG Triad / RAG 三元組

TruLens popularized a specific framing called the RAG Triad — three feedback functions that mirror RAGAS’s core metrics but are designed for continuous, trace-level instrumentation:

TruLens 推廣了一個特定的框架，稱為 RAG 三元組 (RAG Triad)——三個回饋函式，它們對應到 RAGAS 的核心指標，但是為持續性的、軌跡層級 (trace-level) 的儀表化 (instrumentation) 而設計：

                    ┌─────────────┐  
                    │   Question  │  
                    └──────┬──────┘  
            Context         │         Answer  
            Relevance       │         Relevance  
                    ▼        │         ▼  
            ┌─────────────┐  │   ┌─────────────┐  
            │   Context    │◀─┴─▶│   Answer    │  
            │  (Retrieved) │     │  (Generated)│  
            └──────┬──────┘      └─────────────┘  
                    │   Groundedness  
                    └─────────────────▶ (Faithfulness)

• Context Relevance: Is the retrieved context relevant to the question?
• Groundedness: Is the answer supported by the retrieved context?

• Answer Relevance: Does the answer address the question?

• 脈絡相關性 (Context Relevance)：檢索到的脈絡與問題相關嗎？

• 有所本程度 (Groundedness)：答案是否有檢索到的脈絡作為支撐？
• 答案相關性 (Answer Relevance)：答案有回應問題嗎？

If all three are high, you have a well-functioning RAG call. If context relevance is low but groundedness is high, your retriever is the bottleneck — the model is faithfully reporting from irrelevant documents. This triangulation is the fastest way to localize a failure to a specific pipeline stage.

如果這三者都很高，你就有一次運作良好的 RAG 呼叫。如果脈絡相關性低但有所本程度高，那麼你的檢索器就是瓶頸——模型正忠實地從不相關的文件中回報內容。這種三角定位 (triangulation) 是把失敗定位到特定管線階段的最快方法。

Setting Up TruLens / 設定 TruLens

pip install trulens-eval chromadb openai

from trulens_eval import Tru, Feedback, TruChain  
from trulens_eval.feedback.provider import OpenAI as TruOpenAI  
import numpy as np  

tru = Tru()  
provider = TruOpenAI()  
# Define the RAG triad feedback functions  
f_context_relevance = Feedback(  
    provider.context_relevance_with_cot_reasons,  
    name="Context Relevance"  
).on_input().on(lambda x: x["source_documents"]).aggregate(np.mean)  
f_groundedness = Feedback(  
    provider.groundedness_measure_with_cot_reasons,  
    name="Groundedness"  
).on(lambda x: x["source_documents"]).on_output()  
f_answer_relevance = Feedback(  
    provider.relevance_with_cot_reasons,  
    name="Answer Relevance"  
).on_input_output()  
# Wrap your existing LangChain RAG chain - no pipeline changes needed  
tru_recorder = TruChain(  
    qa_chain,  
    app_id="support-bot-v1",  
    feedbacks=[f_context_relevance, f_groundedness, f_answer_relevance],  
)  
with tru_recorder as recording:  
    response = qa_chain.invoke({"query": "What is our refund policy?"})  
# Launch the dashboard to explore traces and scores  
tru.run_dashboard()

Why This Matters More Than It Looks / 為何這比表面上看起來更重要

The dashboard TruLens spins up isn’t a vanity metric board — it’s a leaderboard across pipeline versions. Every time you change a chunking strategy, swap an embedding model, or tune your reranker’s top_n, you record a new app_id. TruLens lets you compare groundedness and relevance distributions across versions side by side, so "did this change actually help?" becomes a five-minute lookup instead of a guess.

TruLens 啟動的儀表板不是一塊浮誇的指標看板——它是一個跨管線版本的排行榜 (leaderboard)。每次你改變切塊策略、替換嵌入模型，或調整重新排序器的 top_n 時，你就記錄一個新的 app_id。TruLens 讓你並排比較不同版本間的有所本程度與相關性分佈，於是「這個變更真的有幫助嗎？」就從一次猜測變成一次五分鐘的查詢。

This is also where evaluation graduates from “a notebook I run before a release” to production observability: you can attach the same feedback functions to live traffic (sampled, for cost reasons) and get an early-warning signal when groundedness quietly drops after, say, a vendor changes an embedding model’s behavior underneath you.

這也是評估從「我在發布前跑的一個 notebook」升級為生產可觀測性 (production observability) 的地方：你可以把同樣的回饋函式附加到線上流量上（基於成本考量採抽樣方式），並在有所本程度悄悄下降時得到一個預警訊號——比方說，當某個供應商在你不知情的底層改變了嵌入模型的行為時。

⚖️ Part 3: LLM-as-Judge — Going Beyond Off-the-Shelf Metrics / ⚖️ 第三部分：LLM-as-Judge——超越現成指標

RAGAS and TruLens cover the generic dimensions — faithfulness, relevance, groundedness. But almost every real production system has domain-specific quality criteria that no off-the-shelf metric captures:

RAGAS 與 TruLens 涵蓋了通用的維度——忠實度、相關性、有所本程度。但幾乎每一個真實的生產系統都有領域特定的品質標準 (domain-specific quality criteria)，是任何現成指標都無法捕捉的：

• A legal RAG bot must cite the correct jurisdiction’s version of a law.
• A medical assistant must flag when it’s giving information vs. a recommendation.

• A customer support bot must match your brand’s tone and never promise refunds outside policy.

• 一個法律 RAG 機器人必須引用正確司法管轄區 (jurisdiction) 版本的法律。

• 一個醫療助理必須標明它是在提供資訊還是在給出建議。
• 一個客服機器人必須符合你品牌的語氣，而且絕不承諾政策範圍外的退款。

This is where LLM-as-Judge comes in: you write a custom rubric, and have a strong LLM (typically a different/larger model than your production model, to avoid self-preference bias) score outputs against it.

這就是 LLM-as-Judge 登場之處：你撰寫一份自訂的評分準則 (rubric)，並讓一個強大的 LLM（通常是與你的生產模型不同／更大的模型，以避免自我偏好偏誤 (self-preference bias)）依此為輸出評分。

Pattern 1: Rubric-Based Scoring / 模式一：基於評分準則的評分

JUDGE_PROMPT = """You are an expert evaluator for a customer support AI.  
Score the RESPONSE on the following criteria, each from 1-5:  
1. POLICY_ACCURACY: Does the response accurately reflect the policy  
   in the CONTEXT? (5 = perfectly accurate, 1 = contradicts policy)  
2. TONE: Is the response professional, empathetic, and on-brand?  
3. COMPLETENESS: Does it fully answer the QUESTION, or leave out  
   key details a customer would need?  
QUESTION: {question}  
CONTEXT: {context}  
RESPONSE: {response}  
Return ONLY valid JSON in this exact format:  
{{"policy_accuracy": <int>, "tone": <int>, "completeness": <int>,  
  "reasoning": "<one sentence justification>"}}"""  
def llm_judge_score(question, context, response, judge_llm):  
    prompt = JUDGE_PROMPT.format(question=question, context=context, response=response)  
    result = judge_llm.invoke(prompt)  
    import json  
    return json.loads(result.content)  
# score = llm_judge_score(question, context, answer, judge_llm=ChatOpenAI(model="gpt-4o"))  
# {"policy_accuracy": 2, "tone": 5, "completeness": 4,  
#  "reasoning": "Response states a 30-day refund window, but context specifies 14 days."}

Pattern 2: Pairwise Comparison / 模式二：成對比較

For A/B testing two pipeline configurations (e.g., “with reranker” vs. “without”), absolute scores are noisy — LLM judges are notoriously inconsistent at assigning the same 1–5 score to similar quality answers across runs. Pairwise comparison is far more reliable: show the judge both answers side by side and ask which is better, randomizing the order to cancel out position bias.

對於兩種管線配置的 A/B 測試（例如「有重新排序器」對「沒有」），絕對分數雜訊很大——LLM 評審器出了名地不一致，難以在多次執行間對相似品質的答案給出相同的 1–5 分。成對比較 (Pairwise comparison) 可靠得多：把兩個答案並排展示給評審器，問哪一個更好，並隨機化順序以抵銷位置偏誤 (position bias)。

PAIRWISE_PROMPT = """Question: {question}  
Context: {context}  

Response A: {response_a}  
Response B: {response_b}  
Which response better answers the question using ONLY information  
from the context, with no unsupported claims? Reply with exactly  
"A", "B", or "TIE"."""  
import random  
def pairwise_judge(question, context, response_1, response_2, judge_llm):  
    # Randomize order to cancel position bias  
    if random.random() < 0.5:  
        a, b, swapped = response_1, response_2, False  
    else:  
        a, b, swapped = response_2, response_1, True  
    prompt = PAIRWISE_PROMPT.format(  
        question=question, context=context, response_a=a, response_b=b  
    )  
    verdict = judge_llm.invoke(prompt).content.strip()  
    if verdict == "TIE":  
        return "TIE"  
    if not swapped:  
        return "response_1" if verdict == "A" else "response_2"  
    return "response_2" if verdict == "A" else "response_1"

Run this across your full eval set for two pipeline variants, tally the wins, and you get a much more stable signal than averaging absolute scores — this is the same technique behind LLM leaderboards like Chatbot Arena.

對兩個管線變體在你完整的評估集上跑這段程式，統計勝場數，你就會得到一個比平均絕對分數穩定得多的訊號——這正是 Chatbot Arena 這類 LLM 排行榜背後所用的同一技術。

The Known Biases of LLM-as-Judge (And How to Handle Them) / LLM-as-Judge 的已知偏誤（以及如何處理）

LLM judges are powerful but systematically biased in well-documented ways. Treat these as known error sources, not edge cases:

LLM 評審器很強大，但會以有充分記載的方式產生系統性偏誤。把這些當作已知的錯誤來源，而非邊角案例 (edge cases)：

Bias What Happens Mitigation Position bias Judge favors whichever answer appears first Randomize order, run both orderings and average Verbosity bias Longer answers score higher regardless of quality Explicitly instruct judge to penalize unnecessary length Self-preference bias A model judges its own family’s outputs more favorably Use a different model family as judge Inconsistency Same input, different score across runs Use temperature=0, run multiple samples and average

偏誤	發生了什麼	緩解方法
位置偏誤 (Position bias)	評審器偏好較先出現的那個答案	隨機化順序，兩種順序都跑並取平均
冗長偏誤 (Verbosity bias)	較長的答案不論品質都得到較高分	明確指示評審器懲罰不必要的長度
自我偏好偏誤 (Self-preference bias)	模型對自己家族的輸出評分較寬容	使用不同的模型家族來當評審器
不一致性 (Inconsistency)	相同輸入，跨執行卻有不同分數	使用 temperature=0，跑多個樣本並取平均

Validating Your Judge Against Humans / 對照人類驗證你的評審器

Before trusting an LLM judge at scale, calibrate it against human annotations on a small sample (50–100 examples). Compute agreement (e.g., Cohen’s kappa, or simple correlation for numeric scores). If the judge disagrees with humans more than humans disagree with each other, the rubric needs work — usually it’s too vague, or conflates multiple criteria into one score.

在大規模信任一個 LLM 評審器之前，先在一個小樣本（50–100 個範例）上對照人類標註 (human annotations) 來校準它。計算一致性（例如 Cohen's kappa，或對數值分數採用簡單相關係數）。如果評審器與人類的分歧大於人類彼此之間的分歧，那麼評分準則就需要改進——通常是它太模糊，或把多個標準混為一個分數。

🏗️ Part 4: Putting It Together — A Continuous Evaluation Pipeline / 🏗️ 第四部分：整合起來——持續評估管線

Here’s how the three approaches combine into a single workflow that mirrors how mature teams operate in 2026:

以下是這三種方法如何組合成一個單一工作流程，反映出成熟團隊在 2026 年的運作方式：

┌────────────────────────────────────────────────────────────────┐  
│  OFFLINE (pre-deploy, on every pipeline change)                  │  
│  • RAGAS on a synthetic + curated test set                       │  
│    → faithfulness, answer relevancy, context precision/recall    │  
│  • Custom LLM-as-Judge rubrics for domain-specific criteria       │  
│  • Pairwise comparison: new config vs. current production         │  
│  → Gate: block deploy if faithfulness/groundedness regresses      │  
└────────────────────────────────────────────────────────────────┘  
                              │  
                              ▼  
┌────────────────────────────────────────────────────────────────┐  
│  ONLINE (continuous, on sampled live traffic)                    │  
│  • TruLens RAG Triad feedback functions on sampled requests       │  
│  • Dashboard tracking groundedness/relevance trends over time     │  
│  • Alerting when scores drift below thresholds                    │  
└────────────────────────────────────────────────────────────────┘  
                              │  
                              ▼  
┌────────────────────────────────────────────────────────────────┐  
│  FEEDBACK LOOP                                                    │  
│  • Low-scoring traces → human review queue                        │  
│  • Confirmed failures → added to the offline eval set            │  
│  • Eval set grows over time, catching regressions earlier         │  
└────────────────────────────────────────────────────────────────┘

A minimal CI gate might look like this:

一個最簡的 CI 關卡 (gate) 看起來可能像這樣：

THRESHOLDS = {  
    "faithfulness": 0.85,  
    "answer_relevancy": 0.80,  
    "context_precision": 0.75,  
    "context_recall": 0.75,  
}  
result = evaluate(eval_dataset, metrics=[faithfulness, answer_relevancy,  
                                          context_precision, context_recall])  
scores = result.to_pandas().mean(numeric_only=True)  

failed = {m: scores[m] for m, t in THRESHOLDS.items() if scores[m] < t}  
if failed:  
    raise SystemExit(f"❌ Evaluation gate failed: {failed}")  
print("✅ All evaluation thresholds passed")

Wire this into your CI pipeline, and a chunking change that quietly tanks context recall gets caught before it ever reaches a user — the same way a failing unit test blocks a bad merge.

把這個接進你的 CI 管線，一個悄悄拖垮脈絡召回率的切塊變更，就會在抵達使用者之前被攔下——就如同一個失敗的單元測試會阻擋一次糟糕的合併。

⚖️ RAGAS vs. TruLens vs. LLM-as-Judge: When to Use What / ⚖️ RAGAS vs. TruLens vs. LLM-as-Judge：何時該用哪個

Tool Best For Requires Ground Truth? Typical Use RAGAS Standardized offline benchmarking; comparing pipeline configurations Only for context recall / answer correctness Pre-deploy regression gate, synthetic test set generation TruLens Trace-level observability; tracking quality over time across live traffic No Production dashboards, version comparison, debugging specific failures LLM-as-Judge (custom) Domain-specific rubrics; brand tone, compliance, pairwise A/B testing No (rubric-defined) Catching the failure modes generic metrics can’t see

工具	最適用於	需要標準答案嗎？	典型用途
RAGAS	標準化的離線基準測試 (benchmarking)；比較管線配置	僅脈絡召回率／答案正確性需要	部署前的回歸關卡、合成測試集生成
TruLens	軌跡層級可觀測性；跨線上流量追蹤品質隨時間的變化	否	生產儀表板、版本比較、除錯特定失敗
LLM-as-Judge（自訂）	領域特定評分準則；品牌語氣、合規性、成對 A/B 測試	否（由評分準則定義）	捕捉通用指標看不見的失敗模式

The honest 2026 answer: most mature teams run all three, layered. RAGAS for the standardized regression suite, TruLens for always-on production observability, and custom LLM-as-Judge rubrics for the handful of criteria that are unique to your product. None of these is a replacement for the others — they answer different questions.

2026 年的誠實答案：大多數成熟團隊會分層地把這三者全部跑起來。RAGAS 用於標準化的回歸測試套件，TruLens 用於永遠在線的生產可觀測性，而自訂的 LLM-as-Judge 評分準則則用於那少數幾個你產品獨有的標準。它們沒有一個能取代其他——它們回答的是不同的問題。

🌍 Real-World Patterns / 🌍 真實世界的模式

A fintech support bot ran RAGAS faithfulness on every PR that touched the prompt template or retriever config, gating merges below a 0.9 threshold — catching three prompt regressions in two months that would otherwise have shipped silently.

一個金融科技 (fintech) 客服機器人 在每一個動到提示詞模板或檢索器配置的 PR 上跑 RAGAS 忠實度，攔下低於 0.9 門檻的合併——在兩個月內抓出三次原本會悄悄上線的提示詞回歸 (regressions)。

An internal documentation assistant used TruLens to discover that context relevance scores dropped sharply for any query containing acronyms specific to one business unit — pointing directly at a chunking issue (acronym definitions were being split from their usage) rather than a model problem, something an end-to-end answer-quality score alone would never have isolated.

一個內部文件助理 用 TruLens 發現，任何包含某一個業務單位特有縮寫 (acronyms) 的查詢，其脈絡相關性分數都會急遽下降——這直接指向一個切塊問題（縮寫的定義被從它們的使用情境中切開了），而非模型問題，這是單靠端到端答案品質分數永遠無法定位出來的。

A legal research tool built a custom LLM-as-Judge rubric scoring “jurisdiction accuracy” — a criterion no generic framework covers — and used pairwise comparison to validate that a new reranker model improved jurisdiction-correct citations by 22% before rolling it out.

一個法律研究工具 建立了一份自訂的 LLM-as-Judge 評分準則，為「司法管轄區準確性」評分——這是一個沒有通用框架涵蓋的標準——並使用成對比較，在推出之前驗證了一個新的重新排序器模型把司法管轄區正確的引用提升了 22%。

❌ Common Evaluation Pitfalls / ❌ 常見的評估陷阱

Evaluating only the happy path — Your test set is full of clean, well-formed questions. Real users send typos, multi-part questions, and queries about things not in your corpus at all (where the correct answer is “I don’t know”). Include all of these in your eval set.

只評估順利路徑 (happy path) — 你的測試集裡滿是乾淨、格式良好的問題。真實使用者會送出錯字、多部分問題，以及關於根本不在你語料庫裡的東西的查詢（這時正確的答案是「我不知道」）。把這些全都納入你的評估集。

Treating evaluation scores as absolute truth — A faithfulness score of 0.87 isn’t a law of physics; it’s an LLM’s opinion, with all the biases discussed above. Use scores for relative comparison (is this version better than that one?) and trend tracking, not as a certification of correctness.

把評估分數當成絕對真理 — 0.87 的忠實度分數不是物理定律；它是一個 LLM 的意見，帶有上面討論過的所有偏誤。把分數用於相對比較（這個版本比那個好嗎？）以及趨勢追蹤，而非當成正確性的認證。

One-time evaluation — Running RAGAS once at launch and never again. Embedding models get updated by vendors, document corpora drift, user query patterns shift. Evaluation needs to be continuous, not a one-time gate.

一次性評估 — 在上線時跑一次 RAGAS 就再也不跑了。嵌入模型會被供應商更新，文件語料庫會漂移 (drift)，使用者的查詢模式會改變。評估需要是持續性的，而不是一次性的關卡。

Conflating retrieval and generation failures — If you only measure end-to-end answer quality, a retrieval regression and a generation regression look identical: “the answer got worse.” The RAG Triad / RAGAS decomposition exists specifically so you can tell these apart and fix the right component.

把檢索與生成的失敗混為一談 — 如果你只衡量端到端的答案品質，那麼一次檢索回歸和一次生成回歸看起來是一模一樣的：「答案變差了」。RAG 三元組／RAGAS 的拆解之所以存在，正是為了讓你能區分這兩者並修復正確的元件。

Ignoring cost — LLM-as-Judge evaluation calls add up fast, especially with pairwise comparisons across large test sets. Sample live traffic for online evaluation rather than scoring every request, and reserve full-dataset runs for pre-deploy gates.

忽略成本 — LLM-as-Judge 的評估呼叫累積得很快，尤其是在大型測試集上做成對比較時。對線上評估採用抽樣線上流量，而非為每一個請求評分，並把完整資料集的執行保留給部署前的關卡。

✅ Your RAG Evaluation Checklist / ✅ 你的 RAG 評估檢查清單

• ✅ Decompose metrics into retrieval (precision/recall) and generation (faithfulness/relevancy)
• ✅ Build a synthetic test set with RAGAS’s TestsetGenerator — don’t rely on a handful of hand-written questions
• ✅ Run RAGAS as a pre-deploy gate with defined thresholds
• ✅ Instrument production traffic with TruLens’s RAG Triad for continuous observability
• ✅ Write custom LLM-as-Judge rubrics for domain-specific criteria generic metrics miss
• ✅ Use pairwise comparison, not absolute scores, when A/B testing pipeline changes
• ✅ Calibrate any LLM judge against human annotations before trusting it

• ✅ Feed confirmed failures from production back into your offline eval set

• ✅ 把指標拆解為檢索（精確率／召回率）與生成（忠實度／相關性）

• ✅ 用 RAGAS 的 TestsetGenerator 建立一個合成測試集——別依賴少數幾個手寫問題
• ✅ 把 RAGAS 當成部署前關卡來執行，並設定好門檻
• ✅ 用 TruLens 的 RAG 三元組對生產流量做儀表化，以進行持續可觀測
• ✅ 為通用指標漏掉的領域特定標準撰寫自訂的 LLM-as-Judge 評分準則
• ✅ 在 A/B 測試管線變更時，使用成對比較而非絕對分數
• ✅ 在信任任何 LLM 評審器之前，先對照人類標註校準它
• ✅ 把生產環境中已確認的失敗回饋到你的離線評估集中

🎬 Conclusion / 🎬 結論

If the last guide’s thesis was “retrieval is the bottleneck,” this one’s is the corollary: you can’t improve what you can’t see. Every hybrid search tweak, every reranker swap, every prompt change you make to a RAG pipeline is a hypothesis — and without an evaluation framework, you have no way to test it except gut feel and a handful of manually-checked examples.

如果上一篇指南的論點是「檢索是瓶頸」，那麼這一篇的論點就是它的推論：你無法改善你看不見的東西。你對一條 RAG 管線所做的每一次混合搜尋 (hybrid search) 微調、每一次重新排序器替換、每一次提示詞變更，都是一個假設 (hypothesis)——而沒有評估框架，你除了憑直覺和少數幾個手動檢查過的範例之外，沒有任何辦法去測試它。

RAGAS, TruLens, and LLM-as-Judge aren’t competing tools — they’re three lenses on the same problem, operating at different stages of your pipeline’s lifecycle: pre-deploy regression testing, continuous production observability, and domain-specific quality assurance. Set up even a basic version of all three, and you’ll catch regressions your users would otherwise find for you.

RAGAS、TruLens 與 LLM-as-Judge 不是互相競爭的工具——它們是看待同一個問題的三個鏡頭，運作在你管線生命週期的不同階段：部署前的回歸測試、持續的生產可觀測性，以及領域特定的品質保證。即使只把這三者各自最基本的版本都建立起來，你也能抓出那些原本得讓你的使用者替你發現的回歸。

Building an eval pipeline for your RAG system? I’d love to hear what’s worked (or broken) for you in the comments.

正在為你的 RAG 系統建構評估管線嗎？我很想在留言區聽聽哪些做法對你有效（或失靈了）。

Key Resources:

重要資源：

• RAGAS documentation and TestsetGenerator: github.com/explodinggradients/ragas
• TruLens RAG Triad and dashboard: github.com/truera/trulens
• “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” — Zheng et al. (2023) — the foundational paper on LLM judge biases
• LangChain evaluation modules: python.langchain.com/docs/guides/evaluation

• LlamaIndex evaluation modules: docs.llamaindex.ai

• RAGAS 文件與 TestsetGenerator：github.com/explodinggradients/ragas

• TruLens RAG 三元組與儀表板：github.com/truera/trulens
• 〈Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena〉——Zheng 等人 (2023)——關於 LLM 評審器偏誤的奠基性論文
• LangChain 評估模組：python.langchain.com/docs/guides/evaluation
• LlamaIndex 評估模組：docs.llamaindex.ai

---

🔤 關鍵術語

英文	繁中譯名	文章中的脈絡 / 簡短說明
RAG (Retrieval Augmented Generation)	檢索增強生成	結合檢索器與生成器的問答系統，本文評估的主體
RAGAS	RAGAS（檢索增強生成評估框架）	最廣泛採用的開源 RAG 評估框架，多數核心指標無需人工標註答案
TruLens	TruLens	提供 trace 層級可觀測性的 RAG 評估工具，以 feedback functions 為核心
LLM-as-Judge	以大型語言模型作評審	用較強的 LLM 依自訂評分準則為輸出打分
Faithfulness	忠實度	生成答案的每個論述是否都能由檢索脈絡推得，用以偵測幻覺
Answer Relevancy	答案相關性	答案是否確實回應問題，而非離題或答非所問
Context Precision	脈絡精確率	檢索到的 chunk 中有多少真正相關、且相關者是否排在前面（衡量 reranker）
Context Recall	脈絡召回率	回答所需資訊有多少實際出現在檢索脈絡中（需參考答案）
Groundedness	立基性	答案是否有檢索脈絡支撐，對應 RAG Triad 中的忠實度面向
RAG Triad	RAG 三要素	TruLens 提出的三項 feedback function：脈絡相關性、立基性、答案相關性
Feedback Functions	回饋函式	可程式化（常基於 LLM）的評估器，即時為應用 trace 打分
Hallucination	幻覺	模型在有相關脈絡下仍生成與來源矛盾的內容
Retriever / Generator	檢索器／生成器	RAG pipeline 中兩個可獨立失效的元件
Reranker	重排序器	對檢索結果重新排序以提升相關 chunk 排名的元件
Embedding Model	嵌入模型	將文本轉為向量的模型，更換時可能悄悄影響檢索品質
Synthetic Test Set Generation	合成測試集生成	RAGAS 的 TestsetGenerator 從文件語料自動產生多樣化問題
Multi-hop / Multi-context Question	多跳／多脈絡問題	需綜合多個 chunk 推理的測試問題類型
Pairwise Comparison	成對比較	A/B 測試時讓評審並列比較兩答案，比絕對分數更穩定
Position Bias	位置偏誤	LLM 評審偏好排在前面的答案，須隨機化順序消除
Self-preference Bias	自我偏好偏誤	模型對同族模型的輸出評分偏高，須用不同模型家族當評審
Cohen's Kappa	Cohen's Kappa（一致性係數）	校準 LLM 評審與人工標註一致程度的指標
Chatbot Arena	Chatbot Arena	採用成對比較技術的 LLM 排行榜，本文評審偏誤論文來源