Glossary
LLM-as-judge evaluation
Using a language model to score another model's output against a rubric, such as faithfulness to sources or instruction following, so quality can be measured on every trace.
LLM-as-judge evaluation means asking a language model to grade another model's output against a defined rubric. Instead of a person reading each answer, a judge model scores it: is it faithful to the retrieved sources, does it follow the instructions, is it relevant, is it toxic.
Why teams use it
Quality problems in agents rarely show up in latency, cost or error rates. A grounded answer and an ungrounded one cost the same and take the same time. Scoring the content is the only way to see the difference at scale, and a judge model makes it cheap enough to run on every trace rather than a sample.
Where it falls short
A generic rubric checks generic quality. It won't know that your refund window is 14 days, or that a figure has to match a specific query result. That's why evaluation works best alongside checks against your own rules and records, which is how Zespan approaches functional failures.
Zespan ships 12 built-in evaluation templates, including faithfulness, groundedness, relevance and instruction following, and runs them on live traffic. Evaluations are part of the Pro plan and above; see evaluations.