Short answer
LLM-as-a-judge is the practice of using one large language model to grade another model's outputs at scale, through rubric scoring, pairwise comparison, or verdict labels such as pass and fail. It replaces some human review with model review during evaluation.The appeal is throughput and consistency. A judge model can score thousands of conversations in the time a human reviewer covers a handful, and it applies the same rubric to the ten thousandth transcript as to the first. The limits are just as real. Judge models favor fluent, confident answers over correct ones, shift with prompt wording, and inherit the blind spots of the model family doing the grading.
The popular framing this page rejects: treating judge scores as ground truth. An uncalibrated judge is an opinion at scale, not a measurement. Scores only mean something once they are checked against human judgment on a shared sample, then re-checked as prompts, policies, and models change. For customer-facing AI the stakes are concrete. A judge that rewards a polished apology over a correct refund decision will certify an agent that sounds excellent and acts wrong.
Human review vs LLM-as-a-judge at a glance
| Dimension | Human review | LLM-as-a-judge |
|---|---|---|
| Throughput | A sample, slowly | Every conversation, fast |
| Consistency | Varies by reviewer and day | Same rubric every time |
| Blind spots | Fatigue, small samples | Favors fluency, misses policy nuance |
| Best role | Sets the standard | Scales the standard |
Aide, the AI platform for customer service, treats automated grading as one layer of evaluation, never the verdict. Agent behavior is tested against real historical conversations in the Agent Simulator before an intent goes live, and human review anchors what a passing grade means.