Study Guide · Chapter 7: Observability · 7 min read
How Tracing and Evaluation Work Under the Hood
Observability in AI systems combines two instruments: tracing, which records what happened, and evaluation, which judges whether what happened was good. Together they turn quality into a number you can gate releases on.
Anatomy of a trace
Each run produces a tree of spans: the outer span is the whole request; inside it sit spans for retrieval queries and hits, tool executions with arguments and results, model calls with prompt versions and token counts.
Spans carry tags — build version, user segment, feature flags — so you can slice quality by any dimension. OpenTelemetry standards keep this portable across vendors.
The evaluation loop
A golden dataset runs through the pipeline on every build. Each output is scored against rubrics for factuality, policy compliance, and tone; LLM-as-judge handles bulk scoring while sampled human reviews calibrate the judge itself.
Results become trend lines, not just pass/fail: factuality at 0.94 this week versus 0.96 last week is a regression even though nothing 'broke'.
Release gates make quality enforceable
Evaluation runs as a CI stage that blocks deployment when metrics fall below floors — factuality under 92 percent, any policy violation at all, p95 latency over budget. Quality becomes a merge requirement like unit tests.
Cost telemetry rides along in the same gates: token-spend spikes are often the first symptom of a quality problem, such as retrieval suddenly returning bloated context.
Key Points
- Traces are span trees covering every step, tagged for slicing by release or segment.
- Evaluate on every build — trends catch regressions that point-in-time demos hide.
- Calibrate automated judges against human labels before trusting their grades.
- Release gates convert 'we think it's fine' into measurable deployment criteria.
All study guides for this chapter: AI Observability, Explained Simply · How Tracing and Evaluation Work Under the Hood · AI Observability in the Real World