AI observability

AI observability means keeping a record of what an AI system does once real customers use it: each question, the answer, the cost and the waiting time. An AI chatbot can give a wrong answer without any error message, so a check that your website is up will not notice. With these records you can see when answers get worse, costs rise or a step slows down.

Where does the term come from?

Observability is an older idea from software engineering: how well you can understand what happens inside a system from the data it produces. The term was borrowed from control theory (Wikipedia). The OpenTelemetry observability primer names three kinds of signals: metrics (numbers aggregated over time, such as the error rate), logs (timestamped messages) and traces (the path of one request through every service it touches). The primer also notes that a system can be up 100% of the time and still be unreliable, for example when clicking "Add to Cart" for a pair of black shoes does not always add black shoes.

What is different for AI?

A request to a language model can succeed while the answer is wrong: the server returns a normal response with a hallucination in it. The model can also change under you. Chen, Zaharia and Zou compared the March and June 2023 versions of GPT-4: at telling prime numbers from composite ones, its accuracy fell from 84% to 51%. They concluded that the "same" LLM service can change substantially in a short time, and that this calls for continuous monitoring. Costs, finally, follow the number of tokens: Anthropic, for example, prices its models per million input and output tokens, so a longer prompt shows up directly on the bill.

How does it work?

• Tracing. Each request becomes a trace, with a span for each step: retrieving documents, the model call, a tool call. Langfuse describes such a trace as a structured log of the exact prompt, the model's response, token usage, latency and the steps in between.
• Shared field names. OpenTelemetry's conventions for generative AI define standard attributes, such as the requested model and the numbers of input and output tokens. Prompts and answers are opt-in, because they are likely to contain personal data. In September 2026 these conventions still have the status "Development".
• Evaluation. Because a successful response can still be a bad answer, answers are scored as well, often by a second model acting as judge ("LLM-as-a-judge"). Zheng et al. (2023) found that GPT-4 as a judge agreed with human preferences more than 80% of the time, as often as humans agree with each other. Platforms such as LangSmith combine traces with dashboards, alerts and such evaluations on live traffic.

Where guardrails stop a bad answer during a request, observability shows afterwards what happened and how often.

Where does it go wrong?

An automatic judge is a language model too. Zheng et al. found position, verbosity and self-enhancement biases in their judges. Shankar et al. (2024) write that LLM-generated evaluators "inherit all the problems of the LLMs they evaluate" and need checking by people. In their study, users only settled on their criteria while grading outputs, an effect they call criteria drift. Logging also creates a new store of sensitive data: full prompts and answers can contain customer details, which is why the OpenTelemetry conventions leave them out by default.

Example

Suppose a webshop runs a support chatbot that looks up answers in its help centre (RAG). Every question becomes a trace with the search step, the model call and the reply, each with its duration, and the model call with its model version and token counts. Each week a second model grades a sample of the answers against the help articles. When the grades for questions about returns drop, the team opens those traces and sees that the search step keeps returning last year's returns page. The uptime check was green the whole time.