An answer is the surface.
Observe the system beneath it.

Observe the product that surrounds the model.

An AI feature is an application workflow. It can retrieve records, assemble context, call a model, run a tool, check a permission, retry after an error, and format a result. A user sees one answer; the team operating the system needs evidence about the path behind it.

AI observability is the practice of instrumenting that path and using its signals to understand reliability, quality, latency, and risk in real use. It does not make an answer correct by itself. It helps the team ask a better question: which step shaped this outcome, and what evidence would confirm the fix?

That distinction matters most for agents. One request can contain several model calls and tool operations, so a success message alone cannot show whether the task was completed safely. See how an agent moves from answer to action ↗.

Do not ask one signal to explain everything.

OpenTelemetry describes traces as the path of a request, metrics as measurements captured at runtime, and logs as event records. Use those signals together: a metric can reveal a rise in tool failures, and a trace can show what happened in one affected request. OpenTelemetry’s signal overview ↗ explains the distinction.

Make each important step part of the same trace.

A trace represents a request or workflow; spans represent operations within it. Parent and child relationships preserve which work happened inside which step. For an AI feature, useful spans may cover retrieval, model generation, tool execution, a handoff, a policy check, and the final response.

1. Receive the request. Give the workflow a correlation ID and record the task or route category.
2. Collect context. Record which retrieval path or source set was used, without copying private documents into telemetry by default.
3. Call the model. Record the model and workflow version, duration, status, and available usage data.
4. Run tools and checks. Record which tool ran, whether permission was granted, and whether the operation succeeded or failed.
5. Confirm the outcome. Capture the application’s observable result, such as a validated state change or a clear reason the task stopped.

OpenAI’s Agents SDK is one concrete example: its tracing can record model generations, tool calls, handoffs, guardrails, and custom events. The exact spans depend on the framework and instrumentation, so teams should check what their own system actually emits. See the SDK tracing guide ↗.

Track outcomes alongside service health.

Segment these views by workflow version, task type, model, or tool so a new failure pattern does not disappear inside an overall average. Treat an HTTP 200 response, a low latency number, or a large token budget as operational facts—not proof of usefulness or correctness.

For higher-stakes or subjective tasks, review a representative sample of traces against human-readable criteria. A dashboard can find a change; people and repeatable evaluations help determine whether it matters.

Use production evidence to improve the next release.

When a failure appears, inspect its trace to locate the step that needs attention. A missing source may call for better retrieval; a mistaken tool choice may call for a clearer routing rule; a valid action rejected by a grader may reveal a faulty criterion. The trace narrows the investigation, but the final diagnosis still needs evidence.

Once a failure is understood, remove private details, turn the task into a reproducible example, and add it to an evaluation set. Re-run that set after a change. OpenAI’s guide to evaluating agent workflows ↗ describes using traces to discover workflow problems, then moving to repeatable datasets and grading.

A useful operating loop is simple: observe a real workflow, identify a specific failure, build a repeatable check, change the system, and compare results. Continue with the guide to evaluating AI agents ↗.

A trace can expose more than a system intended to remember.

Prompts, model outputs, retrieval results, and tool arguments can contain personal information, credentials, customer records, or confidential work. Some SDK tracing options capture generation and function inputs and outputs; useful debugging data can therefore become a sensitive data store.

Decide what to collect before enabling content capture. Prefer metadata that answers an operational question—workflow version, step name, status, duration, permission result, and usage—over full conversation text. Restrict access, define retention, review instrumentation, and use filtering or redaction where needed. OpenTelemetry’s sensitive-data guidance ↗ recommends collecting only what serves an observability purpose and reviewing the data emitted by instrumentation.

Start with one workflow and five clear decisions.

1. Name the job. Define what a successful result means for one real task.
2. Map its steps. Trace the request through retrieval, model calls, tools, validation, and the final state.
3. Select useful signals. Track step status and duration, end-to-end latency, retries, usage, permission decisions, and task outcome.
4. Set data boundaries. Decide which content is omitted, masked, or available only to authorized reviewers; choose retention before launch.
5. Close the loop. Review failures, create a reproducible evaluation, and compare the result after a change.

Then ask what remains invisible: a service that was not instrumented, an outcome the application never verifies, or a group whose failures are hidden by aggregate metrics. Observability improves when it reduces that uncertainty, not when it simply stores more data.

Connect operations to evidence and control.

Learn how to test agent workflows before release ↗, evaluate retrieval and generated answers ↗, and keep permissions at the tool boundary ↗. For the wider vocabulary, explore the AI field guide ↗.