Have a project in mind? Let's talk.
AI agent observability is the ability to see how an agent reached an outcome, not just whether the app returned HTTP 200. A useful view connects model calls, tool invocations, token use, latency, quality scores, safety signals and the user-visible task result. Microsoft describes AI observability as monitoring, understanding and troubleshooting AI systems through evaluation metrics, logs, traces and model outputs. Microsoft Foundry observability
This checklist is for teams moving an agent from a demo to production. It is our synthesis of OpenTelemetry, Microsoft and LangChain guidance, not a claim about a Codesprint-managed production fleet.
Traditional service monitoring starts with a request. Agent monitoring needs a wider unit: one task attempt. That attempt may include planning, retrieval, several model calls, tool calls, approvals, retries and a final user-facing result.
Our baseline is one trace per task attempt. Give it a stable task ID, conversation ID, agent version, environment and release ID. Every model, retrieval, tool, approval and evaluator span should hang below that trace so a reviewer can see the whole path.
OpenTelemetry's GenAI work uses an invoke_agent span with child chat spans for model calls and execute_tool spans for tool invocations. That shape is useful because it separates the agent run from each model and tool step. OpenTelemetry GenAI observability
Dashboards fail when every service names the same fact differently. OpenTelemetry's GenAI semantic conventions cover fields such as agent name and version, conversation ID, provider, requested model, operation name, tool name and call ID, input and output tokens, finish reasons and evaluation scores. OpenTelemetry GenAI attributes
Our minimum trace fields are:
task ID, conversation ID and user or tenant ID
agent name, agent version, prompt or policy version and release ID
provider, requested model and actual response model
operation name, span start time, duration and error status
tool name, tool call ID, authorization principal and side-effect status
input tokens, output tokens, cached tokens and estimated cost
finish reason, retry count and timeout reason
retrieval query ID, document IDs and ranking scores
evaluator name, score, label and explanation where available
Keep IDs consistent across traces, logs and metrics. A token metric without the trace ID can show that cost rose, but it cannot show which task shape caused it.
Model latency and tool latency need separate spans. OpenTelemetry's example metrics include gen_ai.client.operation.duration for model-call latency and gen_ai.client.token.usage for token consumption, with filters for model and token type. OpenTelemetry GenAI observability
Our dashboards break each task into model time, retrieval time, tool time, approval wait and queue wait. This prevents a slow model from hiding a slow CRM write, or a fast model from hiding three failed retries.
Record tool outcome at the provider's postcondition, not only the local HTTP status. A tool can accept a request while the external effect fails, duplicates or arrives late. For side-effecting tools, add an idempotency key and a provider readback marker to the span.
An agent can produce healthy infrastructure metrics while failing the user's task. Microsoft's agent observability guidance includes evaluation for tool-call accuracy and task completion, plus quality, safety and reliability measures. Microsoft Foundry observability
Our production scorecard starts with task completion, groundedness, policy adherence, tool-call accuracy, escalation correctness and user-visible rework. Choose only metrics that change a decision. A score nobody acts on is dashboard decoration.
Do not average away severe failures. Track the pass rate, the worst-case categories and the count of high-impact failures. A 97 percent average can still hide unsafe writes, data leakage or repeated billing mistakes.
LangChain separates evaluation into single-step, trace and multi-turn levels. Single-step evals fit isolated choices such as routing or tool selection. Trace evals score a full execution. Multi-turn evals test whether the agent completed the job across a conversation. LangChain monitoring and observability
Our release gates use all three. Unit-style checks catch known tool-choice mistakes. Trace evals test complete workflows against recorded examples. Multi-turn evals test handoffs, memory and recovery after a user correction.
Use production failures to grow the offline set. When a real trace shows a new failure, sanitize it, add the expected behavior and run it in CI before the next release. This turns incidents into regression coverage.
Microsoft describes post-production monitoring as continuous evaluation of sampled production traffic, scheduled test-set evaluation to detect drift, scheduled red teaming and alerts when quality thresholds fail. Microsoft Foundry observability
Our checklist samples enough traffic to catch drift without turning observability into a second AI bill. Start with all failed or high-risk tasks, all side-effecting runs, all new release traffic and a random sample of normal tasks.
Set alerts on changes that need action: task-completion drop, tool-error spike, cost per completed task, p95 task latency, approval timeout, unsafe-output rate, retrieval failure and unusual repeated loops. Route each alert to an owner with a runbook.
Prompt and tool content can contain secrets, personal data and business records. OpenTelemetry says GenAI telemetry captures no prompt content or tool arguments by default; enabling content capture can include system prompts, messages, tool schemas, tool arguments and tool results. OpenTelemetry GenAI observability
OpenTelemetry also marks tool arguments and results as potentially sensitive. Our default is metadata first: model, tokens, duration, status and IDs. Enable full content only for a bounded environment, a sampled trace or a time-limited investigation, with redaction and access control. OpenTelemetry GenAI attributes
Write a retention rule before launch. Keep aggregate metrics longer than raw prompts. Restrict trace access by role. Never place API keys, card numbers, authentication codes or unredacted customer records in span attributes.
Token use is not the same as value. Our cost view reports cost per completed task, cost per failed task, retry cost, evaluator cost and cost by agent version. The existing AI agent cost control checklist covers budgets and throttles in more detail.
Trace cost regressions to the mechanism: longer prompts, more retrieval, more tool calls, model downgrade loops, evaluator retries or runaway scheduling. This keeps the cost discussion tied to behavior the team can change.
An agent incident review should not require screenshots from five tools. For a failed task, the trace should show the inputs visible to the agent, the model and prompt version, retrieved sources, every tool decision, approval state, side-effect status, evaluator result and final user response.
Our incident triage asks:
Did the agent misunderstand the task or receive stale context?
Did retrieval miss or return the wrong source?
Did a tool call fail, duplicate or hit an authorization boundary?
Did an approval step get skipped, delayed or recorded incorrectly?
Did a model, prompt, policy or dependency release correlate with the failure?
Did the user-visible outcome complete, fail safely or remain unknown?
Security signals belong in the same trace. Pair this checklist with the OWASP Agentic AI Top 10 guide when deciding which policy and egress events to capture.
Before launch, replay known scenarios and prove the dashboard answers them. Include a slow model, a failed tool, a duplicate side effect, an expired credential, a bad retrieval result, a policy block, an approval timeout, a token spike and a multi-turn user correction.
For each scenario, check whether the team can find the trace, identify the failed span, see the release version, measure user impact and decide the next action without asking engineering to add logging. If not, the observability work is not done.
Codesprint helps teams design AgentOps monitoring, evaluation gates and release workflows for production AI agents. Contact us to review an agent before launch.
It is the practice of tracing and measuring an agent's full task attempt, including model calls, retrieval, tools, approvals, quality, safety, cost and the final outcome. Microsoft Foundry observability
No. Logs show events, but agents also need a connected trace across model, retrieval, tool and approval steps plus outcome evaluation. OpenTelemetry's GenAI spans provide one standard way to model that path. OpenTelemetry GenAI observability
Not by default. Prompts, tool arguments and tool results can contain sensitive data. Start with metadata, then enable bounded, redacted content capture for a clear debugging need. OpenTelemetry GenAI attributes
Task completion is the north-star metric. Pair it with cost per completed task, latency, tool-call accuracy, safety failures and user rework so a cheap or fast failure does not look healthy.
Ready to take the first step towards unlocking opportunities, realizing goals, and embracing innovation? We're here and eager to connect.