Your AI Agent Works Until It Doesn't. How to Debug It

Your AI agent works until it doesn't. A run that used to finish in four steps starts calling the same tool, or a tool starts returning an error the model treats as a reason to try again, or a run sits in "thinking" while the token counter moves and the user sees nothing. The application log says the request returned 200, because the HTTP handler returned 200 while the agent was still going, or because the handler gave up and told the user it had succeeded.

Application debugging still matters. This is the extra layer: the steps inside one agent invocation. Without a trace of those steps you will guess. The guess will be "the model got worse." Sometimes the model did. Often a tool failed, a retry hid it, or the run never had a stop condition. Cost is the same trace read for money, in why an agent burns tokens without finishing.

One run, one trace

OpenTelemetry's GenAI semantic conventions are the shared vocabulary. The agent span documentation describes a hierarchy you can actually debug with. A root invoke_agent span is the run. Under it, one chat span per model call, named with the operation and the model. One execute_tool span per tool call. Retrieval can have its own span, named for the data source. A workflow that calls several agents can sit above them.

Put identifiers on the root that you can search later: the agent name, the agent version, a conversation or session id, and how the run ended. Put the model call's detail on the chat span: provider, requested model, response model, finish reason, input tokens, output tokens, and an error type when the call failed. Cache-read and reasoning token counts belong there too when the provider reports them. Put the tool name, duration, and whether it errored on the tool span. Arguments and results are useful and dangerous. They are useful because the bug is often in an argument. They are dangerous because the argument is often a customer's data or a secret the model decided to pass. Record them only where you would be willing to store a support ticket, with the same access control and a deletion date.

The conventions' note on token counts is easy to get wrong, and getting it wrong will make the next incident's numbers lie. Input tokens should include cached tokens. Cache-read, cache-write, and reasoning counts are subsets of the totals, not extra buckets to add on top. If you sum them with the totals you will double count, and you will chase a cost spike that is an accounting bug. When a provider reports both billed tokens and model-consumed tokens, record the billed count if you want the trace to match the invoice.

Honeycomb's explanation of LLM cost on spans makes a related point. Price the inference spans, the ones that are actually model calls. Do not also price invoke_agent or execute_tool, or you will bill the parent and the child for the same tokens. Their estimate is a catalog price, not your invoice. Negotiated rates, credits, and free tiers will differ. Use it to compare runs. Reconcile money against the provider's bill.

What you are looking for when a run goes wrong

A loop. The same tool name appears over and over under one invoke_agent, or the model calls a tool, gets an error, and calls it again with the same arguments. The user's request was one sentence. The trace is dozens of steps. A cap on turns would have stopped it. The trace tells you the cap was absent or set too high, and it tells you which tool the loop is stuck on. Fixing "the agent" is hopeless. Fixing "search is called with an empty query and returns an error the model retries" is a code change.

A tool that fails quietly. The tool span has an error, and the next chat span shows the model apologizing or inventing a result. The user-facing answer looks confident. The trace shows the tool never succeeded. This is the agent version of a swallowed exception. Alert on tool error rate, not only on HTTP 500s from your API. The API can be healthy while every tool call inside it fails.

A slow step that is not the model. The chat spans are fast and one tool span is most of the wall clock. You do not need a faster model. You need to know why that tool blocked: a lock, a cold start, a query with no limit, a third party. Duration on the tool span is the whole debugging method.

A model you did not mean to call. The request asked for one model and the response span shows another, because a fallback fired, or because a router upgraded the call. Finish reason and model name on the span are how you see it. A fallback that activates on every request is a configuration bug that looks like a quality change.

A run that ends for the wrong reason. Finish reasons such as end_turn, max_tokens, and tool_use mean different things. max_tokens means the answer was cut off. If your handler treats every completion as a finished answer, you will store a truncated tool call or a half sentence and show it to the user. Record the finish reason. Branch on it.

Token growth across steps. Each step resends context. A run that looks stable in duration can still be expensive on the last turns. Input-token counts per chat span, in order, show that curve. If the curve is the problem, the cost note is the next read. If one span dominates, you have a prompt that exploded: a tool result pasted a whole table into the context.

Metrics for the runs you will not open

Traces are for one bad run. Metrics are for noticing that bad runs started. OpenTelemetry's token-metric conventions define counters for input, output, cache, and reasoning tokens as the instruments for consumption over time, and histograms of tokens per operation for percentiles. The histograms are for spotting outliers. They are the wrong tool for a monthly total, because a histogram is a distribution, not a sum you should invoice from.

For agents, the useful charts are the ones that move before the invoice does: how long a run takes, how many model calls it makes, how many tool calls it makes, and how long tools take. A jump in tool calls per run is a loop. A jump in tool duration is a dependency. A jump in model calls with flat tool calls is the model thrashing on its own output. Put those next to error type.

Keep metric labels small: model, provider, agent name, operation, outcome. Do not put a user id or a conversation id on a metric. Those belong on the span. A metric labeled by user becomes a bill of its own.

You do not need a vendor's full AI suite to start. You need the spans exported somewhere you can search by conversation id. A log line with the same fields, one JSON object per span, is an acceptable first version if it is actually queried. A dashboard nobody opens is not observability. Pick one question, "why was this user's run slow or wrong," and make sure the trace can answer it.

What not to log

Prompts, retrieved documents, and tool results are personal data when the user typed personal data, and they are secrets when the agent stuffed a key into an argument. The trace that saves you at midnight is also a second copy of production data, sitting in a system your on-call vendor's staff can often read. Decide this before you turn capture on.

A reasonable default: span names, ids, model, token counts, durations, tool names, finish reasons, and error classes always. Prompt bodies and tool payloads only in a restricted store, off by default in production, on for a single conversation id when you are debugging, deleted on a short timer. If you cannot name the timer, you are not ready to capture bodies.

The same caution applies to "add telemetry" as a prompt to a coding agent. New Relic's 2026 survey found most teams asking models to emit logs and traces inside generated code. That is fine when a person decided the fields. It is a leak when the generated log includes the request body. Read one production log line from the agent path before you trust the prompt that added it.

How to debug one bad run

1. Find the trace by conversation id or request id. If you cannot, that is the first bug. You are debugging from the user's screenshot.
2. Count chat spans and tool spans. If the counts are absurd for the task, you are in a loop or a missing cap. Note the repeated tool and the repeated arguments if you are allowed to see them.
3. Open the first tool span that errored, or the slowest span. Read the error type and the duration. Do not start with the model's last message. The last message is the model explaining a failure it may have caused three steps earlier.
4. Check the model name and the finish reason on each chat span. A truncated call or a surprise fallback changes what the next step received.
5. Compare token counts along the run. A single span with a huge input is a tool result or a history dump, not "the model being wordy."
6. Write down the stop condition that should have fired and did not: max steps, max tokens, a tool error that should abort, a repeated call with the same arguments. The fix is that condition, in code, not a prompt that says "try not to loop."

Then keep the trace. The next report of "the agent is worse" gets compared to this one. Without a saved run you will re-debug from scratch and you will not know whether you fixed it.

What to stop doing

Do not debug an agent from the final answer alone. Do not turn on full prompt capture for every user because one incident was confusing. Do not add parent-span cost to child-span cost and then argue with finance about a number you double counted. Do not alert only on your API's status code. The agent can do a great deal of damage inside a 200.

Do not buy a tool and skip the identifiers. If the agent version is not on the span, you cannot tell whether Tuesday's model change caused Tuesday's loops. Put the version on the root span the day you ship it.

A week of traces, used on purpose

Once spans exist, look at them on a schedule or they will rot. Once a week, open the slowest five runs and the five runs with the most tool calls. Write one sentence about each: loop, slow dependency, huge context, bad finish reason, or expected. If you cannot write the sentence, the span is missing a field. Add the field. Do not add a second dashboard.

Compare the week to the previous week by agent version. A version bump that doubles tool calls per run is a regression even if no user filed a ticket yet. Users file tickets after they have retried. The trace moves first. Keep the version on the span or this comparison is a guess about the calendar.

Sample one successful run as well. Teams only open failures, then they have no picture of what normal is, and every long run looks like an incident. A healthy run's shape, four tool calls, one model, a known finish reason, is the template. Deviations from that template are the page. A template you never wrote down will be reinvented, wrongly, at midnight.

If a vendor retains traces for a few days, export the ones you used in an incident. The next similar failure will want the comparison, and the vendor's retention will have deleted it. A single JSON file in the incident note is enough. A data warehouse can wait until you have more than one agent worth of volume.

How Techtribe makes an agent debuggable

We put a trace on the runs you already have: one span for the run, one per model call, one per tool call, with token counts and errors, and with payloads left out until you have a place to keep them. Then we take one real failed conversation and show that the trace explains it. A dashboard that cannot explain a known failure is not done.

That work sits naturally inside a Production MVP, which includes cost, latency, and quality monitoring, or inside a Dedicated Team if the agent is already in front of customers and changing every week. The agent product itself, if you are building one rather than repairing one, is AI agent development. Bring a conversation id that went wrong and whatever logs you have. If the answer is "we only have the user's screenshot," that is a useful start. It tells us the first hour is the identifier.

The fixed scope for taking a generated app to production is a page per tool: Lovable, Bolt, v0, and Replit.