Your AI Agent Works in a Demo and Fails on Customers
A demo agent fails customers when tools, permissions, and unfinished steps were never specified. A better prompt does not make it production-ready.
By Muhammad Saif and Aashan Khan · Techtribe co-founders
The demo went well. You typed the sentence you had practiced. The agent called the right tool, showed a tidy result, and everyone in the room nodded. The first customer typed a different sentence. The agent called a tool they were not allowed to use, or called the right tool twice, or stopped halfway and declared the task done. The room is gone. The customer is in the ticket queue.
This is an agent-behavior failure. It is not the same problem as a vibe-coded screen that will not save, and it is not fixed by another pass at the wording of the system prompt. The prompt is one file. The failure is the missing product around it: which tools exist, what they may touch, what "done" means, who approves the dangerous step, and how you notice the next miss.
How to trace a run once you have logs is your AI agent works until it doesn't. Read that when you cannot see the tool call. Stay here when you can see it and the call is still the wrong thing to have done.
What customers do that your demo did not
They phrase the goal in the words of their job, not yours. They omit a field you always included. They ask for two things in one message. They paste an email that contains an instruction aimed at the agent, which is a different class of bug, written up in prompt injection. They have permissions. The demo often ran as you, and you can see everything.
They also leave. A demo audience waits while you retry. A customer refreshes, or assumes the side effect happened when it did not. An agent that is "usually right if you try again" is not ready for someone who will not try again.
None of this requires a smarter model as the first purchase. It requires the path to be specified for inputs you did not rehearse.
The failures that show up in the first real week
Wrong tool. The agent had three tools and picked the one that sends, when the one that drafts would have been right. The fix is often fewer tools, with names and arguments that cannot be confused, not a paragraph begging the model to be careful.
Invented facts. The tool returned nothing and the agent filled the gap with a plausible record. A production agent treats an empty tool result as empty. It does not narrate a customer, an order, or a price that the tool did not return. Structured output helps: a schema with required fields, and a validation step that rejects a story which does not match the schema.
Repeated actions. The agent charges, emails, or creates a ticket, does not see a success it trusts, and does it again. Side effects need an idempotency key the tool enforces. The model cannot be the only thing standing between a customer and a duplicate charge. If your tools do not enforce that, the agent is not the layer that should call them unsupervised.
Unfinished work. The agent stops after the first step and says the task is complete, or it keeps going into steps nobody asked for. "Done" has to be a state in your workflow, not a sentence the model is allowed to emit. Deterministic flow control means the application decides the next step from the structured result. The model does a step. The code decides whether there is another.
Permission boundaries. The agent uses a key that can do more than the user who asked. So the user asks, in plain language, for someone else's record, and the tool obeys the key. Bind the tool to the user's rights, or to a service account that cannot see other tenants. The MCP-shaped version of "the tool can do too much" is whether those servers are safe. Do not give the agent a token you would not give a new contractor.
Why another prompt is not the project
A prompt can forbid a tool, demand a format, and list a refusal. You should do that. You should also assume it will be ignored on the input you did not list. Production readiness is what remains true when the prompt is only partly followed. The schema rejects the bad shape. The workflow does not advance. The tool refuses the argument. A person sees the high-impact action before it happens. An eval fails in CI, or at least in a script you run before you ship the new wording.
If your plan is "we'll watch it and tweak the prompt," you are planning a hobby on top of a customer. Tweaks are allowed. They are not the control.
The Production MVP on our packages page names the pieces that belong in a product a stranger will use: deterministic flow control, typed tool contracts, routing with fallbacks, an eval set, and monitoring of latency and quality. That is the fixed shape when the agent is the product you are launching. It runs four to six weeks. The SaaS-shaped version of that commitment is an LLM-powered SaaS MVP. A demo that needs all of those pieces is not a two-week Basic MVP with a sterner prompt. Figures for the scopes are in how much an AI MVP costs.
What to add, in an order that pays off
First, write the steps down as a workflow the code owns. Even three steps. Gather, act, confirm. The model is allowed inside a step. It is not allowed to invent step four.
Second, type the tool arguments and the result. Reject the run that does not match. Log the rejection. You will learn more from ten rejections than from ten prompt adjectives.
Third, put a person on the action that charges, deletes, emails a customer, or changes a record in the system the business actually runs. The person can be a queue, not a meeting. The point is that the agent proposes and a human completes. FlowGen Labs is the engagement we point at for this shape: agents inside long business processes, grounded in the customer's procedures, with a human in the loop on high-impact actions. That work was a dedicated-team engagement over a year, on their repository, not a demo week. We do not treat it as a metric you should expect from a fixed MVP. We treat it as evidence that the approval step is a product decision, not a lack of confidence.
Fourth, write the eval. Twenty tasks if you have them, ten if you are honest about how many you have actually seen. Include the phrasing that made the agent pick the wrong tool. Run the set when the prompt, the model, or the tool changes. A set you do not run is a document.
Fifth, monitor the runs. Cost, latency, tool errors, and the rate at which a human rejects the proposal. The observability article is how you trace one bad run. Monitoring is how you know the rate is rising before the next customer writes in. Both are required. Neither replaces the workflow.
What good looks like on a real account
Take one customer who agreed to be first. Turn the agent on for them only. Read every run for a week, or read a sample you defined in advance if the volume is high. Count wrong tools, invented fields, duplicates, and unfinished tasks. If the counts are boring, widen the flag. If they are not, do not widen it and do not respond by rewriting the prompt in the same afternoon. Add the case to the eval, fix the tool or the step, rerun the set.
A fallback belongs here too. When the model times out or returns nonsense, the workflow takes the defined branch: retry once, switch model, or stop and assign a person. The customer sees a status, not a spinner and a shrug. Fallbacks are in the Production MVP because a demo can be rerun and a customer cannot.
Keep the agent's credentials short-lived and narrow. A long-lived admin key in the agent environment is how a behavior bug becomes a data incident. The same rule you would apply to a new hire applies to the process. If that sentence is new to the team, the agent is not the only thing that is early.
Demo debt you should retire on purpose
Hard-coded examples in the prompt that mirror the pitch. A single tenant in the tool, which is you. Tools that return strings the model is expected to parse with hope. A success message the agent is allowed to emit without checking the tool result. A screen that hides errors so the demo stays calm.
Each of those is a reason the room nodded. Each of them is a bug the moment the input changes. Retire them as tickets, with the eval asserting they stay retired. This is ordinary product work. The fact that a model is in the middle does not make it mystical, and it does not make it optional.
If the thing you have is a generated app whose screens fail, you are in a different article. Start with how to debug a vibe-coded app if you cannot yet name the broken path. An agent that calls tools correctly and writes into a broken schema will look like an agent failure. Check the schema. Then come back to the tool contract.
Who this is for
A team that has already spent money on an agent and cannot hand it to customers. You do not need a new model announcement. You need the workflow, the types, the approval, the eval, and the flag. If that list is the launch, ask for the Production MVP and bring the demo as the brief, not as the product. If that list will keep changing because the operations team changes the process every month, ask for the Dedicated Team instead of pretending the fixed scope includes their calendar.
Bring one transcript of a failure, with the tool trace if you have it. We can tell you whether you are missing the product around the agent or whether you cannot see the agent yet. Those are different contracts. The AI MVP development page is the front door for the fixed build. The packages page is the scope. A custom scope is what we name when the agent is tangled in a system neither package describes. What it costs is a separate note. Book the call with the transcript. A call without a failure is a call about a demo, and the demo is the part that already works.
A failure you can file as a ticket
Write it the way you would write any other bug, or your team will keep treating agent misses as weather. Who was the user. What they asked, pasted in full if it does not contain secrets. Which tool ran, with which arguments, and what it returned. What the agent said had happened. What the system of record shows. What you wanted instead.
That ticket is enough to extend the eval and enough to brief a scope call. A ticket that says "the AI was wrong" is not. If you cannot fill the fields because you have no trace, the first piece of work is the observability article, not a new model. Buy the lights before you buy another prompt workshop.
Review a handful of these tickets every week until the rate is dull. The reviewer is a named person, not a channel nobody owns. They have the authority to keep the flag narrow. A reviewer who can only apologize to customers is not a control. They are a spokesperson for a system that is still a demo.
What production-ready does not mean
It does not mean the agent never errs. It means an error has a defined consequence: a rejected schema, a stopped workflow, a human, a retry that cannot double-charge. It does not mean you have fine-tuned a model. Most of the teams we scope do not need a training run. They need the tools and the steps. It does not mean the demo is thrown away. The demo is the happy path, and the happy path should remain in the eval so you notice if you break it while fixing the sad one.
It also does not mean a research project on agent frameworks. Switching frameworks in the week customers arrive is a rewrite. Stay on the stack that produced the demo unless that stack cannot express a typed tool and a stop condition. If it cannot, the migration is the project, and it should be scoped as a project, not slipped into a prompt change.
When the tickets get boring and the flag is wide, you have what the search query was asking for. Until then, the honest status is a demo with a list. The list is the work. We can take the list as a fixed Production MVP when it matches that package, or as a custom scope when the agent is welded to a system the package does not name. Bring the list. Leave the adjective "production-ready" for the week the list is done.
If the list is longer than a fixed package can hold, do not hide the extra items in "and we will harden it." Name them and move them to a second phase, or book the call for a custom scope that includes them on purpose. Customers can survive a narrow agent that keeps its promises. They do not survive a wide agent whose extra tools were implied. Narrow is a launch strategy. It is also, usually, the only schedule that survives contact with the first real account. Widen from evidence, one tool at a time, with the eval growing in step. That is the whole production plan once the demo has already done its job for a real customer.
If this is your app, send the repo.
A 30-minute scope call. We read what is live, name what is dangerous, and quote a fixed scope.
How Techtribe takes an agent from demo to customers
We scope the steps, the tools, and the actions a person must approve. The Production MVP is the fixed version of that work: typed tools, evals, fallbacks, and monitoring, in four to six weeks. The Dedicated Team is the version that stays with the process as it changes. If you cannot yet see why a run failed, start with observability, not with a new prompt.
The scope call is 30 minutes with a founder. Bring one failed customer transcript and, if you have it, the tool trace. Lovable, Bolt, v0, and Replit are relevant only if the shell around the agent was generated there. The service page is AI MVP development. A SaaS boundary around the agent is an LLM-powered SaaS MVP.
Questions
How do I make an AI agent production ready?
Name the steps, the tools, and the permissions. Require a structured result. Put a deterministic check around anything that charges, sends, or deletes. Add a person for the high-impact action, an eval you can rerun, and monitoring. A better prompt is one input to that, not the whole plan.
Why does the agent work in the demo and fail for customers?
The demo uses your example, your account, and your patience. A customer brings a different phrasing, a missing field, and a permission boundary. If those were never in the spec, the demo was a rehearsal of the happy path only.
Where do I look when a tool call is wrong?
Start with the trace of the goal, the tool it chose, and the arguments. That is an observability problem, covered in the agent debugging article. This article is the list of what the product still lacks once you can see the trace.
When is this a fixed build versus a longer engagement?
A defined agent inside a product, with evals and monitoring, lines up with the Production MVP. An agent that must keep changing with the business is the Dedicated Team. The FlowGen work was the second kind: a year on the customer's repo, with a person in the loop for high-impact actions.