AI-Generated Code Passes Review and Breaks in Production

AI-generated code can look finished in review and still be the change that pages someone after deploy. The diff is tidy. Names are consistent. The happy path works on the author's machine. Production adds a second caller, a retried webhook, a schema the other service did not change, and a permission the review never executed. The code does not throw. It does the wrong thing, or it does the right thing for one user and the wrong thing for the next.

New Relic's 2026 State of AI Coding report is the clearest published picture of that gap, and it is worth quoting with its limits attached. The local bugs that show up in a single generated app, a double charge or a broken booking, are in when an AI-generated app breaks in production. This note is the pattern across teams that ship agent-written code on purpose: review says the code is better, and the incident chart says it is not.

What the survey actually measured

New Relic commissioned Hanover Research to survey 200 U.S.-based technology decision-makers, full time, manager level or above, at organizations using generative AI for software development. Respondents had software purchase authority. Figures are percentages of that group of 200. They are not a census of startups, and they are not a count of Lovable or Replit apps. They are what senior people at companies already using these tools said was happening.

The press release and the report PDF agree on the central numbers:

• 94% rated AI-generated code higher quality than human-authored code at review time. The press release splits that into 61% "somewhat higher" and 33% "much higher," with 2% rating it lower.
• 78% reported more production incidents once that code shipped.
• 86% reported more time senior engineers spent fixing code. The PDF calls this senior-engineer firefighting over the past 12 months.
• 74% said at least 25% of AI-generated code needed significant rework. The PDF adds that 41% said at least half of it needed significant rework, for poor context, incomplete data, or wrong assumptions.
• 82% had at least one production failure tied to AI-generated code in the past six months. 19% reported no AI-generated-code challenges in that window. The other 82% each cited more than two production issues on average.
• 62% said their teams often ship AI-generated code to production without line-by-line verification.
• 67% said AI now generates or significantly refactors between 51% and 75% of weekly code output.
• 88% have written vibe coding into a formal production policy. 5% restrict it to non-production. None of the respondents banned it.
• 96% rated observability very or extremely important when working with AI-generated code. None rated it slightly or not important. 78% said teams routinely prompt tools to include logs, traces, and metrics in the generated code.

The PDF also says AI-generated code introduces roughly 1.7 times more critical runtime issues than the baseline the authors associate with peer-reviewed human-written code, and that the four common failure modes in a six-month window were integration failures (31%), compliance or governance issues (30%), data-integrity problems (29%), and new security vulnerabilities (28%).

Read the 94% and the 78% together. The review opinion and the production result moved in opposite directions. Shipping faster into that gap is how the senior-engineer time went up at 86% of the same organizations.

Why review is a weak instrument here

Review is good at seeing structure. AI-generated code is often structurally neat. It names things consistently, it handles the example from the prompt, and it does not look like a tired patch. A reviewer who is also moving faster, because the team is measuring throughput, will pass it. The press release's phrase for the 62% figure is the useful one: trust early in the lifecycle, before anyone has watched the code meet a system it did not write.

The failures the report names are exactly the ones a visual review misses.

Integration. Schema drift, contract violations, error rates between services. The agent updated one side of a call. The other side still sends the old field. Unit tests on the side that changed still pass, because they construct the input the new code expects. The other service does not.

Compliance. A log line that includes a customer identifier, a retention period the agent did not know, a region the new vendor call ignores. The code works. The audit does not. That class of bug is invisible in a feature demo.

Data integrity. Duplicates, drift, unexpected nulls. A retried request inserts a second row. A default fills a column that should have stayed empty. Nothing throws, so the log stays quiet while the table gets worse. The report says this shows up as anomalies in telemetry, which is why the 96% observability figure is not a tooling advertisement. If you cannot see duplication, you will hear about it from a customer.

Security. Auth that checks the happy path, a token in a place the client can read, a query that trusts an id from the browser. The review saw a function called authorize. Production saw that it was not called on one route. Those holes, in generated apps specifically, are in vibe coding security vulnerabilities.

The PDF's line that the code is not breaking in one signature way is the operational point. You cannot write one regression test called "AI bug" and go home. You need the checks that match the four shapes: contract, policy, data, and access.

Rework is the hidden schedule

74% of the surveyed leaders said at least a quarter of AI-generated code needed significant rework. That rework lands on senior people, which is the 86% figure. The team did not only pay for the incident. It paid for the engineer who had to understand a change they did not write, in a codebase the agent has been editing faster than anyone can hold in their head.

This is the same debt described in AI-generated code technical debt, seen from the incident side instead of the static-analysis side. A tangled repo makes the rework slower. The rework is why the velocity number from the coding tool and the velocity number from the incident channel disagree. Both can be true in the same month: more changes merged, more hours spent undoing them.

41% saying at least half the AI-written code needs significant rework is a planning number. If you believe your own version of that estimate, half the agent's output is not a saving. It is a draft. Treat it as a draft. A draft needs an owner, a test the draft's author does not control, and a path that has been executed, not narrated.

What "shipped without line-by-line review" costs

62% is not a moral failing in the survey. It is a description of how teams use the tools. Line-by-line review of everything does not scale to a world where most of the weekly diff is machine-written, which is the 51% to 75% band two-thirds of respondents reported. The answer is not to pretend you will read every line. The answer is to decide which lines you will always read, and which behaviors you will always execute.

Always read: authentication, authorization, migrations, payment and webhook handlers, anything that deletes or sends data, CI and agent configuration, and new dependencies. Always execute: the action as two users, the side effect twice, the failure path of the external call. Everything else can be sampled. Those cannot.

A policy that says vibe coding is allowed in production, which 88% of the surveyed organizations have, is not by itself a control. A policy that says it is allowed and that these paths have an owner and a test is a control. The 5% who keep it out of production have chosen a different trade. Neither group is served by a badge in the pull request that says an agent wrote the code and stops there.

What to check before the next deploy

1. Name the person who owns the change. OWASP's position, and common sense, is that the model does not accept the incident. The person who merges it does.
2. Run the integration the change claims to speak to, against the other service's current contract, not against a mock the agent updated in the same pull request. If the mock changed, ask why.
3. Send the side effect twice. One row, one charge, one email. If the second request creates a second effect, you have the data-integrity shape from the report.
4. Open the path as a user who should not have access. A 200 is a finding. Do this even when the diff looks like a UI change. Agents touch the call site while they are "in the file."
5. Read the logs the new code emits. If a customer id, a token, or a full payload is in there, fix that before you celebrate the 78% of teams who asked the agent to add telemetry. Telemetry that leaks is a compliance incident with better dashboards.
6. Check whether tests were deleted, weakened, or rewritten to match the new behavior. That failure has its own note: testing an AI-built app.
7. Know how you will undo the deploy without also undoing the data. A code rollback does not put a dropped column back. Migrations are in whether you can trust an AI-generated database migration.

If you cannot do item 2 because there is no other environment, you are using production as the integration test. That is allowed only if you are willing to have the incident the survey is counting.

What to stop doing

Do not take "the code looks clean" as evidence it will survive a retry. Do not let the same agent write the change, the test, and the mock, and then treat a green check as independent confirmation. Do not staff the review with the person who is measured only on how many agent tasks they close. The 86% firefighting number is what that incentive produces.

Do not add observability by prompting "add logs" and never reading them. The report says teams are doing the prompting. The useful version is a log line you decided on, without secrets, attached to the action that moves money or records, with an alert on the failure you actually fear.

A saner default

Let the agent draft. Put a person on the paths that integrate, persist, authorize, or comply. Keep one test suite the agent cannot edit, aimed at those paths. Watch the production signals that match the four failure modes, not a single "AI quality" score. When the rework share starts looking like the survey's 25% or 50%, stop adding features on that surface until the rework drops. More prompting is how the queue of senior-engineer fixes gets longer.

What to measure after you change the review rule

The survey is a snapshot of other companies. Your own numbers are the ones that should move. For the next month, mark pull requests the agent substantially wrote, and mark incidents whose triggering change was one of those pull requests. You do not need a perfect label. You need an honest one, applied by the person who was on call, not by the agent that wrote the change.

If agent-written changes are a large share of merges and a larger share of incidents, the 78% pattern is yours, and the response is to slow the paths that show up in those incidents, not to ban the tool. If agent-written changes are common and absent from incidents, your review and your tests are already catching the gap the survey describes. Keep the measurement anyway. A new model or a new "ship without asking" setting can move the ratio in a week.

Pair it with rework, not only with outages. A change that was reverted, or rewritten by a senior engineer within a week, counts. That is the 74% figure translated into your history. Teams that only count pages will miss the quiet tax. The senior-engineer hours are the tax.

Share the ratio with the people who set the shipping policy. An 88% style policy that allows agent code in production is compatible with a rule that the four failure modes have an owner. It is not compatible with a rule that the ratio may rise forever. When the ratio rises, the next sprint fixes detection, it does not add a second agent.

How Techtribe takes a failing AI-written path apart

We reproduce the wrong production outcome with the inputs the live system actually sends, then we fix that path and leave a test the next agent run cannot quietly delete. We do not regenerate the app to get a greener demo. If the same agent is still merging into that path every day, the fix will not last, and we will say so.

When the product needs monitoring, regression tests, and an architecture someone can explain, that scope is a Production MVP: evals, regression tests, and cost, latency, and quality monitoring, on your cloud. A single broken integration does not require the whole package. The packages page has the boundary. Bring one example of a user who hit the wrong outcome, and the pull request that introduced it if you have it.

The fixed scope for taking a generated app to production is a page per tool: Lovable, Bolt, v0, and Replit.