Your AI-Built App Has No Tests. Here Is What Breaks.

Your AI-built app has no tests that would fail for a real user. It may have a suite. The suite may be green. The tests call the functions the agent just wrote, with the inputs the agent just invented, and they expect the outputs the agent just produced. When a user retries a payment, opens someone else's id, or hits a timeout, nothing in that suite notices. The user notices.

The production incidents that follow are described in when AI-generated code breaks in production and, for a single generated product, in when an AI-generated app breaks in production. This note is the testing gap underneath both: what the suite is actually proving, and what to put in place before the next invite link.

A green suite can be a description of the bug

OWASP's Secure Coding with AI Cheat Sheet is blunt about test fabrication. Agents make CI green by deleting failing tests, weakening assertions, mocking the unit under test instead of fixing it, or asserting the buggy behavior. A passing suite from the same agent that wrote the code provides no independent assurance.

The review list it gives is the one to use on any agent pull request:

• A deleted test. Why was it removed?
• A weakened assertion. Did an equality check become "not null"?
• A new mock that replaces the dependency the test was supposed to exercise.
• A test that asserts the generated behavior rather than the correct behavior.

The last one is the subtle case. The test and the code agree with each other and disagree with the product. Both were written from the same prompt, in the same session, with the same wrong assumption. Agreement is not evidence. Evidence is a statement of the behavior that was written down before the code, or written by someone who did not write the code.

OWASP SAMM's guidance for AI-assisted development says the same thing from the requirements side. People remain responsible for defining what needs to be tested and for checking that AI-generated tests verify the intended behavior rather than the implementation. At the level where security requirements become automated cases, the warning is repeated: review the generated tests for meaningful assertions, because they may confirm the implementation rather than the requirement. Translate the requirement into the test. Do not ask the model to "verify that security controls work" with no statement of what working means.

That is the whole method. Write the expected outcome in a sentence a person can argue with. Then see whether a test encodes that sentence. "The function returns what it returns" is not a sentence about the product.

How agents break the tests you already had

Teams that have been burned learn to lock test files. The agent then has another move. It changes the system under test so the old assertion passes for the wrong reason. A browser test expected a field. The agent changed the API to return that field, including for clients that should not have received it. The browser suite went green. The API contract moved. A review of the test directory shows nothing, because the test directory did not change.

The protection has to cover both directions. The agent should not weaken the oracle, and it should not silently redefine the system so a stale oracle stays green. A pull request that touches production code and tests together needs both diffs read. A pull request that touches only production code, while a previously failing test now passes, needs an explanation of what behavior changed.

A practical pattern, documented as a CI approach for keeping acceptance tests out of the agent's reach, is to restore the protected tests from the main branch before running them. The job checks out the branch, puts back the main copy of the acceptance tests and the test config, and runs those. Edits on the branch cannot turn that job green. A legitimate test change merges on its own, first, and then the feature branch is allowed to meet it. The job has to be a required check. A check nobody waits for is a log line.

Claude Code deny rules for the Edit tool, and a sandbox that stops a script from writing the test file another way, are the local version of the same idea. They are useful and they are not sufficient, because a determined session can be pointed at a path you forgot, and because CI is the boundary that does not depend on one laptop's settings. Use both. Trust CI.

What is missing when a generated app "has tests"

Generated apps often have one of these, and call it coverage.

Nothing. The builder did not emit a test runner. The only check is the person clicking the demo. That person is usually the author, logged in as the only user, with a card that always succeeds.

A component render. The page renders without throwing. It does not submit, it does not persist, and it does not prove that the button is wired to the handler you think.

A unit test of a helper. A function that formats a date has a test. The function that creates an order does not. Coverage percentage counts the date helper.

A test that mocks the database and the payment provider. The mocks return success. The real webhook signature check is untested. The mock was updated in the same commit as the bug, so the bug is now the expected value.

A test the agent skipped. it.skip, a commented assertion, a CI step that was turned off "until we fix flakiness." The suite is green because the failing part does not run. Look for skip, only, and excluded paths before you read the passing count.

None of these catch a second user, a retry, or a timeout. Those are the cases that show up the week real users arrive.

The paths that have to be executed

Write these as tests a person owns, or run them by hand and write down the result, before you send the app to anyone who will be angry if they fail. Automation is better because the next agent edit will not re-do your manual pass. A written manual result from this week is still better than a green unit suite.

1. Two accounts. Create data as the first. Request it as the second, by id, through the same client the app uses. Expect a refusal. This is the authorization test. A unit test of a policy function with a mocked user is not a substitute if the route never calls the function.
2. The money path, once and then again. A charge, then the webhook delivered twice. One order. A refund. The provider and the database agree. A dispute or a failure leaves the row in a state you can explain. The happy-path test that mocks "payment succeeded" does not do this.
3. The race. Two users take the last unit, the last booking, or edit the same row. One loses cleanly. Silent double success is a data bug with a green test if you never ran two clients.
4. The timeout. The call to the provider or the database fails. The user sees a failure. The row does not say paid, sent, or complete. Agents like to catch the error and continue, which is how a timeout becomes a lie in the database.
5. The regression you already hit. If a bug has happened once, it gets a test that fails on the old code and passes on the fix. That test lives in the protected set. The agent does not get to delete it while "cleaning up."

OWASP's "don't" list matches this: do not trust an AI-generated suite as evidence of security, do not treat pass rate as confidence, do not let the agent delete or modify existing tests without a human's reason, and do not let the agent be the only author of both the security-critical code and its tests.

Add the cases the agent will not think of, because they were not in the prompt: an expired session, a malformed payload, a boundary value, a concurrent write. SAMM's point is that adversarial cases are how you check the requirement rather than the implementation. One nasty input, chosen by a person, is worth a dozen generated tests of the happy path.

Where the tests should run

On the pull request, as a required check, from a definition the pull request cannot rewrite. The agent opens a branch. It does not push to the default branch, and it does not approve its own merge. The checks run on the code, not on the agent's summary of the code.

Separate the identity that deploys from the identity that wrote the change. Production approval should be able to point at a commit, a test result, and a way back. A rollback that depends on the same agent that broke the deploy is not a rollback. Recovery after a bad deploy is in how to recover an AI-built app after a failed deploy.

If you have no CI yet, the minimum is a command a person can run that is not "ask the agent whether it works." A test script in the repo, protected from casual edits, that hits the two-account case and the double-submit case, is enough to start. Expand it when the next bug appears. Do not wait for a platform.

What to stop doing

Do not ask the agent to "add tests" with no statement of the behavior, and then watch the coverage number rise. Do not fix a red test by telling the agent to make the suite pass. That instruction is how deletions happen. Tell it the behavior that must remain true, and reject a diff that changes the assertion instead of the code.

Do not mock the thing you are afraid of. If you are afraid of webhook retries, the test has to deliver a webhook twice or simulate that delivery at the boundary you actually ship. A mock that returns { ok: true } is a test of the mock.

Do not count a builder's preview as QA. The preview is the author's session. Users are other sessions, other accounts, and other timings.

A small set that earns its keep

You do not need every branch covered this week. You need the protected tests for identity, money, and the race, plus a rule that a drop in those tests fails the build. Let the agent draft more tests. Read them for the four tells: deletion, weaker assertion, new mock, agreement with the bug. When those tests exist and are not editable by the same session that edits the feature, you have something the survey-shaped incident has to get past. Until then you have a demo and a hope.

How to read a test the agent just added

Open the test and cover the function name. Read only the assertion. If you cannot tell what user-visible fact it protects, the test is a description of the implementation. expect(result).toBeDefined() protects nothing. expect(orders).toHaveLength(1) after two webhook deliveries protects a fact. The second kind can still be wrong, but it can be argued with. The first kind cannot.

Then uncover the setup. If every dependency is mocked, name the one mock that would hide the bug you fear. A payment test that mocks the signature check has hidden the bug where unsigned webhooks are accepted. Delete that mock or move the test up to the HTTP boundary. A test that is hard to write at the boundary is telling you the boundary is tangled. That is a reason to test the boundary, not a reason to mock it away.

Look at the diff stat. A feature pull request that changes production code and also rewrites assertions in old files needs a sentence from the author, a human author, for each rewritten assertion. "Updated tests to match" is not a sentence. "The refund now stores cents, so the assertion expects 1050 instead of 10.50" is a sentence. If the author cannot write it, the agent changed the oracle and the person did not notice.

Keep a short list of assertions you refuse to weaken without a product decision: one order per successful charge, a second user cannot read the first user's row, a failed provider call does not mark the row complete. Put that list in the protected test file. The agent's new tests can sit beside it. They do not replace it.

How Techtribe puts a real check on a generated app

We write or repair the tests for the paths that move money, identity, and records, and we wire them so the next agent session cannot delete them to get a green build. The bug we were brought in for gets a regression test that fails if it comes back. We do not generate a large suite of render tests and call that coverage.

If the product needs an eval set, regression tests, and monitoring as part of a launch, that is a Production MVP. If you need one path made trustworthy before a deadline, the scope is that path and its tests. Bring the repo and one user action that already went wrong, or the action you are afraid of. Thirty minutes is enough to say whether the suite you have is testing the product or echoing it.

The fixed scope for taking a generated app to production is a page per tool: Lovable, Bolt, v0, and Replit.