How to Recover an AI-Built App After a Failed Deploy

A failed deploy is a production incident with a cause you chose: a version you shipped. Recovery is returning to a version you trust, or forward to a fix, without losing the data users created while you were down. AI-built apps make this worse in a specific way. The person who shipped may not be able to explain the release, the migration may have been one destructive file, and the only one "who knows the system" is the agent that broke it.

The case where the agent deleted the database and the backups on the same volume is in when an AI coding agent deleted a production database. Permissions that would have stopped that are in what the agent can reach. This note is the afternoon the deploy goes bad and you still have a chance to come back, if the previous version and the backup are real.

What "rollback" has to name

The FFIEC IT handbook's rollback guidance describes a back-out plan as the steps back to a previous secure, functional baseline. It points at NIST SP 800-128 for security-focused configuration management. The useful parts for a small product are concrete. Know the stable recovery point. Archive or image it. Test the procedure. Write it so more than one person can follow it. Decide in advance when you will trigger it, because deciding during the outage, with users waiting, is how you improvise a second outage.

A recovery point for an AI-built app is three things, not one:

• The application artifact: a commit, an image, or a host snapshot you can redeploy. "The agent can regenerate it" is not an artifact. Regeneration will not match the version that was live, and it will not match the database.
• The schema the artifact expects. If this deploy ran a migration, the previous artifact may not run on the new schema. Rollback is then a schema problem. See AI-generated database migrations.
• The data. Orders taken after the backup are not in the backup. You need to know the time the backup represents, which is your recovery point objective, and how long a restore takes, which is your recovery time objective. NIST's data-protection guidance uses those two terms for the maximum age of backup data you can tolerate and the time you can tolerate being down. Write the numbers down before you need them. "As fast as possible" is not a number.

If you cannot name those three for the release you are about to ship, you are not ready to ship it, even if the agent says the tests passed.

A backup you have not restored is a rumor

NIST's contingency-planning bulletin, summarizing the SP 800-34 line of guidance, says testing is how you find deficiencies in the plan, and that tests should run as close to the operating environment as possible. A tabletop where someone reads the steps out loud will catch a missing owner. A restore onto a spare database will catch a backup that does not load, a secret that was not in the backup, and a runbook that skips a step. You want both. You need the restore.

cloud.gov's contingency-exercise template, based on a NIST SP 800-84 sample, uses a scenario that matches this article: a routine release corrupts a database, the team has to notice, restore, and finish or fix the deployment. The exercise is the point. The first time you run that scenario should not be the day it happens. Do a small version. Pick a non-production copy. Restore it. Time it. Write down what failed in the instructions.

NIST's piece on protecting data from ransomware states the 3-2-1 rule in plain language: three copies of an important file, on two different kinds of media, with one copy off-site. It also says to test restoration, including how long retrieval takes. For an agent-built app, translate that into systems you actually have. One copy is the live database. One is the provider's automated backup, ideally not on the same disk the agent can delete. One is an export you control in another account, so a compromised hosting login does not delete every copy. Keys required to restore, including the encryption key and the provider login, are part of the backup. A perfectly good dump you cannot decrypt is not a copy.

Test the restore of the thing you would miss: the orders table, the user table, the file bucket. A successful restore of an empty staging project does not count.

The order on the day

1. Say that it is an incident. One person is in charge. Everyone else, including the agent, gets tasks from them. Tell users the app is down if they can see that it is down. Silence makes the next hour harder.
2. Stop new damage. If a migration is still running, a job is still writing bad rows, or an agent is still "fixing" production, stop it. A second deploy on top of a bad one is how the recovery point disappears.
3. Decide rollback versus roll-forward. If the previous artifact works with the current schema, roll back now. If the schema change was additive and the new code is the problem, roll back the code and leave the extra column. If the schema change destroyed data, stop and go to the backup. Do not keep deploying forward because the agent has a theory.
4. If you restore, restore beside the broken database first when you can, not over the only remaining copy. Confirm the restored data looks right: recent rows, counts, a user you know. Then switch traffic. Overwriting the broken database immediately removes your chance to salvage the last few minutes of writes.
5. Reconcile the gap. Payments, signups, and messages that happened after the backup need a source of truth outside the database: the payment provider, the email log, the identity provider. Pull from those. Do not trust the agent's memory of what the table used to contain.
6. Only then fix forward. The fix is a commit a person reviews, deployed as the next release, with the previous artifact still deployable. The agent may draft the fix. It does not merge it alone, and it does not get a production token as a reward for causing the incident.
7. Write the timeline the same day. What shipped, who shipped it, when the bad behavior started, what you restored, what data you could not get back. That note is how the next one is shorter. It is also the document you will wish you had if personal data was involved. Notification duties are in GDPR and an exposed customer record.

What the agent is allowed to do during recovery

Let it read logs, summarize errors, and draft a patch in a branch. Do not let it run a further migration against the broken production database. Do not let it delete "stuck" resources. Do not let it rotate every secret at once unless you have a list and a way to update the running app, or you will lock yourself out of the system you are trying to save.

The credential it uses, if any, should be unable to delete backups. The person in charge holds the credential that can restore. That split feels slow for twenty minutes and feels correct the moment an agent "cleans up" the snapshot you needed.

If the only deploy path is a button in a builder you do not control, your recovery point is whatever that builder can republish, plus whatever the database host can restore. Find that out this week. A platform that cannot republish a previous version has made the agent your only source of the code. Export the code and be able to run it somewhere else before you need to. Ownership of the repo is the difference between a rollback and a plea to a vendor.

What to have in place on a quiet day

• The commit or image that is live, written down where the on-call person can find it, not only in the agent's session.
• A previous artifact you have deployed once, so you know the mechanism works.
• A backup on a different system, with the date of the last successful restore written next to it.
• A migration policy that does not destroy the old schema in the same release as the new code.
• A deploy that a person approves, and a way to stop a pipeline that is mid-failure.
• A short page: who decides, where the backups are, how to roll the app back, how to reach the host's support. One page is enough. A wiki nobody can open during an outage is not.
• A drill on the calendar. NIST's point about testing close to the real environment means the drill uses the same kind of database and the same kind of deploy, not a thought experiment.

The drill can be an hour. Restore last night's backup to a spare. Point a copy of the app at it. Log in. Place a test action. Measure the time. If you cannot finish, you have found the gap while users were not waiting.

What to stop doing

Do not ship a release you cannot name a rollback for. Do not store the only backup in the project the agent has write access to. Do not let "the agent will remember how it was" substitute for an artifact. Do not run a second experimental deploy because the first one is only a little bad. A little bad, with a known rollback, is recoverable. A little bad plus three more attempts is a new system nobody has seen.

Do not skip the reconcile step. Bringing the site back with a hole in today's orders is a second incident, discovered by the people who paid.

After the site is back

The restore is not the end. For the next day, watch the paths that write: new orders match the payment provider, new signups can log in, jobs that run on a schedule are running once and not twice. A restored database plus a queue that replayed the outage window will double-charge or double-email. Pause those jobs until you know whether the outage window needs a replay or a skip.

Tell users what was true. If an hour of orders was rebuilt from the payment provider, say that, especially if anything could not be rebuilt. A status message written during the panic and never corrected becomes the story. Correct it.

Then change one thing that would have made this shorter, and do it before the next feature. The usual candidates are a previous artifact you have now proved you can deploy, a backup in a second account, or a migration pipeline that no longer runs destructive down migrations on failure. Pick one. Write the date you proved it. An incident review that produces twelve action items and finishes none is how the next deploy fails the same way.

If personal data was exposed or lost, the notification clock is separate from the uptime clock. Do not let the relief of a green homepage close that file. The facts you wrote in the timeline are the input. Someone who is not the agent has to decide whether Article 33 applies and act.

Schedule the drill while the pain is still clear. A month later the runbook will have drifted toward optimism. Running it once while the team remembers the real failure is the cheapest hour you will spend on this product.

What "the agent can redeploy" leaves out

Builders and hosts often keep a previous publication you can click back to. Find out, before the outage, whether that click restores code only, or code and database, and whether environment variables roll back with it. A code rollback that keeps the new broken environment variable is a rollback in the user interface and not in production. Write down the click path and the last time someone used it.

Exports of the project are part of the same gap. If the only copy of the source is inside the builder, a bad day at the builder is a bad day for you, even when your database is fine. A repository you can clone, and a host you can deploy that repository to, is the artifact. Confirm the clone builds. A repository that is three weeks behind the builder is a recovery point from three weeks ago, which may be acceptable if you know it and is a surprise if you do not.

Secrets deserve a line in the runbook that is not "they are in the dashboard." Which dashboard, who can open it, and which of those secrets the previous version needs. Rotating everything during the incident because you are unsure is how the database comes back and the app still cannot log in. Rotate what was exposed. Keep a list of what the running version needs so you can put the same values, or the new values, back on purpose.

How Techtribe recovers a broken release, and how we make the next one recoverable

If the deploy has already failed, we start with what is still there: the previous artifact, the schema, the backups, the payment log. The goal of the first hours is a running version and an honest list of what data did not come back. We do not regenerate the app from a prompt while production is down.

If you are not in an incident and you want the rollback to exist, we wire the artifact, the backup restore, and a migration path that can roll back, and we run the restore once so the runbook is a record instead of a guess. That belongs with monitoring and regression tests in a Production MVP, or with a Dedicated Team when you are shipping every week and someone has to own the drill. Bring the host, whether you can deploy the previous version, and whether a backup has ever been restored. Those three answers decide the call.

The fixed scope for taking a generated app to production is a page per tool: Lovable, Bolt, v0, and Replit.