AI Found 99 Bugs in 90 Hours. People Checked for 80

In 2026 Forescout's Vedere Labs published a comparison they had earned the hard way. Between 2020 and 2021, Project Memoria spent months of human analysis on TCP/IP stacks used in operational technology and IoT devices. In 2026 a harness of ordinary models, supervised by the same lab, went back to that class of software. The press release states the headline carefully enough that the headline is usually misquoted. Analysis time fell from about nine months to 90 hours. Those 90 hours were about 10 hours of independent AI analysis and about 80 hours of validation. The output was 99 validated vulnerabilities, 66 of them previously unknown.

The number people repeat is 99 in 90 hours. The number that should change a product plan is the split. Ten hours of machine time produced candidates. Eighty hours of people produced vulnerabilities. Automated security testing is faster than it was. It is not a button that emits CVEs. What those CVEs look like when they land in a dependency you ship is the PostgreSQL batch of the same season, 52 fixes in one minor cycle, discussed in the November PostgreSQL note. What an AI-built app already tends to get wrong without any research harness is in the security failures that show up in vibe-coded apps. This note is the study, and the part of the study that was still a person.

What they ran, and what they refused to claim

The report is "Finding Critical Infrastructure Bugs with Off-the-Shelf AI." The targets were 10 TCP/IP stacks, seven of which Vedere Labs had already analyzed by hand and three of which were new. The point of including the old ones was a control. A harness that cannot rediscover known bugs is not a harness you should trust with unknown ones. They rediscovered 33 previously known issues, which the press release describes as about 40 percent of the previously known vulnerabilities in the stacks they re-examined. That is a recall number, not a perfection number. A tool that finds less than half of the old bugs will also miss new ones. They still found 66 new ones, and 52 percent of those 66 were high or critical. Fifty-four were reported to vendors and maintainers. Disclosure for the rest was still open when they published.

The method is a harness, not a chat window. Static analysis proposes candidates. The model reasons about them. Dynamic checks try to show the bug on a running stack, with a person able to step in. The lab is explicit that the work was not unattended. Researchers could intervene during dynamic verification. The 90 hours include that intervention. Anyone selling "autonomous pentest" off this press release is selling a different product than the one that was measured.

Cost is the other half of the speed. Static analysis across the 10 stacks was about $935, $2.58 per finding if you divide by the candidate math they publish. The report's own comparison table puts dynamic analysis in a band of roughly $290 to $1,380. The interesting sentence is the caveat: this is the marginal cost of pointing a finished harness at a new target. It does not include designing the harness, writing the static rules, building emulation, or tuning prompts until the agents behaved. A senior researcher for nine months costs more than $2,000 of tokens. A team that has not built the harness does not get the $935 price. They get a subscription and a pile of unvalidated findings.

Binary analysis, for stacks where source is not available, was about six times as expensive and twice as slow as source analysis. It still worked well enough for the lab to say the approach reaches proprietary firmware. "Well enough" is not "paste the firmware into a chatbot." Emulation infrastructure is the expensive object. The model is the cheap object once the emulator exists. That is the same shape as any other agent project: the tokens are not the project.

Why the old timeline mattered

Project Memoria's earlier pass on these stacks found 71 vulnerabilities across the ones this study re-analyzed, over about nine months of code reading inside an 18-month program. Those bugs were not academic. TCP/IP stacks in embedded devices sit under products the device vendor did not write and often cannot patch quickly. Forescout's point, carried over from the earlier research, is that a flaw in the stack is a flaw in hundreds of downstream products from dozens of vendors. Speed of discovery changes who has to move. A nine-month research project gives vendors a research calendar. A 90-hour rerun gives them a queue.

The lab's conclusion is aimed at defenders and at attackers at the same time, because the harness is not a secret weapon. It uses commercially available models. Once the method is published, the cost of a copy is the cost of the harness plus the tokens. They say that cost, for this class of bug, is low enough that both sides can pay it. Device makers and asset owners are the people who live with the result. A camera, a controller, or a badge reader does not get a weekly deploy. The party who can run the harness every weekend has more shots at the firmware than the party who can ship a patch every quarter.

They also argue for deciding, before this is normal, which defensive actions a system may take without a person. Their Assume Autonomy framing is the operational version: name the response you will delegate, the constraint on it, and how you reverse it. An automated patch or an automated firewall rule that nobody can undo is not a defense. It is the next incident. The study does not claim they built that response. It claims the discovery side is arriving first, and the response side is still a meeting.

A second study, with different numbers

Unit 42's NOVA write-up is the study people will mix into this one if you let them. NOVA is a system that reads a project, proposes vulnerabilities, builds a proof of concept, checks it in a clean environment, drafts a patch, and writes a disclosure, with a human at the end rather than throughout. In the window they report, it confirmed 14,090 vulnerabilities across 3,915 open-source projects, 99.4 percent of them previously unreported, about 40 percent high or critical. The strongest results came from an ensemble of models plus ordinary security tools, not from one model asked nicely.

Those numbers are not Forescout's numbers. The targets are different. The human role is different. The definition of "confirmed" is their pipeline's definition. Quote them separately or do not quote them. What you can say, because both labs said it, is that a model plus a harness plus a deterministic check finds real bugs at a price that looks like a cloud bill rather than a year's salary. What you cannot say is that the bugs are exploited, that the patches are correct, or that your application was in the set.

Microsoft's October 2026 note on frontier vulnerability research is a third data point of the same kind, and it stays in its own lane. Their MDASH work on the Linux kernel produced validated crashes at a few dollars and about twenty minutes of model time per proof of concept, on the subset they measured. Kernel crashes are not TCP/IP stack bugs and they are not your CRUD app. The pattern is the price of a candidate falling through the floor, and the validation remaining the scarce object.

PostgreSQL's maintainers have been living on the receiving end. Phoronix reported that the November 2026 release, 52 vulnerabilities and a delayed PostgreSQL 19, arrived while the project was absorbing a surge of AI-assisted reports. A maintainer's queue filled with machine-generated candidates is not a gift if most of them do not reproduce. It is a cost shift. The researcher pays $2. The project pays the afternoon. If you maintain a library, even a private one inside a company, you are about to be on that side of the table whenever someone points a harness at you. Demand a request, a version, and a crash or a diff. A paragraph that says "there is a SQL injection in the login" is not a report.

What this does to an app an agent already built

The stacks in the Forescout study are not a Next.js app. The transfer is the rate. The bugs that show up in generated apps are still mostly the ordinary ones: missing authorization, a service key on the client, a webhook that trusts its body, a bucket that is public. A harness aimed at memory corruption in a TCP stack will not find those, and a harness aimed at your routes might, if someone writes it. You should assume someone can. "We will pentest before launch" was a plan when a pentest took a month to schedule. It is a weak plan when a motivated person can run a first pass over the weekend for the price of a laptop.

The correct response is not to buy the first product that says autonomous pentest and paste its PDF into the board deck. Forescout's own split says why. Eighty of the ninety hours were validation. A report that lists 99 issues and cannot show you the request that proves each one is a list of candidates. Candidates are how you waste a sprint. The lab reported 54 of 66 new bugs to vendors because they believed those still affected current versions. That belief came from a person and a running system. Your scanner owes you the same: version, request, response, and the reason it is yours.

Do the cheap human pass on the bugs you already know generated apps have. Two accounts. One should not see the other's data. The admin route should not answer as a normal user. The file upload should not land in a public bucket. Those checks are not a harness, and they catch the incidents that actually happen to this class of product. A memory-safety harness on your dependencies is a second project, pointed at the native modules and the containers, with someone assigned to upstream the real findings. Running both and staffing neither is how you collect PDFs.

If a model tells you it found a critical bug, ask it for the reproduction before you thank it. If it cannot produce one, the finding is a hunch. Hunches are allowed in a research notebook. They are not allowed to block a release or, worse, to justify a rewrite. The rewrite is the expensive failure mode. An agent that "fixes" forty hallucinated vulnerabilities will churn the codebase and leave the real missing authorization alone, because the real one did not look like a CVE paragraph.

Patch the dependencies that have real advisories. The November PostgreSQL release is one of those. A TCP stack in a device you ship is one of those if Forescout or the vendor named it. A sentence in a chat is not. Track the advisory the way you track any other dependency bump: version in, version out, test the behavior you rely on.

How to read the next paper like this

Separate the clock. How many hours did the model run. How many hours did a person validate. A paper that adds those together and says "AI found" has not lied, and it has also not said what a buyer thinks it said. Forescout added them and then, in the FAQ, took them apart. Prefer labs that do that.

Separate the cost. Marginal token cost is not the cost of the program. Ask what the harness assumed: source code, an emulator, a sanitizer build, a person who already knew the bug class. If your target does not have those, you do not have their cost.

Separate confirmation from discovery. A crash in a sanitizer build is closer to a bug than a model's suspicion. A crash is still not an exploit, and an exploit is still not a breach of your tenants. Score the finding by what you can show a maintainer.

Separate targets. Embedded TCP stacks, the Linux kernel, and a multi-tenant SaaS app fail differently. A technique that works on parsers will flood a C project and say nothing about your row-level security. Buy, or build, the check that matches the failure you have actually had.

Ignore leaderboards that count unconfirmed issues. NOVA's 99.4 percent "previously unreported" is interesting because they required a proof of concept in a clean environment. A vendor who counts every model paragraph will outscore them by breakfast and will have found less. Ask what the denominator is.

What to check before you trust a machine's bug list

1. Split the last report you received into reproduced and not reproduced. The second pile is not a backlog.
2. For each reproduced issue, write the version and the request. If the version is not the one you deploy, it is not your bug yet.
3. Point the next automated pass at a dependency with a known advisory, as a control, before you point it at the whole repo. A harness that misses the known bug is not ready for the unknown one. That is the control Forescout built in.
4. Staff the validation hour. Ten hours of tokens and zero hours of a person is how you reenact the part of the study that was not the result.
5. Keep ordinary authorization tests in the pipeline. They catch the class of bug this TCP study was not looking for, which is the class your app is more likely to have.
6. When a maintainer, including you, receives a model-written report, require a reproduction before it jumps the queue. The PostgreSQL project's load is what happens when that rule slips.
7. If you cannot say which findings in your last scan were shown on a running build, you do not have a security test. You have a document.

Discovery got cheaper. Confirmation did not. The 80 hours are the work.

How Techtribe takes a security repair from here

We read the repo and try the live app with two accounts. You get the critical and high findings we can reproduce, in writing, then a fixed price for the ones you want closed. Working screens stay. We do not quote a rewrite so the invoice is larger, and we do not promote a model's unconfirmed list into that price.

A single pass fits when you have a URL and a repo and you need to know what is actually open. If the same agent is still merging into that repo every week, the findings will come back unless someone owns the tests. That ongoing ownership is a Dedicated Team. The scope call is where we say so. Bring the URL and whatever access you are willing to share. Thirty minutes with a founder is enough to tell you whether the scanner you have is testing the product.

The fixed scope for taking a generated app to production is a page per tool: Lovable, Bolt, v0, and Replit.