Claude Found 29,000 Possible Flaws in Open Source. Humans Have Checked 6,000.

TL;DR: On 8 October Anthropic launched a free, opt-in ai vulnerability scanner for open-source projects. The striking part isn’t the scale — it’s the admission underneath it. Anthropic says it has found over 29,000 candidate vulnerabilities and has been able to review only about 6,000, because “we remain bottlenecked on our human capacity to validate these findings.” Its answer is to stop waiting: reports now go to maintainers fully model-generated, without human review. Every app you’ve built sits on this ecosystem.

What launched

Anthropic announced OSS Scanner on 8 October — “an opt-in vulnerability scanner for the open-source ecosystem”, free to eligible projects and modelled on Google’s OSS-Fuzz. It is the largest ai vulnerability scanner yet pointed at the packages the rest of us build on.

Maintainers enrol by submitting a pull request to a GitHub repository. Eligibility is decided case by case, aimed at projects with a “critical impact on infrastructure and user security.”

Each report contains a self-contained reproducer, an explanation of the flaw — including, where possible, a bisection identifying when the bug was introduced — and a candidate patch where one can be generated.

The line that makes this a story

Anthropic is unusually direct about what it is shipping:

“The outputs of this opt-in vulnerability scanner will be fully model-generated, without human review or triage.”

No person checks this ai vulnerability scanner’s output before it reaches a maintainer. And there is no 90-day coordinated-disclosure clock on the unvalidated findings, which is the convention that normally gives projects time to fix something quietly.

Human-verified reports still go through the traditional disclosure process. This ai vulnerability scanner is an optional fast track for maintainers who want everything as soon as it exists.

Why they did it

Because the humans ran out. These are Anthropic’s own figures about its own programme:

• “We have discovered over 29,000 candidate vulnerabilities.”
• It has “only been able to manually review and triage approximately 6,000 of these.”
• It has “sent nearly 5,000 reports directly to maintainers after they asked to receive everything we had.”
• “we remain bottlenecked on our human capacity to validate these findings.”

Read “candidate” carefully. 29,000 is the number of things the model flagged, not 29,000 confirmed bugs. Several outlets have run that figure as though it meant confirmed flaws. It doesn’t, and Anthropic doesn’t say it does.

Is this ai vulnerability scanner any good?

Anthropic ran its own check, and the result is its own measurement rather than an independent one. Its penetration testers reviewed 97 critical and high-severity findings across 48 projects. “Of these, 85 (88%) met the bar for our CVD process.” Of the remaining twelve, eleven were real but duplicated known issues, and one was a false positive.

One false positive in ninety-seven is a striking hit rate for an ai vulnerability scanner with no human in the loop — but it is a vendor testing its own product, on a sample it chose, and should be read that way until someone outside Anthropic repeats it.

Maintainers quoted in the announcement are broadly positive, though all of these reach us through Anthropic’s own write-up:

• Noah Misch, PostgreSQL: “An unusually high fraction of OSS Scanner’s findings uncovered PostgreSQL defects.”
• Anton Arapov, OpenSSL Corporation: reports “were as good and sometimes better than what we get from people.”
• Daniel Stenberg, curl: “OSS Scanner has helped us find multiple issues in curl worthy of addressing.”
• Eddie Kohler, HotCRP: “The bug reports were thorough and clear.”

Anthropic also publishes the complaint: “Some have told us severity ratings can be inflated”, and the scanner can misread a project’s threat model.

What an ai vulnerability scanner means if you don’t code

Here is the connection nobody is drawing. Nothing you build sits on its own.

An app from Lovable, Replit, Bolt or Figma Make rests on a stack of open-source packages — the database driver, the authentication library, the web framework. You didn’t choose most of them and you can’t name them. When one has a flaw, your app has it too.

We’ve written this story twice from the sharp end: 16,326 apps leaking user data through an unticked database setting, and a critical framework flaw that Lovable found and shielded its customers from before most of them knew it existed.

An ai vulnerability scanner working through that ecosystem at this scale is, on balance, very good news for you. The packages under your app get more eyes, even if those eyes belong to an ai vulnerability scanner rather than a person.

The uncomfortable half: a disclosed-but-unpatched flaw is a window in which the problem is public and the fix isn’t. Twenty-three thousand unreviewed candidates sitting in a queue is not a crisis, but it isn’t nothing either, and the people who have to act on them are mostly volunteers.

What to actually do

1. Keep your dependencies updated. Unglamorous, and the single highest-value thing. Most builders have an update button; press it.
2. Stop treating “no known vulnerability” as “safe”. No ai vulnerability scanner has looked at everything yet. It increasingly means nobody has looked yet — and the gap between found and fixed is now wider than it was.
3. Know what your app is built on. Your builder can tell you. You don’t need to understand the list, just to have seen it.
4. Don’t try to enrol. This is for maintainers of established infrastructure projects, not for your app. Nothing to do here.
5. If you maintain anything open-source, even small, it’s worth reading the eligibility criteria.

Who should care (and who shouldn’t)

• Running a live app with users: point 1, today. Your dependencies are the attack surface you didn’t choose.
• Building for clients: point 3 gives you an answer when someone asks what the app depends on. They will ask.
• Maintaining an open-source project: a free ai vulnerability scanner from a frontier model, with the honest caveat that reports arrive unreviewed.
• Still learning, nothing live: file it. The lesson — your code is the small part of what you ship — is worth learning before you need it.
• Worried this means your app is riddled with holes: it doesn’t. It means the ecosystem is being examined more closely than before, which is the opposite of bad.

Our take

Publishing the bottleneck was the brave bit, and it’s the part worth respecting.

A company could easily have announced a free scanner as pure generosity. Anthropic instead published the number it can’t keep up with, and then shipped a change that makes sense only because of it. Last week we wrote about the same company taking the internet away from its own agents after finding behaviour it didn’t like. Twice in a fortnight it has published something inconvenient about itself. That is a habit worth noticing in a vendor.

On an ai vulnerability scanner with no human in the loop: we think it’s right, narrowly. The alternative is 23,000 candidate findings nobody ever looks at, which helps no one. Sending them to maintainers who asked for them, clearly labelled as unreviewed, is better than a queue. The one-in-ninety-seven false-positive rate — Anthropic’s own measurement — suggests the noise is manageable.

What we’d want next from this ai vulnerability scanner is somebody else’s number. The accuracy claim is the only part of this that needs independent checking, and it’s the part a vendor can least credibly supply. Until then, treat 88% as a vendor’s figure.

And the thing for this audience to take away is the simplest one. The most important security work on your app is being done by people you’ll never meet, on code you didn’t write, and the only lever you have is keeping it up to date. Press the button.

Not sure what your tool is actually building on? Take the 60-second Vibe Coding Tool Finder quiz →

FAQ