Article · AI code detection

Are AI code detectors just BS?

The industry selling "this code is 87% AI-written" verdicts cannot deliver them, their own research says so, and managers are starting to make people-decisions on numbers that don't survive a reformat. I measured 136 codebases to check what's actually knowable. Here are the receipts.

August 2026 · every figure re-runnable: the evidence map · the polite version of this argument: can you tell AI code from human code?

The claim being sold

There is now a product category that promises to tell you which code an AI wrote — per file, per commit, sometimes per developer. The pitch lands with managers because the question is real: boards are asking, compliance is asking, and "we don't know" feels like an answer you can't give.

Here's what the detection field's own published research says about that promise.

The receipts, from their own literature

Let me spell out what the field's own papers admit, because no vendor deck will.

The best multilingual classifier anyone has published scores an F1 of 84.1%. That number sounds great in a demo. Now do the arithmetic on your actual estate: five hundred files, and something like one call in six comes back wrong — and every wrong call has a developer's name attached to it. Picture actually running that meeting. "The tool says your parser is AI-written." "It isn't. I wrote it in 2019." "Well, the tool says—". Congratulations: you've procured a machine for insulting your own staff at scale, with decimals for authority.

And what is the classifier actually looking at when it makes these calls? This is my favourite detail in the entire literature. When researchers tested which features carry the weight, the winner wasn't program structure, wasn't control flow, wasn't anything an engineer would respect. It was the comments. The comments! The most casually editable text in a source file moved the decision boundary more than the abstract syntax tree did. Strip them, reword them, run the formatter while you're at it — and the verdict flips. I've measured code for twenty years and I genuinely struggle to think of a flimsier basis for pointing a finger at a colleague.

Then there's the shelf life. These things are trained on the output of last year's models and quietly rot against this year's — the researchers themselves now treat "does it still work on newer models?" as an open question to be re-tested, which tells you everything. The target moves every release cycle. Your detector doesn't. Nobody selling you a licence mentions the decay curve.

But the problem I find fatal is simpler than any of that: the thing these detectors claim to detect barely exists. Whose code is purely anything anymore? I edit what the model writes; the model refines what I wrote last month; the same file passes through both of us in a week. The field's own dataset-builders ran into this and had to invent a third category — "machine-refined" — because honest pure cases were too rare to build on. The binary question these products charge money to answer dissolves before it can be asked.

As for the neural detectors reporting 97% accuracy: measured on their own datasets. Every objection above still applies, just with more confident decimals.

None of this is my measurement — it's theirs, and the papers are linked, claim by claim, on the evidence page (surveyed properly in our detection paper). So when a vendor offers you per-file authorship verdicts, ask them which of these problems they've solved — brittleness, drift, mixed authorship — and where they published it. The reviewers of those papers would be delighted to hear.

What arithmetic can prove

Now the part that actually works — and it's embarrassingly simple. Forget the text of the code; measure the shape of the change. Classify every changed statement between two versions as added, deleted, or edited in place. Compute one ratio: the share of churn spent editing what already exists. Call it REWORK. No ML. No stylistic guessing. A diff and a division, reproducible by anyone from two commit hashes.

I ran it across 136 of the largest public codebases, year by year back to 2010 — about 393 million changed statements. What comes out:

• Established hand-maintained projects cluster near 17% — one changed statement in six is an in-place repair. curl, PostgreSQL, Express: different languages, different decades, same band. Even bug-fix-only maintenance branches sit in it.
• An intensively agent-built application: 0.19%. One in five hundred. A ninety-fold collapse in repair, because generated code gets regenerated, not fixed. The signature is workflow mechanics — it can't be prompt-engineered away, and better models make it stronger.
• And the caveat the vendors won't lead with: fast-growing young projects also score low — Kubernetes in 2015 hypergrowth hit 0.5% with no AI in sight. A low number alone proves nothing. The honest inference is longitudinal: an established codebase whose REWORK falls out of its own historical band has changed how it's developed. That's a process fact, not a personnel verdict.

Notice what this measurement refuses to do: it will not tell you which developer used AI, and it will not tell you which file to be angry about. That's not a weakness. That's the measurement declining to lie to you.

The question the detector industry is distracting you from

While everyone argues about who wrote a function, the question with actual consequences goes unasked: is there an AI inside your software, and what can it reach? Code that calls a model and pipes the output into exec, a shell, or a network write is an agent with hands — in your tree or three dependencies deep. That's not a stylometric guess; it's static analysis with file-and-line evidence. In July 2025 an autonomous coding agent deleted a production database during a declared code freeze and misreported that rollback was impossible. You cannot ask the agent. You can inventory the code.

Break my numbers

Every figure above reproduces from public git history: the hash pairs are printed in the papers and collected on the evidence page, and the instrument is free to run. If you re-run a row and get a different answer, I want to hear about it — that's a standing offer no per-file detector vendor will make you, and the difference between measurement and astrology is exactly that offer.