Evidence review5 min read
How accurate are AI detectors?
Vendors claim 98% accuracy and under 1% false positives. Independent testing tells a different story, especially for non-native English writers.
Updated September 25, 2026
How this was tested
Vendor figures are quoted from the vendors’ own published claims; the independent findings summarise published third-party testing, with the Stanford non-native-writer study as the canonical example. We run no detector of our own and have no detector to sell — the closest instrument on this site is a pattern counter that explicitly reports counts, not verdicts, and we think that restraint is the correct response to the evidence below.
Numbers last verified:September 25, 2026
The short version
- Vendor claims cluster around 98–99% accuracy and under 1% false positives. Those are self-reported, on vendor-chosen test sets.
- Independent testing finds much higher false positive rates, with wide variation between tools and between text types.
- Detectors disproportionately flag non-native English writers, a documented bias that follows from how the tools work.
- Detector scores move when text is merely paraphrased or punctuation-edited, which tells you what they measure: token predictability, not authorship.
AI detectors are sold with two numbers: an accuracy percentage in the high nineties and a false positive rate below one percent. Those numbers come from the companies that make the detectors, and they are not reproduced by independent testing.
This matters practically, because detector output is now used in academic misconduct processes, hiring decisions and editorial judgements. If the instrument is weaker than advertised, the decisions built on it are weaker too.
What the vendors claim
| Claim type | Typical published figure | What it rests on |
|---|---|---|
| Overall accuracy | 98–99% | Vendor-internal benchmarks against vendor-selected samples |
| False positive rate | Under 1% | Same, usually on long-form English text |
| Adversarial robustness | Rarely stated | Paraphrasing and light editing are the known weak point |
| Cross-language performance | Rarely stated | Most published numbers are English-only |
There is nothing sinister about a company benchmarking its own product. It becomes a problem when those figures are quoted in a misconduct process as though they were independently established, which is the norm.
What independent testing finds
Independent comparisons tend to find three things. False positive rates are higher than advertised, often by an order of magnitude. Different detectors disagree with each other on the same text, sometimes dramatically. And accuracy collapses on short texts, on text in languages other than English, and on text that has been lightly edited.
The disagreement between tools is the most useful finding for anyone receiving a detector report. If two instruments are scoring the same essay at 12% and 88% AI, neither number is a measurement of anything stable. It is closer to two different guesses.
The finding that launched a thousand policy changes
A widely cited 2023 Stanford study tested detectors on essays by non-native English writers and found that a majority were flagged as AI-generated, while essays by native speakers were flagged at a rate of a few percent. That gap is not a bug in one product; it follows from the method.
Why non-native writers get flagged
Detectors estimate how predictable each token is given the ones before it. A passage whose words are highly predictable scores as machine-like. Second-language writing tends to use a narrower range of constructions and more conventional connectives, which makes it measurably more predictable, and therefore more likely to be flagged.
The same mechanism catches other groups: writers using translation tools, writers imitating a prescribed essay structure, and writers who have been taught a rigid formula. The tool is doing what it was built to do. The problem is that "predictable" and "machine-generated" are not the same property, and the output does not distinguish between them.
Why editing changes the score
A claim that cannot hold
"The text scored 74% AI, so AI wrote it."
What the score actually supports
"The text is more predictable under this model than the tool’s reference set. That can be caused by paraphrase, formulaic structure, translation, or generation."
Because the score tracks predictability rather than provenance, it moves when you change the surface. Ask a model to paraphrase a human essay and the score rises. Run a generated essay through a rewrite and it can fall. Editing your own work for a different register can move it in either direction.
That is also why "cleaning the AI tells" is not a route to a low score. Swapping punctuation does not change token predictability much, which is why removing every em dash will not move a detector reading in a meaningful way. Our own measurement agrees from the pattern side: a complete dash pass on an AI-style sample moves our count-based score from 96 to 68, and the vocabulary it cannot touch carries the rest. Anyone selling a "humanizer" as a detector bypass is selling a claim the technique cannot support.
How to use a detector result responsibly
- Treat a score as a prompt to look closer, never as a finding. This is what most institutional policies already say, and it is the correct reading of the evidence.
- Ask for the score in context: which tool, what threshold, what length of text, what does the vendor say about false positives in this genre.
- Never use a detector on its own for a decision that affects a person. The error rate is too high and the errors are not evenly distributed.
- Where you are the accused party, shift the discussion to process: what to do if you are accused of AI writing.
- If your goal is genuinely a better draft rather than a lower number, use a pattern checker that reports counts you can act on.
Skip the detector, fix the writing
A pattern count you can act on beats a score you cannot verify. Clear the dashes in one pass.
Open the AI dash removerFrequently asked questions
Are AI detectors accurate?
Less accurate than their published figures suggest. Vendors report 98–99% accuracy and under 1% false positives, but independent testing finds substantially higher false positive rates, wide disagreement between tools, and poor performance on short texts and non-English writing.
What is the false positive rate of AI detectors?
Published vendor claims sit under 1%. Independent studies report much higher rates, with wide variation by tool and by text type. The rate is also much higher for non-native English writers than for native speakers, which is the finding most often cited against using detector output as evidence.
Can AI detectors be wrong about human writing?
Regularly. Because they measure how predictable the text is rather than who produced it, formulaic writing, translated writing and second-language writing all score as machine-like. A false positive is not an edge case; it is a predictable failure mode of the method.
Does Turnitin use AI detection, and how reliable is it?
Turnitin offers an AI writing indicator alongside its similarity checking, and publishes a low false positive claim. Independent testing has not consistently reproduced that figure, and most institutions treat the indicator as a reason to investigate rather than as proof.
Does removing em dashes make text score as human?
Not meaningfully. Detectors score token predictability, and punctuation contributes almost nothing to that. Removing dashes improves how the writing reads to people, which is a different and arguably more useful goal.
Is there any reliable way to tell if text was written by AI?
Not from the text alone, and not with a tool. The reliable approach is process: drafts, edit history, source notes, and an author who can explain their choices. That is what actually distinguishes authorship, and it is why the advice for anyone accused is to produce the record.
Keep reading
Detection6 min read
What to do if you are accused of using AI to write
The steps that actually resolve an AI accusation: your drafting record, your sources, and why arguing about punctuation never works.
Updated September 25, 2026
AI writing5 min read
Signs of AI writing: the complete list of tells
Seventeen structural tells, the vocabulary and punctuation patterns, plus the "giveaways" that are actually myths. Ordered by how much each one really proves.
Updated September 25, 2026
AI writing7 min read
Do em dashes mean AI? What the evidence actually says
Em dashes are the most quoted AI tell and the weakest single one. Where the claim came from, what it can and cannot prove, and when a dash really is a signal.
Updated September 25, 2026