Are AI detectors accurate?

Some are accurate on the text they were tested on, and none is right every time. The number that decides whether you can trust a flag is how often human writing gets flagged, weighed against how much of what you read was written by AI to begin with. When AI text is rare, even a detector that looks accurate on paper can point at more people than machines.

How accurate AI detectors are, in published numbers

Every detector makes two kinds of mistake. A false positive flags human writing as likely AI-written. A false negative lets AI-written text through. Tuning a detector trades one for the other: flag more, and you catch more AI text but also more people.

Published results span a wide range:

• OpenAI’s own text classifier, released in January 2023, labeled 26% of AI-written text as likely AI-written on its challenge set of English texts and wrongly flagged human text 9% of the time. OpenAI withdrew it on July 20, 2023, citing its low rate of accuracy.
• A 2023 Stanford study of seven detectors found they flagged 61% of essays by non-native English writers as AI-generated, on average, while doing well on essays by US eighth graders.
• Turnitin said in May 2023 that its document false positive rate is under 1% for documents where it detects more than 20% AI writing, and about 4% for individual sentences. These are the vendor’s own figures.
• GPTZero says it keeps its false positive rate at no more than 1% when evaluating AI against human text, also a vendor figure.
• Pangram’s July 2026 technical report gives a false positive rate of 0.0041% on over a million human-written English texts it does not name, counting a “Mixed” result as a false positive. We cover how Pangram compares with Botfilter separately.
• On a held-out test set scored once, after the model was fixed, Botfilter at its default setting caught 97% of AI-written texts and flagged between 0.2% and 1.8% of human texts, depending on the kind of writing.

These numbers come from different texts, so they do not rank the tools against each other. They do show the pattern that matters: error rates depend on what you test on, and nobody has measured every kind of writing.

A score is a probability, not a verdict

A good detector does not answer “was this written by AI?” It answers “how likely is it, given text like this?” Botfilter shows that directly. When you hover over a flagged post, it reads, for example, “Likely AI-written · 98%”, with a Not AI button beside it for when you disagree.

That percentage is calibrated on our validation texts, where about 1 in 8 texts was AI-written: across texts shown at about 90%, about 90% were. Where AI text is rarer, the same number overstates the chance, for the base-rate reason explained below. On that validation mix, the average gap between the shown probability and the observed share ranged from 1.5 to 5.5 points depending on the kind of writing, with web pages the least well calibrated.

A flag is a separate decision: Botfilter marks text when its score clears a threshold set by the sensitivity you choose. The settings page describes the trade: Fewer flags about 1 in 200 human posts by mistake, Balanced about 1 in 100, and More about 3 in 100 while catching more AI text. Those are targets. On the held-out test set, Balanced flagged 0.2% to 1.8% of human texts, and More reached 5.1% on web pages, as the table further down shows.

The thresholds differ for short and long text. At Balanced, a flagged post of 25 to 80 words always shows 92% or more, while longer text can be flagged from about 70%. At More, long text can be flagged from about 21%.

Why a small error rate can mean many wrong flags

A false positive rate tells you how often human text gets flagged. It does not tell you how often a flag is wrong. That second number depends on how much of what you read was written by AI in the first place: the base rate.

Take 1,000 short posts and a detector that catches 96% of AI-written ones, roughly what Botfilter measured on short social posts. The two tables use false positive rates we measured for short text at Balanced: 0.5%, as on short Reddit posts, and 2.0%, as on short passages from web pages. The shares of AI-written posts are examples, not measurements of any site.

When AI-written text is rare, even a low false positive rate produces a large share of wrong flags: at 1 in 100 and 2.0%, two of every three flags point at a person. The same arithmetic applies to any detector, including ones with far lower published rates, and it is why a false positive rate measured on your kind of writing matters more than a headline accuracy figure.

This is why a score that is fine for tidying a feed is unfit for judging one essay. Filtering a feed spreads the cost of a mistake thin: you skip a post you might have liked. Judging a person puts the whole cost on one individual.

Whose writing gets flagged by mistake

Detectors learn what machine text tends to look like: predictable word choices, even rhythm, familiar structure. Some human writing looks like that too.

Non-native English writers

Liang and colleagues at Stanford ran 91 TOEFL essays through seven public detectors in March 2023. The detectors wrongly flagged 61.22% of them on average. All seven flagged 18 of the essays, and at least one flagged 89 of the 91. The authors suggest a cause: writers with a smaller vocabulary choose more predictable words, and detectors that score predictability read that as machine text. When the researchers had ChatGPT rewrite the same essays with richer word choices, the average false positive rate fell to 11.77%. When it rewrote US eighth-grade essays to sound like a non-native speaker, misclassification rose from 5.19% to 56.65%.

Some newer detectors report low rates on this kind of text. Pangram’s report gives 1 false positive in 24,586 non-native English samples, including the Stanford TOEFL essays, and GPTZero says its rate on TOEFL essays is 1.1%. Both are vendor figures. We have not measured Botfilter on this kind of text, for the reason given further down.

Formal, templated, and polished writing

Official statements, boilerplate email, and text run through an editing tool share traits with model output. Our highest measured false positive rates are on work email and web pages, at 1.6% and 1.8% at Balanced. In our robustness tests on validation texts, adding a template sign-off with an unfilled placeholder (“Best regards, [Your Name]”) to human chat messages raised their flag rate by 4.4 points, the largest effect of any edit we tried. A real sign-off such as “Thanks, Mike” did not change it.

Edited and paraphrased text

Mixed authorship blurs the question itself. We tested human texts that a model then changed. Light polish was flagged 11% of the time, heavy rewrites 56%, and notes expanded into full prose 94%. Whether each of those should count as a hit depends on what you want to filter.

Editing also works against detectors. Sadasivan and colleagues showed that repeatedly paraphrasing AI text significantly lowers detection rates across several detector types, with only a small loss in quality in many cases. A 2023 study of 12 public tools and two commercial systems, Turnitin and PlagiarismCheck, concluded they were “neither accurate nor reliable”, leaned toward calling text human-written, and did worse on obfuscated text.

If a detector flagged your own writing

A score cannot show who wrote something or how. It reports how much the text resembles what the detector learned to call machine writing, and the research above shows that careful, formal, or non-native writing can resemble it.

Start by finding out which tool produced the score, at what setting, and what false positive rate that tool publishes for your kind of writing and length. Turnitin, for example, said in May 2023 that it marks scores under 20% with an asterisk because they are less reliable, that it raised its minimum from 150 to 300 words, and that educators should use the score to start a dialogue with the student.

Your process is better evidence than any score. Drafts, notes, sources you read, and the version history in Google Docs or Word show how a piece took shape. If English is not your first language, the Stanford study is published, peer-reviewed evidence that detectors have flagged writing like yours far more often than native speakers’ work.

Why short text gets no score

A sentence or two carries too little signal. OpenAI said its classifier was unreliable below 1,000 characters. Botfilter does not score anything under 25 words: short replies, captions, and one-line comments are left alone rather than guessed at.

Even above that line, length matters. On the held-out test set at Balanced, Botfilter missed 9% of AI-written articles of 25 to 80 words, against under 1% of longer ones. Short text also needs a stricter threshold to hold the same false positive rate, which is why short and long text have separate settings.

What we measured for Botfilter, by kind of writing

We held back a test set that was scored once, after the model and its thresholds were fixed. Where the human texts carry dates, they predate ChatGPT, to keep AI writing off the human side. Here is how often each kind was flagged.

On the same test set at Balanced, Botfilter caught 94% of text from Gemini 3.8 Flash, a model it never trained on, and 99% from Granite 3.3 8B, a model family it never saw. Across the 20 models that wrote the test texts, it caught between 93% and 100%. The training data sources, the model, and the full results are described in how the Botfilter detector was trained.

What we did not measure

We could not license human-written posts from X or LinkedIn, so false positive rates there are unmeasured; pre-ChatGPT Reddit, chat, and email are the closest kinds of writing we did measure. We also found no essay collection by non-native English writers that we could license, so we cannot tell you whether Botfilter repeats the pattern in the Stanford study. Botfilter was trained and tested on English only. It still scores other languages, and how often it flags them is unmeasured.

How to use a detector without harming anyone

• Use a flag to decide what to read. Do not use it to grade, hire, ban, or accuse anyone.
• Look at the probability as well as the flag. A post shown at 97% and one shown at 71% carry different weight.
• Choose the sensitivity by the cost of a mistake: Fewer where a wrong flag would bother you, More where missed AI text bothers you more.
• Press Not AI when Botfilter gets a post wrong. It stops marking that text, and the choice is saved in your browser.
• Check where a detector sends the text. Most detectors are web services, so checking someone’s message means sending it to a server. Botfilter’s model runs in your browser on your device: page text is scored there, is never sent anywhere or stored, and there is no account. The guide to how Botfilter runs on your device covers what it stores, including synced settings, and how to check it yourself.

If you need a verdict on one document, a hosted checker with published accuracy may suit you better; the comparison of AI detectors covers where each sends your text. If you want less AI-written text in your own feeds, see how to get Botfilter.