Are AI Detectors Accurate? An Honest Answer

Are AI detectors accurate?

Are AI detectors accurate? They are accurate enough to flag content worth a closer look, and not accurate enough to prove anything on their own. Every detector makes mistakes in both directions, and how often depends heavily on what you feed it.

Be wary of any single accuracy figure. A number measured on one test set, with one set of generators and clean files, rarely holds on the edited, compressed and mixed content you meet in real life.

The history of the field shows it. OpenAI launched its own AI text classifier in early 2023 and withdrew it in July 2023, noting in an update to its announcement that the tool was no longer available due to its low rate of accuracy.

How do AI detectors work?

AI detectors work in three broad ways: they read provenance data, they use a classifier trained to recognize generated content, or they measure statistical properties such as predictability. Most serious tools combine at least two.

Provenance is the only method that can name a generator with certainty, when the data is present and valid. Our guide to C2PA and Content Credentials explains how it works and why it is often missing.

Why do AI detection scores drift?

AI detection scores drift because the content they judge keeps changing while the detector stands still. A detector learns the traces left by the generators it was built on; a new generator leaves different traces.

• New generators: each model release can produce output that older classifiers have never seen.
• Compression: social platforms and messaging apps recompress images, audio and video, erasing fine traces.
• Editing: cropping, filters, paraphrasing, retouching and mixing human and AI work all blur the signal.
• Short content: a tweet, a short voice note or a small function gives too little evidence to judge.
• Language and domain: detectors are usually strongest on the languages and genres they were tested on.

This is why the same file can score differently in two tools, or in the same tool a few months apart. Neither result is a lie; both are estimates.

Drift also runs the other way. People who read a lot of AI output can pick up its phrasing without ever pasting a word of it. A detector cannot separate a person influenced by AI style from AI itself.

False positives and false negatives: who gets hurt?

A false positive labels human work as AI; a false negative lets generated content pass as human. Both cause harm, but false positives land on a specific person who did nothing wrong.

Research has documented who bears the false positives. A 2023 study by Stanford researchers (Liang and colleagues, published in the journal Patterns) found that several widely used GPT detectors consistently misclassified essays by non-native English writers as AI-generated, while judging essays by native writers far more accurately. The authors cautioned against using such detectors in evaluative or educational settings.

The mechanism is the one described in our guide on how to tell if text is AI generated: simpler, more predictable vocabulary looks machine-like to a predictability-based detector.

What does an AI percentage or AI detection score really mean?

An AI percentage is usually a model confidence score, not the share of the text that was generated and not the probability that someone cheated. Reading it as either is the most common misuse of AI detection.

• A “70% AI” label does not mean 70% of the words came from a chatbot.
• It does not mean there is a 70% chance the person broke a rule.
• Scores from different tools are on different scales and cannot be compared.
• A precise-looking number invites precise-looking decisions, such as a cut-off that turns an estimate into a verdict.

Context matters as much as the score. If almost everything you check is human-written, even a rare false positive will make up a large share of the items flagged. The rarer real AI use is in your setting, the more careful you need to be with every flag.

Why AnyFact uses three levels instead of a percentage

AnyFact gives a verdict in three levels, probably human, uncertain or probably AI, because that is the precision the evidence actually supports. There is never a bare percentage.

Text, document and website results can also be mixed, when only some parts look generated. Every verdict comes with the signals behind it, a plain-language explanation, the limits of the method, a downloadable PDF report and a button to say the result is wrong.

1. 01Provenance first: C2PA content credentials and EXIF/XMP metadata are checked. If they name a known AI generator, the verdict is probably AI and the reason is stated.
2. 02An open-source detector scores the content.
3. 03When the score is borderline, Claude (Anthropic) reviews the content as a second opinion. It can move the score only a little and never decides alone.
4. 04Claude writes the explanation in English, French or Arabic.

The same three levels apply across the AI image detector, the AI text detector and every other detector, so a result means the same thing whatever you checked.

How should you use AI detection results responsibly?

Use AI detection results to prioritize your attention, then gather independent evidence before you act. A result is an indication, not proof.

1. 01Never sanction a person on a detector result alone.
2. 02Corroborate with evidence of a different kind: drafts, version history, source checks, a conversation.
3. 03Read the signals and the stated limits, not only the label.
4. 04Note what makes the case hard: short, compressed, edited content or a non-native writer.
5. 05Let the person explain, and record their explanation.
6. 06Write a policy in advance that says how results are used, so decisions are consistent.

Schools can adapt these rules from our page for teachers and schools. Code needs extra care, as explained in how to detect AI-generated code.

Keep the report. A result that lists its signals and limits lets a colleague, a reviewer or the person concerned see exactly what the tool found and what it could not see. That transparency protects you as much as it protects them.

How can you test an AI detector yourself?

You test an AI detector by running it on content whose origin you already know, in the conditions you actually face. A short test of your own tells you more than any marketing claim.

• Collect known human samples from your own context: past essays, your own photos, recordings of colleagues.
• Generate a few samples with current tools, then edit some of them the way a real user would.
• Include the hard cases: short texts, screenshots, compressed voice notes, non-native writing.
• Check whether the tool explains its verdict and states its limits, or only shows a number.
• Read its privacy terms: how long uploads are kept and who processes them.

If you are choosing between tools, our comparison of the best AI detectors describes what each one covers, including where another tool is the better fit.

Frequently asked questions