Do AI Detectors Work? How They Work and Why They Get It Wrong

AI detectors guess whether text was written by AI by measuring how predictable the words are (often called perplexity) and how much sentence length and style vary (burstiness). They cannot prove anything: they give a probability, they miss a lot of AI text, and they wrongly flag human writing, especially from non-native English writers. OpenAI withdrew its own AI text classifier in 2023 because of low accuracy, and a Stanford study found detectors misjudged most essays written by non-native English speakers. Use detector scores as a hint, never as evidence on their own.

“But I wrote this myself”

Picture a university student in Sylhet. English is her second language, so she writes carefully: short sentences, standard phrases she learned from textbooks, no slang. She spends a whole weekend on an essay. On Monday her teacher says a detection tool scored it 80% AI.

She has no AI chat history, because she did not use AI. But she also has no easy way to prove a negative. Her clean, careful writing is exactly the kind of text that detectors find “too predictable”.

This is not a rare edge case. It is a known weakness of how these tools work, and anyone who writes, teaches, hires or publishes should understand it.

How AI detectors work

What the evidence says about accuracy

• OpenAI: stopped offering its AI classifier on 20 July 2023, citing its low rate of accuracy.
• Stanford researchers (Liang et al., 2023): tested seven widely used detectors on TOEFL essays written by non-native English speakers. On average 61.3% were falsely labelled as AI-generated.
• Vanderbilt University: disabled Turnitin’s AI detector in August 2023. It noted that even a claimed 1% false positive rate could mean around 750 of its 75,000 yearly papers being wrongly flagged.

Why human writing gets flagged

• Simple vocabulary and textbook phrases, common in second-language writing.
• Very formal, structured writing such as reports, legal text and academic abstracts.
• Short texts, where there is too little signal to judge.
• Lists, definitions and templates, which are predictable by nature.
• Grammar tools that “smooth” text into more predictable wording.

Why AI writing often slips through

• Light human editing changes the statistics enough to confuse detectors.
• Newer AI models write with more variety than the ones detectors were trained on.
• Prompts that ask for a personal style or specific details produce less predictable text.

If you are a writer or student

1. Keep your processSave drafts, notes and version history (Google Docs and Word track this). It is your best evidence.
2. Write with specificsPersonal examples, local details and your own opinions make writing both better and more clearly yours.
3. Be honest about AI useFollow your school’s or client’s rules. If AI helped with grammar or ideas, say so.
4. Ask for a conversationIf falsely flagged, calmly offer your drafts and explain your process. Point to the research above.

If you are a teacher, editor or employer

• Treat a detector score as a reason to talk, not a verdict.
• Ask the writer to explain their argument or reproduce part of the work.
• Look at the writing process: drafts, outlines, in-class samples.
• Set clear policies about what AI help is allowed.

The better goal: writing people trust

Chasing a “0% AI” score is the wrong target. Readers and search engines care whether content is accurate, useful and genuinely helpful. Learn the editing techniques in how to make AI text sound human, see the AI phrases to avoid, and read what Google says about AI content.

Sources

• Inquirer Technology: OpenAI closes low-accuracy AI detector (2023)
• Times Higher Education: AI text detectors biased against non-native English speakers (2023)
• Vanderbilt University: Why we are disabling Turnitin’s AI detector (2023)

Frequently asked questions