How AI Content Detectors Work (and Why They Get It Wrong)

Our own detector used to be wrong in the most embarrassing way possible. We fed it a paragraph written by a current language model and it returned 14 out of 100 — "Likely human". Then we fed it a paragraph a person actually wrote, and got 13. One point apart. The tool was not detecting anything; it was mostly checking whether the text contained the word "furthermore".

That failure is worth explaining, because the reasons behind it are the same reasons every detector — ours, the paid ones, the one your school bought — is less reliable than its marketing suggests.

Perplexity: how predictable are you?

The signal most commercial detectors lean on is perplexity: how surprising a text is to a language model. LLMs are prediction machines. Given the words so far, they know what usually comes next, and generated text tends to walk those high-probability paths because that is literally how it was produced. Humans wander. We write odd metaphors, pivot mid-paragraph, drop in strangely specific memories.

Try it yourself: "The chef seasoned the soup with…" A model expects salt or pepper. A person writes "a childhood memory of her grandmother's kitchen." Statistically bizarre, unmistakably human. Detectors average that surprise across a whole text: low perplexity reads as predictable, which reads as machine-made. Note the word "reads" — we will come back to it.

Burstiness: the rhythm thing

The second classic signal is how much sentence length varies. Humans write in bursts. A long winding sentence stuffed with clauses, and then a short one. Like that. Default model output marches at an even tempo: similar lengths, similar connective glue, paragraph after paragraph. The flatness is measurable even when every sentence is perfectly correct.

What a browser-based detector can actually do

Here is the honest limitation nobody advertises. Computing real perplexity means running a language model over your text. That is a server, a GPU bill, and your draft leaving your machine. A tool that runs entirely in your browser — like ours — cannot do it, and any browser tool claiming a "perplexity score" is describing something else.

So ours measures the traces instead of the cause. Seven signals: stock phrasing, sentence-length variation, informal human markers, em-dash and colon density, vocabulary variety, repeated sentence openings, and the three-item list habit. Each is weaker than perplexity on its own. Together they catch a useful share of generated text without a single byte leaving your device.

The mistake that made ours useless

The 14-out-of-100 failure had a specific cause, and it is a good illustration of how these tools go wrong quietly. Every signal was scored on a scale where 0 meant human and 100 meant AI, and the final number was their weighted average. Sounds reasonable. It is not: a signal that finds nothing scores 0, which is an active vote for "human". Six mostly-silent signals could outvote two that were screaming.

The fix was to put "no evidence" in the middle at 50 and accumulate evidence rather than average it, so quiet signals contribute nothing instead of contributing innocence. Two other bugs surfaced while we were measuring. The punctuation signal ran backwards: it had been written when models used plain commas and periods, so it scored heavy em-dash use as human — the opposite of what current output looks like. And the vocabulary signal was really measuring text length, because unique-words-over-total-words always looks high in a short passage.

One more attempt failed instructively. We tried counting contractions as evidence of a human author. It backfired: current models write "isn't" and "you're" as readily as anyone, so generated prose started scoring more human than before. Only genuine sloppiness survived as a signal — a lowercase "i", a trailing ellipsis, conversational filler.

Why innocent people get flagged

Some honest human writing is predictable by design. Legal boilerplate. Technical documentation. Academic writing in a second language, where you reasonably reach for common words and safe structures. Corporate template reports. All of it reads flat to a detector, and all of it gets flagged.

The pattern is not random either: non-native English writers get flagged more often. That bias is documented, and in a classroom it lands on exactly the students least equipped to argue back.

The guilty walk free too

It cuts both ways just as hard. Generated text that a person has edited, or that was produced with high-randomness settings, or that was simply asked to vary its rhythm, sails straight through. Generators improve every month while detectors chase the statistical footprints of last year's models. Both error directions are alive and well.

Using a detector without hurting anyone

Treat a score like a smoke alarm, not a verdict. It tells you where to look, never what happened. Give it enough text — our AI Text Detector needs 80 words at minimum and works noticeably better above 200, and it marks signals as unavailable rather than guessing when a sample is too short. Read the signal breakdown instead of the headline number, and never let a detector be the only evidence in a decision affecting someone's grade or job.

A middle score genuinely means undecided. Ours says "Inconclusive" between 40 and 60 rather than rounding toward an accusation, because for carefully edited text that is the truthful answer.

If it happens to you

Keep your drafts. A versioned history with timestamps is the strongest defence there is, because generated text tends to arrive in one block while human work accumulates. Point out that detector vendors themselves warn against punitive use. And write more like yourself: vary your sentence lengths, choose the precise word over the expected one. Distinctive writing is the best proof of authorship there has ever been.