AI fiction and human fiction split on narrative structure, not prose

The question that survives deleting the prose

Machine-text detectors read the surface. They key on word choice and punctuation: the em-dash, the reach for “delve” and “tapestry,” the small syntactic regularities a model leaves behind. That signal works until the model changes its habits. StoryScope opens on the reason to distrust it. GPT-5.4 cut its em-dash use, and the paper reports that fine-tuning a model to imitate human style drops one creative-writing detector from 97% to 3%,¹ drawing on work by Chakrabarty and colleagues in which readers come to prefer the fine-tuned prose and its machine signature nearly disappears.²

So the paper withholds style and asks what is left. Its subject is discourse-level narrative choice: how a story is built above the sentence, in its timeline, its handling of theme, the agency it gives a protagonist, the way it stages emotion. The hypothesis is that humans and machines make systematically different choices at this level, and that those choices persist after every lexical tell is stripped away.¹ The bet behind the bet is that structure is harder to edit than diction: changing a timeline or a resolution is a rewrite, not a find-and-replace.¹

From a story to three hundred questions

StoryScope does not hand-pick the features it will measure. It induces them, and the induction is its own multi-stage machine. The scaffold is NarraBench, a narrative-analysis taxonomy of twelve aspects from Hamilton and colleagues; StoryScope adopts ten of them (agent, social network, event, plot, structure, setting, time, revelation, perspective, style) and drops paratext and motivation as out of scope.³ The pipeline turns those ten dimensions into a bank of concrete, discrete questions asked of every story.

1. Template each story

   GPT-5.1 rewrites every story as a structured JSON template along the ten dimensions, forcing later stages to reason over narrative content rather than prose. An ablation confirms the step matters: a template pipeline and a raw-text pipeline share only 6 of their top-20 features, and the raw-text set is style-heavy where the template set is structure-heavy.¹

2. Compare across sources

   A held-out discovery pool of 600 stories over 100 prompts goes to GPT-5.1 in a high-reasoning mode, which writes per-source and cross-source observations about how the six authors differ.¹

3. Propose and prune features

   Ten expert prompts, one per dimension, ask GPT-5.1 to turn those observations into closed-form questions with discrete answers, constrained to five response types. Three runs yield 408 candidates; embedding-based deduplication merges near-identical questions down to 304 final features, a mix of categorical, ordinal, scale, binary, and multi-select.¹

A separate model does the measuring. Gemini 3 Flash scores all 304 features on each of the 61,608 stories, and the scoring is reproducible: inter-run agreement reaches Krippendorff α = 0.90, and on a 240-item validation set the model agrees with human annotators at Cohen κ = 0.84, slightly above the human-to-human κ of 0.74.¹ The corpus itself is parallel by construction. For each of 10,272 human short stories drawn from Books3, one of the book corpora in the Pile,⁴ the authors ask a model to infer the writing prompt, then hand that same prompt to all five LLMs, so plot and premise are matched across the human version and its five machine mirrors. The released artifacts include the code, the 10,272 prompts, 51,336 AI narratives, and the per-story features; the human stories are withheld for copyright.⁵

One tell, extracted by hand

A single feature, walked on one passage, teaches what the corpus table only confirms. One of the thirty core features asks a binary question of each story: does the narrator explicitly state the theme? The extraction is an operation on text, not a vibe. A story that closes on a line naming its own lesson answers yes. A story that leaves the same understanding for the reader to assemble answers no. The rule is coarse enough that two readers, or a reader and a model, will usually agree on the call, which is why it can be scored at scale.

Run it across the corpus and the two distributions pull apart in one direction.

That is the shape of every tell that follows. Each is a question with a discrete answer, each separates the human and machine distributions by some margin, and none is a rule that holds on any one story. The detector's power comes from stacking many weak, legible signals, not from a single giveaway.

The machine's tidier defaults

The core features sort into a few consistent habits, and each is a signed gap between the human mean and the AI mean rather than a verdict on a story. Read together they describe a single disposition, which the paper calls over-determination: the machine leaves less for the reader to do.

The theme is over-explained. Beyond the 77%-to-52% gap on stating it outright, AI stories score higher on thematic explicitness and moralizing (3.94 vs 3.28 on a five-point scale, the largest single scale gap), and stage dialogue as philosophical debate more often (59% vs 34%).¹ The structure is streamlined. AI stories resolve through the protagonist's own agency (69% vs 46%), end on internal acceptance (47% vs 27%), and run without subplots (79% vs 57%), while human stories carry more temporal complexity, scoring higher on chronological discontinuity (2.40 vs 2.12) and nonlinear framing.¹

The body is over-written. The widest gap in the whole feature set is emotion conveyed through embodied metaphor, present in 81% of AI stories against 38% of human ones; the mirror-image gap is that human authors are three times as likely to name an emotion plainly, calling a character afraid rather than tightening a throat.¹ And the human stories reach outside themselves. They name real texts nearly twice as often (47% vs 24%), address the reader directly more often, and give the protagonist a morally ambivalent turn the machine smooths over.¹ The throughline is not that any one of these is decisive. It is that the machine consistently chooses the tidier option.

Ninety-three percent, and the detectors that beat it

Fed to an XGBoost classifier, the 257 narrative features that survive a style audit separate human from AI at 93.2% macro-F1 on a held-out test set of 8,301 stories, with prompts grouped across splits so no story leaks between train and test.¹ Macro-F1 is the class-balanced average of precision and recall, so a model cannot win by always guessing the larger class. Adding the stylistic features back lifts the score only to 96.0%, meaning the narrative-only model recovers over 97% of the full model's performance from structure alone.¹

The honest comparison is less flattering, and the paper makes it itself. Trained classifiers that read the raw text nearly separate the corpus perfectly.

ModernBERT on raw text reaches 99.9%, a stylometric classifier 99.8%, and a TF-IDF model 99.7%.¹ The narrative model does not beat any of them. It loses roughly seven points to the surface detectors it is meant to complement, and its claim on attention is not raw catch rate. What it offers instead is a decomposition a person can read and, the paper argues, a signal that outlasts the style edits that would defeat the others. Against that floor, the zero-shot detectors fare worst of all. Binoculars, which reads how surprising a passage is to one model against another, manages only 55.9% on these long stories, barely above chance.⁶ Its family assumes that longer passages give a surprise-based detector more to work with,⁷ an assumption that plainly does not carry over to this fiction.

The durability claim gets one real test. The authors run LAMP, a span-level rewriter built from professional editors' fixes to machine prose, which strips seven AI artifact categories such as cliche and unnecessary exposition,⁸ over 278 Gemini stories, then rescore. The narrative detector barely moves, from 95.5% to 93.9% macro-F1, a drop of 1.6 points, because the features it reads are orthogonal to the surface artifacts LAMP removes.¹ Editing the prose does not touch the plot, so the structure detector keeps working where a stylometric one would degrade.

Five models, one narrow region

The same features that separate human from machine also tell the machines apart, though less cleanly. Six-way authorship attribution, across the human author and all five models, reaches 68.4% macro-F1 from narrative features alone against a 16.7% chance baseline, and 77.3% once style is added back.¹ The per-class scores show where the signal thins. Narrative-only per-class F1 runs 0.89 for human and 0.77 for Claude, then falls to 0.60 for Gemini, 0.57 for DeepSeek, and 0.55 for Kimi, which sit in a confused cluster the classifier struggles to pull apart.¹

The models that are separable have signatures the paper reads off as fingerprint features, meaning a feature whose weight concentrates on one source.

The counts are lopsided. Claude carries 26 fingerprint features and GPT and Gemini eleven each, while Kimi has only three and sits, in the paper's phrase, at the generic center of the AI distribution.¹ GPT reaches for gossip as a plot mechanism (64% vs 44 to 55% for the others) and subverts expectations more than its peers; Gemini defaults to bleak settings (88% tagged bleak or oppressive).¹ Stand back and the five converge. Their centroids sit close together and far from the human one: the mean human-to-AI distance is 1.6 times the mean AI-to-AI distance (6.6 vs 4.3), and even the closest human-AI pair is farther apart than the most distant pair of models.¹ Human stories are also rarer in the feature space, landing at the 0.71 mean rarity percentile against the AI 0.49, and being the rarest of the six versions of a prompt 57.8% of the time where chance is 16.7%.¹

What a determined rewrite would erase

The paper's confound work is unusually careful, and it earns the benefit of the doubt on the obvious objections. Human stories run longer, but a length-only classifier scores just 55.9%, and on a length-matched subset the narrative model is unchanged at 93.2%, so length is not the tell.¹ Memorization is bounded: an n-gram audit flags 0.70% of generations as high-overlap with likely training text, and dropping those prompts shifts every headline number by at most 0.3 points.¹ Topic is ruled out by a Kruskal-Wallis test across six subject categories that finds no significant difference (p = 0.46).¹

The weaker point is the one the paper leans on hardest. Its case for narrative features over lexical ones is that structure is durable, and the motivating threat is the fine-tuning that collapses style detection from 97% to 3%. But that threat is never run against the narrative model. The only robustness test is LAMP, which by design rewrites style and leaves structure alone, so passing it shows the features are orthogonal to style, not that they survive an adversary who targets structure.¹ A writer who deliberately fragments a timeline, withholds the theme, and roughens a resolution is exactly the attacker the framing invokes and the experiments never build. Durability against style is measured. Durability against structural editing is a hypothesis wearing the measured result's clothes.

Two further limits bound how far the numbers travel. The whole study runs over five specific 2026 models, so the tight AI cluster and the attribution scores describe today's systems and carry no promise about the next ones, which could diverge as easily as converge. And the pipeline is circular in a way high agreement mitigates but does not erase: a class of LLMs writes the stories, infers the prompts, templates the narratives, proposes the features, and scores them, so the tells are what one model judges about another's output.¹ The parallel corpus adds its own asymmetry. The AI stories are fresh completions of an inferred prompt, while the human stories are pre-existing published fiction, so the contrast holds plot and premise fixed but not era, editorial polish, or the difference between writing freely and writing to a spec.⁵