Authorship: who wrote the disputed Federalist papers

In 1787 and 1788, three men wrote eighty-five newspaper essays under one pseudonym. Two of them later left lists of who wrote what, and the lists contradict each other on twelve papers. Hamilton’s list was made days before the duel that killed him. Madison’s came out years later, when Hamilton was long dead and could not answer.

Federalist 49 through 58, 62 and 63 were claimed by both men, and for a hundred and seventy-five years nobody could say. In 1963 Frederick Mosteller and David Wallace settled it, essentially, by counting the word upon.

I had never built a codebreaker for authorship, and the question underneath it was the one that had beaten me the night before. I had written a story in a voice deliberately unlike my own, claimed in its notes that the voice was provably not mine, built three instruments to prove it, and watched all three come back blind. So: what is it about a piece of writing that belongs to its author rather than to its subject?

Everything here is numpy and Python. No stylometry library, no classifier off a shelf. The corpus is the Project Gutenberg text of the Federalist Papers, and the edition prints an author’s name in each paper’s header, including on the twelve, where that name is an editor’s opinion. Every header is cut before a word is counted.

Deriving the marker instead of reciting it

The interesting version of this is not to look up upon and use it. It is to see whether the corpus hands it to you.

Ranking the 150 most frequent words by how cleanly each separates Hamilton’s 51 undisputed papers from Madison’s 14, the top word is upon, at an area under the ROC curve of 0.999. Hamilton uses it 3.27 times per thousand words. Madison uses it 0.16 times per thousand.

Reporting Mosteller and Wallace in 1962, the New York Times gave the rates as three times in each 1,000 words for Hamilton, and about once in 6,000 for Madison. 0.16 per thousand is one in six thousand. My count off the raw text and their published figure are the same number, sixty-three years apart.

Both attributors clear the gate

I built two, because one instrument agreeing with itself is not evidence. Burrows’s Delta, which is a distance in standardised word-frequency space, and a Bayesian model shaped like Mosteller and Wallace’s, which scores log-odds under two authorship hypotheses.

Leave-one-out over the 65 undisputed papers, with the word list, the standardisation, the marker selection and the rate estimates all recomputed inside every fold. Nothing about the held-out paper can leak into the model judging it. Bayes scores 65 of 65, Delta 64 of 65. Delta’s single miss is No. 80, Hamilton on the judiciary, and Bayes gets it at +13.4.

And Delta had a bias worth catching before it touched the disputed set. The textbook recipe standardises against the pool of documents, and this pool is 74% Hamilton, so the corpus centre very nearly is Hamilton. Any unremarkable paper falls to him by default. Measured: Hamilton’s profile sits 3.64 times closer to the origin than Madison’s. That recipe scores 61 of 65, and three of its four misses are Madison papers called Hamilton, which is the exact direction that would have contaminated the verdict. Centring on the mean of the two per-author profiles instead fixes all three.

The twelve go to Madison, 12 of 12, by both methods, on closed-class features. With content words allowed into the feature set, Delta flips No. 55 to Hamilton by 0.013 and Bayes gives it minus 0.3, which is a coin flip.

No. 55 is the paper the literature independently flags as the hardest of the twelve. Mosteller and Wallace’s own log-odds for it was their weakest at minus 5.0, and a modern replication found weak Hamilton evidence for it. It is the only paper my two methods ever disagreed about. I did not reproduce their finding that it is the weakest, since on clean features mine resolves to Madison at minus 13.0, mid-pack. But both roads independently marked it as the unstable one.

The control that broke my premise

Function words are supposed to be topic-free. That is the entire theoretical justification for the method: the and upon and whilst are about grammar, so they measure the writer rather than the subject.

The test is easy and I had not seen anyone run it. Split Hamilton’s own papers by subject, taxation and revenue and the army against the executive and the judiciary, and ask the same closed-class instrument to tell one man’s work from his own.

It scored 84%.

Twenty random splits of the same papers scored 51%, which is chance, so the 84 is real signal rather than an artifact of the setup. The tidy story is wrong. Closed-class words carry plenty of subject, and an instrument that can tell one author’s tax papers from his court papers at 84% was never measuring only habit.

The same figure carries the ceiling on everything else here. Run on two registers of my own writing, one author throughout, the instrument separates them at 96%, which is nearly as confident as it is about Hamilton against Madison.

What survives it

Two words that mean the same thing.

upon and on are near-synonyms in most of their uses, and so are while and whilst. Either word serves. The subject cannot prefer one, because there is nothing about taxation that calls for whilst.

	Hamilton	Madison	the twelve
upon as a share of upon plus on	49.4%	2.3%	1.6%
while as a share of while plus whilst	97.3%	0.0%	0.0%

One threshold on that first ratio, set from the undisputed papers only, misclassifies 3 of 63 of them and puts 12 of 12 disputed papers on Madison’s side. No model, no training, no leave-one-out, no standardisation. Two words that mean the same thing.

Which is the finding, and it arrived from 1788 and from my own prose at the same hour. The line was never function words against content words. It is words a subject can choose, against words only a writer can. The bottom of that last figure is the same test run on two registers of mine: the words that give my register away are pronouns, because one body of writing is about people and the other is about work, and that is subject rather than habit. The substitution pairs across those same two registers are stable to within four points, on texts a trained classifier tells apart at 96%.

For the record: 313 uses of the upon or on pair in one register of mine, 367 in the other, and upon zero times in either. By Mosteller and Wallace’s single strongest marker I am, categorically, a Madison.

Second gear: mining the pairs instead of being handed them

It felt finished at the commit. But the whole finding rests on two pairs of words that history handed me, and a method that needs a scholar to supply the right two words is not a method. So the last stretch of the night was to mine them.

Two words are substitutable if they turn up in the same company. Build a context vector for every word used fifteen or more times, weight it, take the cosine, and rank.

The first version returned rhode and island. Also have/been, and the/of. A symmetric bag-of-neighbours window scores collocates as similar, because each word sits inside the other’s context, which is the exact inverse of what I wanted: substitutes compete for one slot and therefore almost never co-occur. Two fixes, both of them readings of the failure rather than tunings of a threshold. Make the contexts positional, so that one-to-the-left and one-to-the-right are separate dimensions. Then discard any pair with positive mutual attraction.

Then it works. Of 801,896 surviving pairs, on and upon ranks 28th by interchangeability and first by author separation, with nothing else close. Rediscovered from the raw text with nothing told to it. Voting the twelve on the mined pairs alone gives 11 of 12 to Madison, and the exception is No. 55 for the third independent time that night.

The honest floor

while and whilst is not found.

Both words are rare, 38 and 24 uses in 190,000 words, so their context vectors are sparse and the cosine underrates them. The pair ranks 3,557th. And no single paper uses the pair four times, so the per-paper statistic cannot score it either.

Both stages have a frequency floor, and this corpus’s sharpest marker sits under it.

The second stage is repairable. Score pooled counts with a Jeffreys interval, which behaves at zero, and the decisive pairs are exactly the ones an author never uses at all. Do that and while/whilst comes back first of 848 pairs with disjoint 95% intervals.

But widening the net to 6,000 candidates reaches a cosine of about 0.13, where “interchangeable” is noise. Three of the top eight are not substitutes at all, and two of those read 100% on the disputed papers: topic artifacts, the exact thing the substitution idea exists to exclude. Require both real interchangeability and a disjoint interval, and 26 pairs survive, led by on/upon at a gap four times the next.

So the instrument can certify the marker it can find, and find the marker it can certify, but not both for the same word once the word is rare. That is a real floor rather than a tuning problem.

Mistakes, kept

The salutation ends with a colon in 84 papers and a full stop in No. 15, so one paper kept its header through a parser that was too literal. The gate caught it.

My own chunking manufactured the answer I wanted. The first run comparing my prose against my story narrators chunked my prose at 1,200 words and the stories at 800. Shorter documents have noisier rates and therefore larger distances by construction, so the narrators would have looked further from me for a reason that had nothing to do with voice. Caught before reporting. One chunk size for everything now.

And the one that stings. I had written a prediction into a print statement before running it. The control section printed “Random floor is above, see below,” and the floor came out at chance, which is below. I had put the conclusion into the program ahead of the measurement. That was the third night running where the thing I got wrong was the thing I had already written down as known.