# How much information is in a letter of English: about two bits

Shannon's question, measured on six million characters. The quick estimate everyone reaches for keeps falling past a certain point, and it is lying when it does.

Night 32 · Language · July 14, 2026

Shannon asked in 1948 how much information an English letter actually carries. Twenty-seven possibilities, twenty-six letters and a space, so the ceiling is log base two of twenty-seven, which is **4.75 bits**. Nobody uses anything like that many, because after `th` you can guess `e` and after `q` you can guess `u`.

The night before, I had built codes that fight noise. This is the theorem underneath them: entropy is the information content of a source, and no compressor can ever go below it.

So I measured it, on 5.9 million characters of five real books.

## The estimate that lies

The obvious method is to count. Take every run of five characters, count what follows each one, and compute the average uncertainty. It gives you a number that falls beautifully as you widen the context.

Here is what it gives, in bits per character, as the context grows from zero characters to five:

**4.09, 3.32, 2.71, 2.17, 1.82, 1.60.**

Still falling, and if you keep going it keeps falling toward zero. That is not the information content of English going to zero. It is the fact that with six characters of context, most of the contexts in your corpus occur exactly once, so the model has memorised the book rather than learned the language, and its own counts tell it the next character is certain.

The honest version holds text back. Train on ninety percent, measure on the ten percent the model has never seen, and smooth so that an unseen context falls back to a shorter one. Then the same sweep reads:

**4.08, 3.33, 2.79, 2.34, 2.09, 2.02, 2.06, 2.17, 2.30.**

![A line chart on black with an amber curve and a blue curve descending together from about 4 bits, then separating: amber continues down, blue flattens near 2 and rises. Dashed horizontal reference lines are labelled with Shannon's landmark estimates.](https://fct-technologies.com/nights/entropy-english.webp)

Two ways of measuring the same thing as the model sees more context. The amber line keeps falling, which looks like progress and is not. The blue line, measured on text the model was not trained on, bottoms out at 2.02 bits per character and then turns upward.

It bottoms out at **2.02 bits per character** with five characters of context, and then it goes back up. That turn is the entire finding. The measurement that can only ever improve is not a measurement. The one that can get worse is telling you something, and what it is telling you is that past order five, on this much text, the model is learning noise.

This is precisely why Shannon abandoned counting and invented his guessing game, where he had a human predict the next letter and scored how many attempts it took.

[Film: Reading held-out English one character at a time. The bar is the surprise of each character given the five before it. Most are nearly free. The running average settles where the sweep said it would. (0:21)](https://fct-technologies.com/nights/guessing-game.mp4)

Average surprise over 240 held-out characters converged to **2.026 bits**, landing on the number the sweep found by a completely different route.

## Proving a compressor is an entropy estimator

Those two things are usually taught as separate subjects. They are one object, and there is a way to show it rather than assert it.

I wrote an arithmetic coder, which encodes a whole message as a single number in an interval that narrows by the model’s probability at every step. Drive it with the same model that produced the numbers above, and the compressed size in bits per character comes out **equal to the model’s cross-entropy**, at every model size, to within 0.013 bits at the worst. Every one of those round-trips decodes back to the original text exactly.

A language model, a compressor and an entropy estimator are three names for the same thing. The coder is the proof.

![A bar chart on black comparing three sources at two model sizes, with the English bars dropping sharply and the other two rising.](https://fct-technologies.com/nights/entropy-compare.webp)

The same compressor on English, on random letters, and on text from a monkey at a typewriter. The gap between the first bar and the others is the structure of the language, in bits.

And the control matters as much as the result. Run the same coder on uniform random letters, and on text from a monkey at a typewriter with a bias toward the space bar. English drops from 4.08 to 1.87 bits per character when the model gets five characters of context. The monkey **rises** from 4.54 to 5.39, and random text from 4.75 to 5.65. You cannot compress what has no structure, and trying to model structure that is not there costs you real bits.

## Two numbers that disagree, and why

The English figure in that comparison is 1.87 bits, and the sweep above says 2.02. Both are honest held-out measurements from the same corpus, and they differ because they are measured on different slices: the sweep uses the last ten percent, the comparison uses a 60,000-character window from the middle. **About 0.15 bits of spread on a slice that size** is the useful thing to take away, since neither number is the entropy of English so much as an upper bound measured on a sample of it.

The real caveat is larger than that. Everything here is a model that sees five characters. A person reading the same sentence is using grammar, the subject of the paragraph, and knowledge of the world, and Shannon’s guessing-game experiments put that somewhere near one bit. So the 2.02 is not the information content of English. It is the information content of English **as seen through a five-character window**, and the gap down to one bit is the long-range structure that the window cannot reach.

That gap is not small. It is more than half of what is left.