A computer can't read. The chips inside a computer only work with numbers: they add numbers, multiply numbers, compare numbers and move numbers around. Not one chip knows what a word is.

Yet you can type a question to a chatbot and get back a paragraph that sounds like a person wrote it. The programs behind chatbots are called large language models, or LLMs.

A guess needs something to guess from, and a computer only has numbers. So the first problem is this: how do you hand a word to a machine that only understands numbers?

Words become numbers

Everything inside a computer is stored as numbers. A photo is a grid of numbers, one for the colour of each tiny dot. Text is numbers too.

Our machine needs only a simple rule: give every word its own number. The first word gets 1, the next new word gets 2, and so on. A word that shows up again keeps the number it already has.

Real LLMs give numbers to pieces of words, not just whole words, so a model can handle any word, even a new one. The pieces are called tokens. On this page, every token is a whole word.

Now a sentence is a row of numbers. But a row of numbers just sits there. What can a machine do with the row?

Guess the next word

Here is the whole job. Look at the words so far and guess the word that comes next. Add that word to the end. Then look again and guess again.

Real models do something much bigger inside each guess, but the game is the same: predict the next piece of text.

You play this game all day without noticing. Read “Once upon a”, and “time” jumps into your head. Can a machine that can't read play the game as well as you?

Beat the machine

Tap the word you think comes next, then see which word the machine picked and which word was real.

The machine read one short story about a cat and a dog, and nothing else. The machine never saw the six sentences in the game. Even so, the machine got half of them right.

Look at the blue word in each sentence: the last word before the blank. Sentences 1 and 4 both end in “a”, and the machine answered “time” both times. How can one word give the machine a guess?

A machine that counts

The machine counted. Word by word through the story, the machine noted which word came right after each word. Then the machine added up the tallies.

To guess, the machine finds the last word of your sentence in the tally and picks the biggest count. After “the”, the biggest count is “cat”. That count is why the machine wrote “The cat sat on the cat.”

Tap any word in the story and the bars count what came right after it. Then press Edit the text and type your own: a song, a recipe, a text message.

Counting neighbours like this has a name: a bigram model. “Bi” means two, and a bigram is a pair of neighbouring words.

The whole machine is this table of counts. So far the table guesses one word at a time. Can the same table write whole sentences?

Let it write

Writing is guessing over and over. Start with “the” and look up the words that came after “the”. Pick one. The new word is now the last word, so look up the words after the new word, and pick again.

But which word should the machine pick each time? The simplest rule is to always take the biggest count.

The fix is to roll dice. Each word gets a chance that matches its count. “Cat” came after “the” 8 times out of 27, so the dice pick “cat” about 8 times out of every 27 rolls:

Press Next word a few times and watch bigger blocks catch the marker more often. Then switch to Always the top word and watch for a loop.

With the dice, the machine writes sentences like “The dog sat on the cat ate the fish.” Every pair of neighbouring words makes sense, and the whole sentence almost does. The machine has no idea what a cat is. The machine only knows which word tends to come next.

Many chatbots also pick with a bit of chance, which is one reason the same question can get a different answer each time.

Counting pairs gets surprisingly far. So where does counting pairs break?

One word of memory

Read a sentence the way the machine reads one. Cover every word except the last, and guess.

Why not count longer runs, like the last three words? Longer runs do help, for a while. But the number of runs to count explodes. Suppose the machine knows 50,000 words:

No pile of text is big enough to show the machine every run, so most of the table would stay empty. Real LLMs can look back over thousands of words at once, and the newest ones much further. No table of counts could do that.

A table of counts can't fix that blind spot. So how could a machine use the whole sentence?

Ask Phy

• How do real LLMs split text into tokens? Show me how a long word gets split up.
• Here is a short text. Build its bigram table and write me three sentences with it.
• Why does picking with a bit of chance make a chatbot's answers vary?