Tokens

A computer cannot read words the way you do. It only works with numbers.

So first, text is chopped into pieces. The pieces are called tokens.

Take this sentence. Here are its tokens.

Common words are usually one token each. A rare word gets split into smaller parts.

The space before a word usually sticks to that word.

There is a fixed list of every possible token, called the vocabulary. It holds tens of thousands of them.

Each token's number is just its place in that list. So the sentence becomes a list of numbers.

The same token always gets the same number. Text goes in. Numbers come out.