Embeddings

A sentence has become a list of token IDs. But an ID is only a name tag. The number itself means nothing.

So the model keeps a table, with one row for each token in the vocabulary. Each row is a vector.

To embed a token, use its ID to find its row. That vector is the token's embedding.

Each token in the sentence gets its own vector.

Nobody writes these numbers by hand. They are tuned, so that tokens with similar meanings get vectors that point in similar directions.

Real embeddings have hundreds of entries. These made-up ones have just two, so we can draw them.

Cat and dog point almost the same way. Their cosine similarity is point nine eight. Cat and truck point far apart: point two one.

The ID says which token it is. The embedding says what it is like.