techtree / Transformer / ← prev 26 of 26 next →

What is a transformer?

The whole machine, end to end.

45 seconds

Transcript

A transformer takes in text, chops it into tokens, and looks up a vector for each one, with its position added.

The vectors rise through a stack of blocks. In each block, attention lets the tokens share information, and an MLP works on each token by itself.

At the top, the last vector becomes a score for every token in the vocabulary. Softmax makes them probabilities, and one token is picked.

It is added to the text, and the whole machine runs again.

All of its skill lives in its weights: the embedding table, the attention matrices, and the MLPs.

Text in. One token out. Again and again. That is a transformer.

Needs

Everything this builds on (23 topics, in order)
  1. Vectors
  2. Adding and scaling vectors
  3. Dot product
  4. Length, angle, and cosine similarity
  5. Matrices
  6. Matrix times vector
  7. Exp and softmax
  8. Nonlinearity (ReLU)
  9. Mean, variance, and normalization
  10. Tokens
  11. Embeddings
  12. Position information
  13. A neural network layer
  14. MLP
  15. Residual connections
  16. Queries, keys, and values
  17. Attention scores
  18. Self-attention
  19. Causal masking
  20. Multi-head attention
  21. The transformer block
  22. Stacking blocks
  23. Next-token prediction and sampling

Leads to

This is the top of the tree.