techtree / Transformer / ← prev 24 of 26 next →

Stacking blocks

The same block design, repeated many times, each with its own weights.

45 seconds

Transcript

One block makes one round of edits to the token vectors. A transformer stacks many blocks, one on top of the next. What comes out of one block goes into the next.

Every block has the same design, but its own weights: its own attention matrices, and its own MLP.

The residual stream runs up through all of them. Each block reads the vectors, and adds its edit.

Small models have about a dozen blocks. The largest have around a hundred.

Early blocks tend to pick up simple patterns, like which words sit nearby. Later blocks build on those, toward meaning.

One design, stacked many times, each copy with its own weights.

Needs

Everything this builds on (21 topics, in order)
  1. Vectors
  2. Adding and scaling vectors
  3. Dot product
  4. Length, angle, and cosine similarity
  5. Matrices
  6. Matrix times vector
  7. Exp and softmax
  8. Nonlinearity (ReLU)
  9. Mean, variance, and normalization
  10. Tokens
  11. Embeddings
  12. Position information
  13. A neural network layer
  14. MLP
  15. Residual connections
  16. Queries, keys, and values
  17. Attention scores
  18. Self-attention
  19. Causal masking
  20. Multi-head attention
  21. The transformer block

Leads to