techtree / Transformer / ← prev 23 of 26 next →
The transformer block
Attention then MLP, each wrapped in normalization and a residual.
Transcript
A transformer block puts the pieces together. In come the token vectors, with their positions added.
First, normalize each vector. Then multi-head attention, with the causal mask, lets the tokens share information. Its result is added back to the input.
Normalize again. Then the MLP works on each token's vector by itself, and its result is added back too.
Attention moves information between tokens. The MLP works within each token.
Out come the same number of vectors, each the same size as before, each one edited.
Normalize, attend, add. Normalize, MLP, add. That is one block.
Needs
- Multi-head attention video
Run several small attentions side by side and combine them. - Causal masking video
Tokens may only look backward, never at the future. - MLP video
Two layers in a row; widen, bend, narrow. - Residual connections video
Add a block's output back to its input, so each block makes an edit. - Mean, variance, and normalization video
Shift and scale a vector so its entries have mean 0 and spread 1. - Position information video
Add a position vector to each token vector so order is not lost.
Everything this builds on (20 topics, in order)
- Vectors
- Adding and scaling vectors
- Dot product
- Length, angle, and cosine similarity
- Matrices
- Matrix times vector
- Exp and softmax
- Nonlinearity (ReLU)
- Mean, variance, and normalization
- Tokens
- Embeddings
- Position information
- A neural network layer
- MLP
- Residual connections
- Queries, keys, and values
- Attention scores
- Self-attention
- Causal masking
- Multi-head attention
Leads to
- Stacking blocks video
The same block design, repeated many times, each with its own weights.