techtree / Attention / ← prev 20 of 26 next →
Self-attention
Each token's output is a weighted average of the value vectors.
Transcript
Here are the attention weights for draws, and here is the value vector of each token.
Scale each value by its weight. Plotter's stays long. The others shrink.
Now add them up, tip to tail. The sum is the new vector for draws.
The weights add up to one, so this is a weighted average of the values. It leans toward plotter, the token that draws attends to most.
Every token does this at once, each with its own weights. Each one gets a new vector, blended from the tokens it attends to.
This step is called self-attention: the tokens of one sequence, attending to each other.
Needs
- Attention scores video
Dot each query with every key, then softmax into weights. - Adding and scaling vectors video
Add vectors entry by entry (tip to tail); scale by multiplying every entry.
Everything this builds on (11 topics, in order)
Leads to
- Causal masking video
Tokens may only look backward, never at the future. - Multi-head attention video
Run several small attentions side by side and combine them.