techtree / Attention / ← prev 21 of 26 next →

Causal masking

Tokens may only look backward, never at the future.

46 seconds

Transcript

A model that writes text writes one token at a time. While it works on a token, the tokens after it do not exist yet.

So each token may look at itself and the tokens before it. Never ahead.

Here are the scores, one row for each token. Every score that looks ahead is replaced with minus infinity.

Then softmax. e to the minus infinity is zero, so those weights vanish. The others still add up to one.

The first token can only look at itself. The last can look at everything.

The weights form a triangle. This is called a causal mask. Each token sees the past, and only the past.

Needs

  • Self-attention video
    Each token's output is a weighted average of the value vectors.
Everything this builds on (12 topics, in order)
  1. Vectors
  2. Adding and scaling vectors
  3. Dot product
  4. Length, angle, and cosine similarity
  5. Matrices
  6. Matrix times vector
  7. Exp and softmax
  8. Tokens
  9. Embeddings
  10. Queries, keys, and values
  11. Attention scores
  12. Self-attention

Leads to