techtree / Linear algebra / ← prev 6 of 26 next →

Matrix times vector

Each output entry is a dot product of one row with the vector.

43 seconds

Transcript

Here is a matrix, and a vector. The matrix has three columns, and the vector has three entries, so they fit.

Take the first row, and dot it with the vector. That gives four, the first entry of the answer.

Now the second row, dotted with the vector: thirteen.

The answer is a new vector: four, thirteen. Three entries went in, two came out, one for each row.

Here is another way to see it. Scale each column by the matching entry of the vector, then add them up. Same answer.

A matrix takes in one vector and gives back another.

Needs

Matrices video
A matrix is a grid of numbers; a stack of vectors.
Dot product video
Multiply matching entries and add them up; one number that measures agreement.
Adding and scaling vectors video
Add vectors entry by entry (tip to tail); scale by multiplying every entry.

Everything this builds on (4 topics, in order)

Matrix times matrix video
Multiply a matrix by many vectors at once.
A neural network layer video
Multiply by a matrix, add a bias, apply a nonlinearity.
Queries, keys, and values video
Three matrices turn each token vector into a query, a key, and a value.
Multi-head attention video
Run several small attentions side by side and combine them.
Next-token prediction and sampling video
Turn the last vector into a score per token, softmax, and pick one.