techtree / Building blocks / ← prev 17 of 26 next →

Residual connections

Add a block's output back to its input, so each block makes an edit.

40 seconds

Transcript

Here is a block: a layer that takes in a vector and gives one back. On its own, its output replaces the input.

A residual connection adds a second path, around the block. The input is added to the block's output.

Here is the input, as an arrow. The block's output is a small arrow. Add it tip to tail, and you get the input, nudged.

So the block does not rewrite the vector. It makes an edit.

If the block gives back all zeros, the input passes through unchanged. Doing nothing is easy.

The input always has a clear way through.

Needs

A neural network layer video
Multiply by a matrix, add a bias, apply a nonlinearity.
Adding and scaling vectors video
Add vectors entry by entry (tip to tail); scale by multiplying every entry.

Everything this builds on (7 topics, in order)

The transformer block video
Attention then MLP, each wrapped in normalization and a residual.