MLP

An MLP, a multi-layer perceptron, is two layers in a row. It bends: with enough width, its output can follow almost any curve. Here is how. A vector goes in.

The first layer makes it wider: two entries become four. Then ReLU zeroes the negative one.

The second layer makes it narrow again, back to two entries. This one usually has no ReLU at the end.

So: widen, bend, narrow.

What do the bends buy you? Feed in a single number, and plot the single number that comes out.

Each neuron in the wide middle can add one bend to the line. Two neurons, two bends. Three, three.

With enough of them, the output can follow almost any curve.

That is the job of the wide middle.