Exp and softmax

Here is a vector of scores. Two, one, zero, minus one.

We want to turn them into weights: all positive, and adding up to one.

A score can be negative, so first, make everything positive.

Take a number called e. It is about two point seven. Raise it to the power of each score.

That gives about seven point four, two point seven, one, and zero point four.

All positive. And the biggest score pulls ahead.

Add them up: about eleven and a half. Then divide each one by that total.

Now they add up to one. Point six four, point two four, point zero nine, point zero three.

The order stays the same: a bigger score gets a bigger weight.

That is softmax. Raise e to each score, then divide by the total.