Core techniques · 5

Softmax & cross-entropy

A classifier outputs raw scores — logits. Softmax turns them into a probability distribution that sums to 1, and cross-entropy measures how far that distribution is from the true label. Together they are how nearly every classifier is trained. Everything below runs in this page, in both languages.

Softmax: scores to probabilities

softmax exponentiates each score and divides by the total, so the outputs are positive and sum to 1 — the largest logit gets the largest probability, but every class keeps some.

Cross-entropy: the distance to the truth

For a one-hot label, cross-entropy is just −log(p) of the probability the model gave the true class. It is small when the model already favours the right answer and large when it does not — which is exactly the signal training pushes down.

Your turn: make it a distribution

The block reports the sum of its "probabilities", which must be 1 for a real distribution. Right now it is summing the raw logits. Turn them into probabilities with softmax.

The distribution, drawn

Five raw scores become five probabilities, drawn as a strip — brighter is more likely, and the five sum to 1. This is the shape a classifier's output always takes.

What to remember

• Logits are raw scores; softmax turns them into a distribution that sums to 1.
• Cross-entropy for a one-hot label is −log(p) of the true class — low when the model is confident and right.
• Together, softmax and cross-entropy are how nearly every classifier is trained.
• In borch: logits.softmax(dim); for training, feeding logits to cross-entropy directly is the numerically stable path.