Softmax Function Explained: Formula, Python Code and Stable Version
The softmax function turns a list of raw scores into probabilities that add up to 1. Learn the formula with a worked example, code it in NumPy, see why the naive version returns NaN and how one subtraction fixes it, and understand temperature and cross-entropy, all with tested output.
- The softmax function exponentiates each score and divides by the total, giving positive values that sum to 1.
- Scores [2.0, 1.0, 0.1] become probabilities [0.659, 0.2424, 0.0986]. The order never changes, so argmax stays the same.
- The naive formula overflowed and returned
nanfor inputs near 1000. Subtracting the maximum first fixed it with no change to the result. - Dividing scores by a temperature controls how confident the output is, the same knob used in LLM sampling.
A neural network that classifies an image as cat, dog or bird does not output probabilities directly. Its last layer produces raw scores called logits, such as 2.0, 1.0 and 0.1, which can be any real numbers. The softmax function converts those scores into a probability distribution you can read, compare and train on. It sits at the end of most classifiers and inside every large language model.
All code below ran on Python 3.13.6 with NumPy 2.5.5, and the printed output is shown as it appeared. No machine learning framework is needed to follow along.
What is the softmax function?
Softmax takes a vector of K real numbers and returns a vector of K probabilities. Every output is between 0 and 1, all outputs add up to exactly 1, and larger inputs always get larger probabilities. It is often described as a “soft” version of argmax: instead of giving all the weight to the biggest score, it shares the weight in proportion to how much bigger each score is.
The softmax formula
For a vector of scores z with K elements, the probability for element i is:
softmax(z)_i = exp(z_i) / ( exp(z_1) + exp(z_2) + ... + exp(z_K) )
In words: raise e to the power of each score, then divide each result by the sum of all of them. The exponential makes every value positive; the division normalises them so they sum to 1.
Worked example by hand
Take the logits [2.0, 1.0, 0.1]:
| Logit z | exp(z) | exp(z) / total |
|---|---|---|
| 2.0 | 7.3891 | 0.6590 |
| 1.0 | 2.7183 | 0.2424 |
| 0.1 | 1.1052 | 0.0986 |
| Total | 11.2125 | 1.0000 |
The first class gets about 66 percent. Notice that a score of 2.0 is only twice 1.0, yet its probability is almost three times larger. Exponentials magnify gaps, which is why softmax outputs often look confident.
Softmax in Python with NumPy
import numpy as np
def softmax_naive(z):
e = np.exp(z)
return e / e.sum()
logits = np.array([2.0, 1.0, 0.1])
p = softmax_naive(logits)
print(p.round(4), p.sum()) # [0.659 0.2424 0.0986] 1.0
print(np.argmax(logits), np.argmax(p)) # 0 0
The code matches the table, the outputs sum to 1, and argmax picks the same class before and after softmax. That last point matters in practice: if you only need the predicted class, you can skip softmax and take the argmax of the logits directly.
Why use the exponential?
Why not just divide each score by the sum of scores? Because logits can be negative or zero, which would give negative “probabilities” or division by zero. The exponential is always positive, it is strictly increasing so the ranking is kept, and its derivative is simple, which makes training with gradient descent clean. It also pairs naturally with the logarithm in the cross-entropy loss, as we will see.
The overflow problem: when softmax returns NaN
Large logits break the naive version:
softmax_naive(np.array([1000.0, 1001.0, 1002.0]))
# RuntimeWarning: overflow encountered in exp
# RuntimeWarning: invalid value encountered in divide
# [nan nan nan]
softmax_naive(np.array([-1000.0, -1001.0]))
# [nan nan]
exp(1000) is far beyond the largest 64-bit float (about 1.8e308), so it becomes infinity, and infinity divided by infinity is NaN. Very negative logits fail the other way: exp(-1000) underflows to 0, and 0 divided by 0 is also NaN. Logits this size are rare in a healthy model, but they do appear with bad initialisation, unscaled attention scores or exploding training, and one NaN can poison a whole training run.
The numerically stable softmax
def softmax(z, axis=-1):
z = np.asarray(z, dtype=float)
e = np.exp(z - z.max(axis=axis, keepdims=True))
return e / e.sum(axis=axis, keepdims=True)
print(softmax([1000, 1001, 1002]).round(4)) # [0.09 0.2447 0.6652]
print(softmax([-1000, -1001]).round(4)) # [0.7311 0.2689]
Subtracting the maximum makes the largest value exactly 0, so the biggest exponential is exp(0) = 1, and nothing can overflow. At least one term in the sum is 1, so the denominator can never be 0 either. Both previously broken inputs now give sensible answers. Every serious library, including PyTorch, TensorFlow and SciPy, uses this trick internally.
Tip: never ship the naive two-line softmax. The stable version costs one extra subtraction and removes a whole class of NaN bugs.
Why subtracting works: shift vs scale
print(softmax(logits + 100).round(4)) # [0.659 0.2424 0.0986]
print(softmax(logits * 2).round(4)) # [0.8638 0.1169 0.0193]
Adding the same constant to every logit leaves softmax unchanged, because the constant becomes a common factor exp(c) that cancels between numerator and denominator. That is exactly why subtracting the maximum is safe. Multiplying the logits is different: doubling them made the top class jump from 66 to 86 percent. Softmax depends on the differences between scores, and scaling stretches those differences.
Temperature: making softmax sharper or flatter
Dividing the logits by a temperature T before softmax controls that stretch deliberately:
| Temperature | Probabilities for [2.0, 1.0, 0.1] |
|---|---|
| 0.5 | [0.8638, 0.1169, 0.0193] |
| 1.0 | [0.6590, 0.2424, 0.0986] |
| 2.0 | [0.5017, 0.3043, 0.1940] |
| 10.0 | [0.3661, 0.3312, 0.3027] |
Low temperature sharpens the distribution toward the top choice; high temperature flattens it toward uniform. In language models this is exactly the temperature setting that trades predictable answers for creative ones, as our LLM temperature guide explains. Temperature is also used in knowledge distillation to expose a teacher model’s softer preferences.
Batches and the axis argument
batch = np.array([[2.0, 1.0, 0.1],
[0.0, 0.0, 0.0],
[5.0, -1.0, 3.0]])
pb = softmax(batch, axis=1)
print(pb.round(4))
# [[0.659 0.2424 0.0986]
# [0.3333 0.3333 0.3333]
# [0.8789 0.0022 0.1189]]
print(pb.sum(axis=1)) # [1. 1. 1.]
print(softmax(batch, axis=0).sum(axis=1).round(4)) # [0.7622 0.2962 1.9417]
Real models process many examples at once, one per row. Softmax must run along the class axis, here axis=1, so each row sums to 1. Equal logits, as in the second row, give a uniform distribution. Using the wrong axis is a silent bug: the code runs but normalises across examples instead, and the row sums came out as 0.76, 0.30 and 1.94. Always assert that rows sum to 1 in tests.
Softmax and cross-entropy loss
y = 0 # the true class
loss = -np.log(p[y])
print(round(float(loss), 4)) # 0.417
# log-softmax computed directly, without exponentiating twice
logsm = logits - logits.max() - np.log(np.exp(logits - logits.max()).sum())
print(logsm.round(4)) # [-0.417 -1.417 -2.317]
Classifiers are trained by minimising cross-entropy: the negative log of the probability assigned to the correct class. A probability of 0.659 gives a loss of 0.417; a perfect prediction gives 0. Libraries compute log-softmax in one stable step instead of taking the log of a softmax output, which avoids log(0). That is why PyTorch’s CrossEntropyLoss expects raw logits, not probabilities. Feeding it softmax outputs applies softmax twice, a common beginner bug.
Softmax vs sigmoid
print(1 / (1 + np.exp(-(2.0 - 1.0)))) # 0.7311
print(softmax([2.0, 1.0])[0]) # 0.7311
With two classes, softmax reduces to the sigmoid of the difference between the logits, and our run gave the same 0.7311 both ways. The practical rule:
- Softmax: exactly one correct class out of many (multi-class), such as cat or dog or bird.
- Sigmoid per output: several labels can be true at once (multi-label), such as an article tagged both Python and testing.
Softmax inside large language models
A large language model ends each step with one logit per token in its vocabulary, which can mean tens of thousands of scores. Softmax turns them into a probability for every possible next token, and sampling picks one. Softmax also appears inside every attention layer, where it turns similarity scores between tokens into weights. Those scores are divided by the square root of the vector size before softmax precisely to keep them from growing too large, the same overflow and saturation concern shown above. Read more in what is a large language model.
How fast is softmax?
We ran the stable NumPy version on a batch of 1,000 rows with 50,257 columns each, the vocabulary size of GPT-2, as 32-bit floats. The best of three runs took 874.9 ms on our machine’s CPU, roughly 0.9 ms per row. GPUs do this far faster, and frameworks fuse softmax with neighbouring operations, but the cost of the exponential over a huge vocabulary is why tricks such as sampled softmax exist for very large output layers.
Final thoughts
The softmax function is a small formula with a big job: it turns any list of scores into clean probabilities. Exponentiate, normalise, and always subtract the maximum first so large logits never become NaN. Apply it along the right axis, pair it with log-softmax for cross-entropy, and use temperature when you need sharper or softer outputs. With those habits, softmax will be the most reliable line in your model. For related maths, see cosine similarity.
Frequently asked questions
What does the softmax function do?
It converts a vector of raw scores, called logits, into probabilities. Each output is between 0 and 1, all outputs sum to 1, and higher scores always receive higher probabilities.
Why does my softmax return NaN?
Large logits overflow when exponentiated, and infinity divided by infinity is NaN. Subtract the maximum logit before calling exp. The result is mathematically identical and numerically stable.
What is the difference between softmax and sigmoid?
Softmax produces one distribution across classes that sums to 1, for problems with exactly one correct class. Sigmoid scores each output independently, for multi-label problems where several labels can be true.
Does softmax change the predicted class?
No. Softmax keeps the order of the scores, so the class with the highest logit also has the highest probability. If you only need the label, argmax of the logits is enough.
What is temperature in softmax?
Temperature is a number that divides the logits before softmax. Values below 1 make the output more confident, and values above 1 make it flatter and closer to uniform.
Keep learning
Need help with testing, WordPress or SEO?
We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.