1. Home
  2. Blog
  3. Artificial Intelligence

Artificial Intelligence

Softmax Function Explained: Formula, Python Code and Stable Version

The softmax function turns a list of raw scores into probabilities that add up to 1. Learn the formula with a worked example, code it in NumPy, see why the naive version returns NaN and how one subtraction fixes it, and understand temperature and cross-entropy, all with tested output.

9 min read
Softmax function explained: formula, Python code and stable version

Key takeaways

  • The softmax function exponentiates each score and divides by the total, giving positive values that sum to 1.
  • Scores [2.0, 1.0, 0.1] become probabilities [0.659, 0.2424, 0.0986]. The order never changes, so argmax stays the same.
  • The naive formula overflowed and returned nan for inputs near 1000. Subtracting the maximum first fixed it with no change to the result.
  • Dividing scores by a temperature controls how confident the output is, the same knob used in LLM sampling.

A neural network that classifies an image as cat, dog or bird does not output probabilities directly. Its last layer produces raw scores called logits, such as 2.0, 1.0 and 0.1, which can be any real numbers. The softmax function converts those scores into a probability distribution you can read, compare and train on. It sits at the end of most classifiers and inside every large language model.

All code below ran on Python 3.13.6 with NumPy 2.5.5, and the printed output is shown as it appeared. No machine learning framework is needed to follow along.

What is the softmax function?

Softmax takes a vector of K real numbers and returns a vector of K probabilities. Every output is between 0 and 1, all outputs add up to exactly 1, and larger inputs always get larger probabilities. It is often described as a “soft” version of argmax: instead of giving all the weight to the biggest score, it shares the weight in proportion to how much bigger each score is.

The softmax formula

For a vector of scores z with K elements, the probability for element i is:

softmax(z)_i = exp(z_i) / ( exp(z_1) + exp(z_2) + ... + exp(z_K) )

In words: raise e to the power of each score, then divide each result by the sum of all of them. The exponential makes every value positive; the division normalises them so they sum to 1.

Worked example by hand

Take the logits [2.0, 1.0, 0.1]:

Logit z exp(z) exp(z) / total
2.0 7.3891 0.6590
1.0 2.7183 0.2424
0.1 1.1052 0.0986
Total 11.2125 1.0000

The first class gets about 66 percent. Notice that a score of 2.0 is only twice 1.0, yet its probability is almost three times larger. Exponentials magnify gaps, which is why softmax outputs often look confident.

Softmax in Python with NumPy

import numpy as np

def softmax_naive(z):
    e = np.exp(z)
    return e / e.sum()

logits = np.array([2.0, 1.0, 0.1])
p = softmax_naive(logits)
print(p.round(4), p.sum())   # [0.659  0.2424 0.0986] 1.0
print(np.argmax(logits), np.argmax(p))   # 0 0

The code matches the table, the outputs sum to 1, and argmax picks the same class before and after softmax. That last point matters in practice: if you only need the predicted class, you can skip softmax and take the argmax of the logits directly.

Why use the exponential?

Why not just divide each score by the sum of scores? Because logits can be negative or zero, which would give negative “probabilities” or division by zero. The exponential is always positive, it is strictly increasing so the ranking is kept, and its derivative is simple, which makes training with gradient descent clean. It also pairs naturally with the logarithm in the cross-entropy loss, as we will see.

The overflow problem: when softmax returns NaN

Large logits break the naive version:

softmax_naive(np.array([1000.0, 1001.0, 1002.0]))
# RuntimeWarning: overflow encountered in exp
# RuntimeWarning: invalid value encountered in divide
# [nan nan nan]

softmax_naive(np.array([-1000.0, -1001.0]))
# [nan nan]

exp(1000) is far beyond the largest 64-bit float (about 1.8e308), so it becomes infinity, and infinity divided by infinity is NaN. Very negative logits fail the other way: exp(-1000) underflows to 0, and 0 divided by 0 is also NaN. Logits this size are rare in a healthy model, but they do appear with bad initialisation, unscaled attention scores or exploding training, and one NaN can poison a whole training run.

The numerically stable softmax

def softmax(z, axis=-1):
    z = np.asarray(z, dtype=float)
    e = np.exp(z - z.max(axis=axis, keepdims=True))
    return e / e.sum(axis=axis, keepdims=True)

print(softmax([1000, 1001, 1002]).round(4))   # [0.09   0.2447 0.6652]
print(softmax([-1000, -1001]).round(4))       # [0.7311 0.2689]

Subtracting the maximum makes the largest value exactly 0, so the biggest exponential is exp(0) = 1, and nothing can overflow. At least one term in the sum is 1, so the denominator can never be 0 either. Both previously broken inputs now give sensible answers. Every serious library, including PyTorch, TensorFlow and SciPy, uses this trick internally.

Tip: never ship the naive two-line softmax. The stable version costs one extra subtraction and removes a whole class of NaN bugs.

Why subtracting works: shift vs scale

print(softmax(logits + 100).round(4))  # [0.659  0.2424 0.0986]
print(softmax(logits * 2).round(4))    # [0.8638 0.1169 0.0193]

Adding the same constant to every logit leaves softmax unchanged, because the constant becomes a common factor exp(c) that cancels between numerator and denominator. That is exactly why subtracting the maximum is safe. Multiplying the logits is different: doubling them made the top class jump from 66 to 86 percent. Softmax depends on the differences between scores, and scaling stretches those differences.

Temperature: making softmax sharper or flatter

Dividing the logits by a temperature T before softmax controls that stretch deliberately:

Temperature Probabilities for [2.0, 1.0, 0.1]
0.5 [0.8638, 0.1169, 0.0193]
1.0 [0.6590, 0.2424, 0.0986]
2.0 [0.5017, 0.3043, 0.1940]
10.0 [0.3661, 0.3312, 0.3027]

Low temperature sharpens the distribution toward the top choice; high temperature flattens it toward uniform. In language models this is exactly the temperature setting that trades predictable answers for creative ones, as our LLM temperature guide explains. Temperature is also used in knowledge distillation to expose a teacher model’s softer preferences.

Batches and the axis argument

batch = np.array([[2.0, 1.0, 0.1],
                  [0.0, 0.0, 0.0],
                  [5.0, -1.0, 3.0]])
pb = softmax(batch, axis=1)
print(pb.round(4))
# [[0.659  0.2424 0.0986]
#  [0.3333 0.3333 0.3333]
#  [0.8789 0.0022 0.1189]]
print(pb.sum(axis=1))                         # [1. 1. 1.]
print(softmax(batch, axis=0).sum(axis=1).round(4))  # [0.7622 0.2962 1.9417]

Real models process many examples at once, one per row. Softmax must run along the class axis, here axis=1, so each row sums to 1. Equal logits, as in the second row, give a uniform distribution. Using the wrong axis is a silent bug: the code runs but normalises across examples instead, and the row sums came out as 0.76, 0.30 and 1.94. Always assert that rows sum to 1 in tests.

Softmax and cross-entropy loss

y = 0   # the true class
loss = -np.log(p[y])
print(round(float(loss), 4))   # 0.417

# log-softmax computed directly, without exponentiating twice
logsm = logits - logits.max() - np.log(np.exp(logits - logits.max()).sum())
print(logsm.round(4))          # [-0.417 -1.417 -2.317]

Classifiers are trained by minimising cross-entropy: the negative log of the probability assigned to the correct class. A probability of 0.659 gives a loss of 0.417; a perfect prediction gives 0. Libraries compute log-softmax in one stable step instead of taking the log of a softmax output, which avoids log(0). That is why PyTorch’s CrossEntropyLoss expects raw logits, not probabilities. Feeding it softmax outputs applies softmax twice, a common beginner bug.

Softmax vs sigmoid

print(1 / (1 + np.exp(-(2.0 - 1.0))))    # 0.7311
print(softmax([2.0, 1.0])[0])            # 0.7311

With two classes, softmax reduces to the sigmoid of the difference between the logits, and our run gave the same 0.7311 both ways. The practical rule:

  • Softmax: exactly one correct class out of many (multi-class), such as cat or dog or bird.
  • Sigmoid per output: several labels can be true at once (multi-label), such as an article tagged both Python and testing.

Softmax inside large language models

A large language model ends each step with one logit per token in its vocabulary, which can mean tens of thousands of scores. Softmax turns them into a probability for every possible next token, and sampling picks one. Softmax also appears inside every attention layer, where it turns similarity scores between tokens into weights. Those scores are divided by the square root of the vector size before softmax precisely to keep them from growing too large, the same overflow and saturation concern shown above. Read more in what is a large language model.

How fast is softmax?

We ran the stable NumPy version on a batch of 1,000 rows with 50,257 columns each, the vocabulary size of GPT-2, as 32-bit floats. The best of three runs took 874.9 ms on our machine’s CPU, roughly 0.9 ms per row. GPUs do this far faster, and frameworks fuse softmax with neighbouring operations, but the cost of the exponential over a huge vocabulary is why tricks such as sampled softmax exist for very large output layers.

Final thoughts

The softmax function is a small formula with a big job: it turns any list of scores into clean probabilities. Exponentiate, normalise, and always subtract the maximum first so large logits never become NaN. Apply it along the right axis, pair it with log-softmax for cross-entropy, and use temperature when you need sharper or softer outputs. With those habits, softmax will be the most reliable line in your model. For related maths, see cosine similarity.

Frequently asked questions

What does the softmax function do?

It converts a vector of raw scores, called logits, into probabilities. Each output is between 0 and 1, all outputs sum to 1, and higher scores always receive higher probabilities.

Why does my softmax return NaN?

Large logits overflow when exponentiated, and infinity divided by infinity is NaN. Subtract the maximum logit before calling exp. The result is mathematically identical and numerically stable.

What is the difference between softmax and sigmoid?

Softmax produces one distribution across classes that sums to 1, for problems with exactly one correct class. Sigmoid scores each output independently, for multi-label problems where several labels can be true.

Does softmax change the predicted class?

No. Softmax keeps the order of the scores, so the class with the highest logit also has the highest probability. If you only need the label, argmax of the logits is enough.

What is temperature in softmax?

Temperature is a number that divides the logits before softmax. Values below 1 make the output more confident, and values above 1 make it flatter and closer to uniform.

Keep learning

Work with Stack Quora

Need help with testing, WordPress or SEO?

We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.

Keep reading

More guides on testing, AI & modern development

Enjoyed this article? Pick a topic and jump straight into our newest tutorials.

Artificial Intelligence

Softmax Function Explained: Formula, Python Code and Stable Version

The softmax function explained: formula, worked example, NumPy code, the stable version that avoids NaN, temperature, cross-entropy and softmax vs…

Read article →
Software Development

CORS Explained: Preflight Requests, Headers and How to Fix CORS Errors

CORS explained: origins, simple and preflight requests, the response headers your server must send, credentials, and how to fix common…

Read article →
DevOps

Git Diff Explained: Staged Changes, Branches and Commits

Git diff explained: read the output, compare unstaged and staged changes, commits and branches, two dots vs three dots, and…

Read article →
Test Automation

Pytest Xdist: Run Tests in Parallel and Cut Test Time

Run pytest tests in parallel with pytest-xdist: install, the -n option, distribution modes, real timings and how to fix shared-state…

Read article →
Software Testing

Verification vs Validation in Software Testing: Example and Table

Verification vs validation in software testing explained with a comparison table and a tested coupon example where every spec test…

Read article →
Databases & SQL

SQL CAST Explained: Convert Data Types With Tested Examples

SQL CAST explained with tested examples: text to number, truncation, integer division, numbers stored as text, dates, bad input and…

Read article →
Databases & SQL

SQL DISTINCT: How to Remove Duplicate Rows (and Its Traps)

SQL DISTINCT explained with tested queries: one and many columns, NULLs, COUNT DISTINCT, DISTINCT vs GROUP BY, the SUM DISTINCT…

Read article →
Software Development

JavaScript Reduce Explained: 12 Practical Examples

JavaScript reduce explained step by step with 12 practical examples: sums, tallies, grouping, Maps, pipelines, sequential promises and the empty…

Read article →
Software Development

Python Random Module: randint, choice, seed and More

The Python random module explained: randint, randrange, choice, choices, sample, shuffle, seed, weights and when to use secrets instead, with…

Read article →
Software Development

Python Slicing Explained: Start, Stop and Step With Examples

Python slicing explained with start, stop and step, negative indexes, reversing, copying and slice assignment, with tested examples and a…

Read article →
Artificial Intelligence

Levenshtein Distance: Formula, Worked Table and Python Code

Levenshtein distance explained: the edit operations, the dynamic programming table for kitten and sitting, Python code with two-row memory, normalised…

Read article →
Software Development

URL Encoding Explained: Spaces, Plus Signs and UTF-8

URL encoding explained with tested Python and JavaScript code: percent encoding, reserved characters, %20 vs plus, UTF-8 bytes, encodeURI vs…

Read article →
DevOps

Gitignore: Patterns, Negation and Fixing Tracked Files

A tested gitignore guide: pattern rules, leading and trailing slashes, negation with !, the ignored parent folder trap, git check-ignore…

Read article →
Software Testing

Acceptance Criteria: Given When Then Examples and Checklist

How to write acceptance criteria testers can run: rule-based and Given When Then formats, a worked discount code example turned…

Read article →
Software Testing

State Transition Testing: Diagram, Table and Example

State transition testing explained with a login lockout example: states, events, the diagram, the state table, valid and invalid transitions,…

Read article →
Databases & SQL

Second Highest Salary in SQL: 5 Tested Ways and the Tie Trap

Find the second highest salary in SQL five ways: MAX subquery, LIMIT OFFSET, DENSE_RANK, a correlated subquery and per department,…

Read article →
Databases & SQL

SQL ROW_NUMBER: Top N per Group, Dedupe and Pagination

SQL ROW_NUMBER explained with tested queries: PARTITION BY, top N per group, deleting duplicates, pagination, the tie trap and how…

Read article →
Software Development

JavaScript Set: Union, Intersection and Removing Duplicates

JavaScript Set explained with tested examples: add, has, delete, deduplicating arrays, the new union and intersection methods, equality rules and…

Read article →
Test Automation

Python Unittest: Assert Methods, setUp and Mocks With Examples

A practical Python unittest tutorial: TestCase, assert methods, setUp and tearDown, subTest, skips, expected failures and mock, with a test…

Read article →
Artificial Intelligence

Jaccard Similarity: Formula, Python, MinHash

Jaccard similarity measures overlap between two sets as intersection over union. Work the formula, compute it in Python for words…

Read article →
DevOps

Git Hooks: Pre-commit and Commit-msg Examples

Git hooks run scripts at key moments such as commit and push. See tested pre-commit, commit-msg and prepare-commit-msg hooks, sharing…

Read article →
Test Automation

API Testing in Python: Pytest + Requests Suite

Build an API testing suite in Python with pytest and requests: fixtures, status codes, headers, JSON schema validation, CRUD flows,…

Read article →
Software Testing

Test Design Techniques: 9 Methods + Examples

Test design techniques help you write fewer tests that catch more bugs. Learn 9 black-box, white-box and experience-based methods with…

Read article →
Databases & SQL

SQL Temporary Table: Create, Use and Drop It

SQL temporary tables store interim results for one session. Learn syntax for SQL Server, PostgreSQL, MySQL, Oracle and SQLite, session…

Read article →
Databases & SQL

SQL CROSS JOIN: Every Combination + Examples

SQL CROSS JOIN returns every combination of rows from two tables. See the row math, product variants, calendar grids that…

Read article →
Artificial Intelligence

TF-IDF Explained: Formula and Worked Example

TF-IDF scores a word by how often it appears in a document and how rare it is across all documents….

Read article →
DevOps

Git Tag: Create, Push and Delete Release Tags

Git tag explained with tested commands: annotated vs lightweight tags, listing and sorting, pushing with follow-tags, deleting local and remote…

Read article →
Test Automation

Pytest Coverage: pytest-cov and Branch Gaps

Measure pytest coverage with pytest-cov: term-missing reports, branch coverage, HTML reports, pyproject settings and a fail-under gate for CI, with…

Read article →
Software Testing

Error Guessing in Testing: Checklist + Demo

Error guessing uses tester experience to predict where code breaks. Get a ready checklist of risky inputs, a step by…

Read article →
Artificial Intelligence

BM25 Explained: How Keyword Search Ranks Results, With Python Code and Hybrid Search

BM25 explained with a pure Python implementation we ran: the formula, IDF, term saturation, the k1 and b parameters, why…

Read article →
DevOps

Git Bisect Explained: Find the Commit That Broke Your Code, Manually or Automatically

Git bisect explained with a tested repository: find the one bad commit among 41 in 5 test runs, automate it…

Read article →
Test Automation

pytest Markers Explained: Custom Marks, skip, skipif, xfail and -m Filters

pytest markers explained with a tested suite: register custom marks, run only smoke tests with -m, combine filters with and,…

Read article →
Software Testing

Positive vs Negative Testing: Differences, 24 Real Test Cases and a Checklist

Positive vs negative testing explained with a real signup form: 6 positive and 18 negative test cases run in pytest,…

Read article →
Artificial Intelligence

Cosine Similarity Explained: Formula, Worked Example and Python Code

Cosine similarity explained simply: the formula, a worked example by hand, pure Python and NumPy code, cosine vs Euclidean distance…

Read article →
DevOps

Git Squash Commits: 4 Easy Ways, Tested Step by Step (rebase -i, reset, merge)

Git squash commits explained with 4 tested methods: interactive rebase, reset –soft, merge –squash and fixup with autosquash. Before-and-after logs,…

Read article →
Test Automation

pytest parametrize Explained: 7 Patterns with Tested Examples

pytest parametrize explained with 7 tested patterns: basic cases, readable ids, stacked decorators, pytest.param with xfail and skip, indirect fixtures…

Read article →