1. Home
  2. Blog
  3. Artificial Intelligence

Artificial Intelligence

Confusion Matrix Explained: Precision, Recall, F1 and Thresholds in Python

A confusion matrix shows exactly how a classifier is right and wrong: true positives, false positives, false negatives and true negatives. We built one by hand in Python for a fraud detector, derived accuracy, precision, recall, specificity and F1, showed why a model that never predicts fraud beats it on accuracy, and chose a threshold by business cost.

9 min read
Confusion matrix explained: precision, recall, F1 and thresholds in Python

Key takeaways

  • A confusion matrix counts predictions in four cells: TP, FP, FN and TN; every common classification metric is built from them.
  • Our fraud model caught 79 of 85 frauds (recall 0.929) but only 46 percent of its alerts were real (precision 0.459).
  • A “model” that never predicts fraud scored 91.5 percent accuracy, beating the real model’s 90.1 percent while catching nothing.
  • The best F1 came at threshold 0.6, but the lowest business cost was at 0.5, because missed fraud cost 25 times more than a false alarm.

“94 percent accurate” sounds impressive until you learn the model never catches the thing it was built to catch. A single number hides how a classifier fails, and different failures have very different costs: missing a fraudulent payment is not the same as flagging a genuine one. A confusion matrix lays out every kind of right and wrong answer in a small table, and from it you can derive the metric that actually matches your problem.

We computed everything from scratch with NumPy 2.5.5 on Python 3.13.6, using a synthetic, seeded dataset. scikit-learn’s confusion_matrix and classification_report produce the same numbers; it was not installed in our environment, so we did not use it.

What is a confusion matrix?

Actual vs predicted Predicted positive Predicted negative
Actually positive True positive (TP) False negative (FN)
Actually negative False positive (FP) True negative (TN)

Rows are what really happened and columns are what the model predicted (some tools swap them, so always check the labels). The diagonal holds the correct predictions. False positives are false alarms, also called Type I errors; false negatives are misses, Type II errors.

Our fraud example

rng = np.random.default_rng(21)
y = (rng.uniform(size=1000) < 0.08).astype(int)       # about 8% fraud
score = np.where(y == 1, rng.normal(0.7, 0.18, 1000),   # fraud scores higher on average
                         rng.normal(0.3, 0.15, 1000)).clip(0, 1)
# positives: 85 of 1000

1,000 transactions, 85 of them fraudulent, each with a model score between 0 and 1. Fraud tends to score high and genuine payments low, but the two overlap, as real model outputs do. The score could come from logistic regression, a random forest or any other classifier.

Building the confusion matrix in Python

def cm(y, p):
    tp = int(((p == 1) & (y == 1)).sum()); tn = int(((p == 0) & (y == 0)).sum())
    fp = int(((p == 1) & (y == 0)).sum()); fn = int(((p == 0) & (y == 1)).sum())
    return tp, fp, fn, tn

p = (score >= 0.5).astype(int)
cm(y, p)    # TP 79, FP 93, FN 6, TN 822

At the default threshold of 0.5, predictions are 1 (fraud) when the score is at least 0.5. Counting the four combinations of actual and predicted labels gives the matrix. With scikit-learn the equivalent is confusion_matrix(y, p).

Threshold 0.5 Predicted fraud Predicted genuine
Actual fraud (85) 79 (TP) 6 (FN)
Actual genuine (915) 93 (FP) 822 (TN)

Reading the four cells

The model caught 79 frauds and missed 6. It also raised 93 false alarms, genuine customers whose payments would be blocked or reviewed. Of the 172 alerts in total, fewer than half were real fraud. Whether that is acceptable depends on what a false alarm costs (a support call, an annoyed customer) compared with a missed fraud (a chargeback). The matrix makes that trade-off visible; a single accuracy figure hides it.

Metrics derived from the matrix

accuracy    = (TP + TN) / total      = (79 + 822) / 1000 = 0.901
precision   = TP / (TP + FP)         = 79 / 172          = 0.459
recall      = TP / (TP + FN)         = 79 / 85           = 0.929
specificity = TN / (TN + FP)         = 822 / 915         = 0.898
F1          = 2PR / (P + R)                               = 0.615
  • Precision: of everything flagged, how much was right. Matters when false alarms are expensive.
  • Recall (sensitivity, true positive rate): of all real positives, how many were found. Matters when misses are expensive.
  • Specificity: of all real negatives, how many were correctly left alone.
  • F1: the harmonic mean of precision and recall, which is high only when both are high.

The accuracy trap

always predict "not fraud":  TP 0, FP 0, FN 85, TN 915
accuracy 0.915   precision 0.0   recall 0.0   specificity 1.0   F1 0.0

A model that simply answers “genuine” for every transaction scored 91.5 percent accuracy, higher than our real model’s 90.1 percent, while catching zero frauds. With 8.5 percent positives, accuracy is dominated by the easy majority class. This is why accuracy is misleading for imbalanced problems such as fraud, disease screening, defect detection or spam. Look at the matrix, recall and precision instead, and always compare against this trivial baseline. Similar care applies to naive Bayes spam filters, where spam may be rare.

Changing the threshold

Threshold TP FP FN TN Precision Recall F1
0.3 85 473 0 442 0.152 1.000 0.264
0.4 84 241 1 674 0.258 0.988 0.410
0.5 79 93 6 822 0.459 0.929 0.615
0.6 60 29 25 886 0.674 0.706 0.690
0.7 41 6 44 909 0.872 0.482 0.621

The confusion matrix depends on the threshold, not just the model. At 0.3 every fraud was caught, at the cost of 473 false alarms; at 0.7 only 6 false alarms remained, but more than half the frauds slipped through. Precision and recall pull in opposite directions, and F1 peaked at 0.6 in this data. Plotting precision against recall across all thresholds gives the precision-recall curve, a fuller picture than any single matrix.

Choosing the threshold by cost

cost = FN * 500 + FP * 20       # a missed fraud costs 500, a false alarm 20
threshold 0.3: 9460
threshold 0.4: 5320
threshold 0.5: 4860   lowest
threshold 0.6: 13080

F1 treats precision and recall as equally important, which real businesses rarely do. Assigning a cost to each cell made the decision concrete: with a missed fraud 25 times as expensive as a false alarm, threshold 0.5 minimised cost, even though 0.6 had a better F1. Moving from 0.5 to 0.6 saved 64 false alarms but let 19 more frauds through, which cost far more. Agree these costs with the business before choosing a threshold, and revisit them when the costs change.

Tip: report the confusion matrix together with the threshold that produced it. A precision or recall figure without its threshold cannot be compared with anything.

Multiclass confusion matrices

# support tickets: rows = actual, columns = predicted
             bug  feature  question
bug          109       13        12
feature        6       80        10
question       5        7        58

With more than two classes, the matrix grows to one row and column per class. Each row shows where the tickets of one true class ended up: 109 bugs were classified correctly, 13 were mistaken for feature requests and 12 for questions. Off-diagonal cells show which classes the model confuses, often the most useful insight for improving data or features. Per-class precision and recall come from treating each class in turn as positive and all others as negative.

bug       precision 0.908  recall 0.813  F1 0.858  support 134
feature   precision 0.800  recall 0.833  F1 0.816  support 96
question  precision 0.725  recall 0.829  F1 0.773  support 70

Macro, weighted and micro averages

macro F1 0.816    weighted F1 0.825    micro F1 = accuracy 0.823

Macro averaging takes the plain mean of per-class scores, so small classes count as much as large ones; use it when every class matters equally. Weighted averaging weights by support (number of true examples per class), so it leans towards large classes. Micro averaging pools all predictions; for single-label multiclass problems it equals accuracy. Always state which one you report.

Normalised matrices

             bug   feature  question
bug          0.81  0.10     0.09
feature      0.06  0.83     0.10
question     0.07  0.10     0.83

Dividing each row by its total turns counts into proportions, so the diagonal shows recall per class and classes of different sizes become comparable. Our model got about 81 to 83 percent of each class right, but the raw counts hid that the smallest class, questions, had the lowest precision. Show both: counts reveal volume, normalised rows reveal behaviour.

Practical tips

  • Evaluate on held-out data, never on the training set; models such as unpruned decision trees score perfectly on their training data.
  • Check the orientation: confirm which axis is actual and which is predicted.
  • Start from the baseline: compare with always predicting the majority class.
  • Pick the metric from the cost of errors, then choose the threshold on validation data.
  • Watch for drift: recompute the matrix regularly in production; class balance and costs change.
  • Reuse the idea in testing: a test suite is itself a classifier of builds, and false alarms are flaky tests; the same reasoning also applies to distance-based models such as k nearest neighbors.

Final thoughts

A confusion matrix turns “how good is the model?” into four honest counts. From them come precision, recall, specificity and F1, each answering a different business question. Our fraud example showed why that matters: a do-nothing model beat the real one on accuracy, F1 and cost disagreed on the best threshold, and per-class views revealed which ticket types were confused. Always show the matrix with its threshold, and choose metrics from what each kind of mistake actually costs.

Frequently asked questions

What is a confusion matrix?

A table that counts a classifier predictions against the true labels, showing true positives, false positives, false negatives and true negatives, so you can see exactly how it errs.

What is the difference between precision and recall?

Precision is the share of positive predictions that were correct. Recall is the share of actual positives that the model found. Raising one usually lowers the other.

Why is accuracy misleading for imbalanced data?

When one class is rare, predicting the common class every time gives high accuracy while finding nothing. In our example that scored 91.5 percent and caught zero frauds.

What is a good F1 score?

It depends on the problem and the baseline. Compare F1 with simple baselines and earlier models on the same data, and remember that F1 weights precision and recall equally.

How do I choose a classification threshold?

Compute the confusion matrix at several thresholds on validation data, assign a cost to false positives and false negatives, and pick the threshold with the lowest total cost.

Keep learning

Work with Stack Quora

Need help with testing, WordPress or SEO?

We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.

Keep reading

More guides on testing, AI & modern development

Enjoyed this article? Pick a topic and jump straight into our newest tutorials.

Artificial Intelligence

Confusion Matrix Explained: Precision, Recall, F1 and Thresholds in Python

Confusion matrix explained with Python: TP, FP, FN, TN, precision, recall, F1, the accuracy trap, threshold choice by cost and…

Read article →
Software Development

API Pagination: Offset vs Cursor vs Keyset, Tested on 1 Million Rows

API pagination explained: offset vs keyset and cursor pagination timed on 1 million rows, duplicates when data changes, the ties…

Read article →
DevOps

Git Submodule Explained: Add, Clone, Update and Remove Safely

Git submodule explained: add, clone, update –init, update –remote, the detached HEAD trap, foreach and clean removal, tested step by…

Read article →
Test Automation

Pytest tmp path Explained: tmp_path, tmp_path_factory and Temp Files

Pytest tmp_path explained: per-test temporary folders, tmp_path_factory for shared files, monkeypatch.chdir, legacy tmpdir, retention and basetemp.

Read article →
Software Testing

Test Case Prioritization: 6 Techniques Compared With APFD

Test case prioritization compared: risk, coverage, cost, history and random orders scored with APFD and time to find all faults…

Read article →
Databases & SQL

Full Text Search in SQL: FTS5, Ranking, Phrases and Speed

Full text search in SQL with SQLite FTS5: MATCH queries, phrases, prefixes, NEAR, stemming, BM25 ranking, snippets, two gotchas and…

Read article →
Databases & SQL

Generated Columns in SQL: Virtual vs Stored, Indexes and Limits

Generated columns in SQL explained: virtual vs stored, automatic updates, write errors, storage cost, indexing, JSON fields and support in…

Read article →
Software Development

Object Freeze in JavaScript: Shallow vs Deep, Seal and const

Object freeze in JavaScript explained: freeze vs seal vs const, strict mode errors, shallow vs deep freeze, frozen arrays, Map…

Read article →
Software Development

Python hashlib: SHA-256, File Hashes, HMAC and Password Hashing

Python hashlib explained: SHA-256 for strings and files, algorithm speeds, why passwords need PBKDF2 or scrypt, safe verification and HMAC…

Read article →
Software Development

Python UUID Explained: uuid1, uuid4, uuid5, uuid7 and Database Keys

Python UUID explained: uuid4, uuid5, uuid1 privacy, time-ordered uuid7, parsing, storage formats and UUIDs as database keys, tested on Python…

Read article →
Artificial Intelligence

Random Forest in Python: Bagging, Feature Sampling and OOB From Scratch

Random forest in Python from scratch: bootstrap sampling, feature sampling, number of trees, out-of-bag evaluation and impurity vs permutation importance.

Read article →
Software Development

MIME Types Explained: Content-Type, nosniff and a Common Types List

MIME types explained and tested in a browser: Content-Type, nosniff, scripts and stylesheets with wrong types, a common MIME types…

Read article →
DevOps

Git Clean Explained: Remove Untracked Files Without Regret

Git clean explained with dry runs: -n, -f, -d, -x, -X, exclusions, paths, nested repositories and interactive mode, plus a…

Read article →
Test Automation

Pytest approx: Comparing Floats With rel, abs and Collections

Pytest approx explained: default tolerances, the near-zero trap, rel and abs, lists and dicts, NaN, NumPy, and how it differs…

Read article →
Software Testing

Accessibility Testing: WCAG Checklist, Automated Tools and Manual Checks

Accessibility testing explained: WCAG basics, an axe-core scan of a broken and a fixed page, what automation missed, contrast checks…

Read article →
Databases & SQL

SQL UNPIVOT: Turn Columns Into Rows, 4 Ways Tested

SQL UNPIVOT explained: turn columns into rows with UNION ALL, CROSS JOIN with CASE, JSON functions or native UNPIVOT, with…

Read article →
Software Development

Intl NumberFormat in JavaScript: Currency, Compact, Percent and Units

Intl.NumberFormat in JavaScript: locales, currency, compact notation, percent, units, rounding, formatToParts, the non-breaking space trap and formatter reuse.

Read article →
Databases & SQL

SQL EXPLAIN Plan: How to Read Query Plans and Fix Slow Queries

SQL EXPLAIN plan explained: read scans, index searches, covering indexes and temporary sorts, with before and after timings for five…

Read article →
Artificial Intelligence

Decision Tree in Python: Gini, Entropy and Overfitting From Scratch

Decision tree in Python from scratch: Gini impurity, entropy, information gain, best splits, readable rules, and how depth and leaf…

Read article →
DevOps

Git Blame Explained: Find Who Changed a Line, and Why

Git blame explained with a real repository: ignore whitespace, line ranges and functions, ignore-revs files for formatting commits, moved code…

Read article →
Test Automation

Pytest Benchmark: Measure Speed and Catch Performance Regressions

Pytest benchmark tutorial: the benchmark fixture, reading results, parametrized and pedantic benchmarks, saving baselines and failing the build on regressions.

Read article →
Software Testing

Defect Density: Formula, Worked Examples and How to Use It

Defect density explained: the formula, a five-module worked example, per KLOC vs per function point, severity weighting, defect removal efficiency…

Read article →
Databases & SQL

Case Insensitive Search in SQL: LOWER, LIKE, COLLATE and Indexes

Case insensitive search in SQL: LOWER, LIKE, COLLATE NOCASE and expression indexes compared with query plans, plus accented letters and…

Read article →
Databases & SQL

ON DELETE CASCADE in SQL: CASCADE vs SET NULL vs RESTRICT

ON DELETE CASCADE in SQL tested: CASCADE vs SET NULL vs RESTRICT, multi-level cascades, the SQLite foreign key pragma, row…

Read article →
Artificial Intelligence

Logistic Regression in Python: Sigmoid, Log Loss and Gradient Descent From Scratch

Logistic regression in Python from scratch: sigmoid, log loss, gradient descent, odds ratios, thresholds, sigmoid overflow and separable data, tested…

Read article →
DevOps

Git Worktree Explained: Work on Two Branches at Once

Git worktree explained: add, list, remove, prune, lock and detached worktrees, a hotfix workflow, and the errors you will hit,…

Read article →
Test Automation

Pytest ini Explained: addopts, markers, testpaths and Precedence

Pytest ini explained: testpaths, addopts, strict markers, xfail_strict, filterwarnings, minversion, config file precedence and command-line overrides, tested.

Read article →
Software Testing

Test Estimation Techniques: WBS, Three Point PERT and Monte Carlo

Test estimation techniques worked through: WBS, three point PERT, total uncertainty, a Monte Carlo check, percentage rules and Wideband Delphi.

Read article →
Artificial Intelligence

Linear Regression in Python: Least Squares From Scratch, With R Squared

Linear regression in Python from scratch: closed form, NumPy least squares and gradient descent, R squared, RMSE, outliers and curved…

Read article →
DevOps

Git Remote Explained: Add, Set-URL, Rename, Remove and Push URLs

Git remote explained: add, set-url, separate push URLs, rename, remove and prune, plus how to avoid leaking tokens in remote…

Read article →
Test Automation

Pytest Mock: mocker.patch, spy, side_effect and autospec Explained

Pytest mock explained: mocker.patch, return_value, side_effect, spy, autospec and PropertyMock, with tested examples for APIs, clocks and retries.

Read article →
Software Testing

System Testing Explained: Types, Process and a Real End-to-End Example

System testing explained: types, entry and exit criteria, the process, and a real end-to-end test suite run against a small…

Read article →
Artificial Intelligence

Naive Bayes Classifier Explained: Build a Spam Filter in Python

Naive Bayes classifier explained by building a spam filter in Python: Bayes rule, the naive assumption, Laplace smoothing, log probabilities…

Read article →
DevOps

Git Branch Explained: Create, Switch, Rename, Delete and Recover

Git branch explained with tested commands: list, create, switch, rename, delete, merged checks, recovering deleted branches, remote tracking and naming.

Read article →
Test Automation

Pytest raises: Testing Exceptions, match and excinfo

Pytest raises explained with tested examples: match regexes, excinfo, custom and chained exceptions, parametrized error cases, pytest.warns and common mistakes.

Read article →
Software Testing

Non Functional Testing: Types, Metrics and a Real Load Test

Non functional testing explained: 10 key types, measurable requirements, latency percentiles and what a small real load test revealed about…

Read article →