1. Home
  2. Blog
  3. Artificial Intelligence

Artificial Intelligence

Logistic Regression in Python: Sigmoid, Log Loss and Gradient Descent From Scratch

Logistic regression predicts the probability of a yes or no outcome: will a student pass, will a customer churn, is this email spam? We build it from scratch in NumPy, train it with gradient descent on log loss, read the coefficients as odds, tune the decision threshold, and run into the two classic failures: sigmoid overflow and perfectly separable data.

9 min read
Logistic regression in Python: sigmoid, log loss and gradient descent from scratch

Key takeaways

  • Logistic regression passes a linear score through the sigmoid function to get a probability between 0 and 1.
  • Trained with gradient descent on log loss, our model found the true 5-hour pass boundary at 5.04 hours and scored 90.5 percent accuracy.
  • Each extra hour of study multiplied the odds of passing by 3.67.
  • On perfectly separable data the weights kept growing (slope 2.14, then 4.0, then 6.21); L2 regularisation held it at 1.22.

Many real questions have a yes or no answer: pass or fail, fraud or genuine, click or no click. Logistic regression is the standard first model for them. It is fast, easy to explain to non-specialists, gives calibrated probabilities rather than bare labels, and often performs surprisingly close to far more complex models on tabular data.

We implemented it from scratch with NumPy 2.5.5 on Python 3.13.6, using a synthetic, seeded dataset so we know the true answer. Every number below is from those runs. We did not use scikit-learn, whose LogisticRegression does the same job with extra solvers and regularisation by default.

What is logistic regression?

Despite the name, logistic regression is a classification model. It computes a linear score, exactly as linear regression does, then squashes it into a probability:

z = b0 + b1 * x                 # linear score, any real number
p = 1 / (1 + exp(-z))           # sigmoid: probability between 0 and 1
predict 1 if p >= 0.5 else 0

Training finds the weights b0 and b1 that make the predicted probabilities match the observed outcomes as closely as possible.

Why not just use linear regression on 0 and 1?

b = np.linalg.lstsq(X, passed, rcond=None)[0]
linear reg on labels: min -0.2  max 1.197  -> outside 0..1

Fitting a straight line to 0/1 labels produced “probabilities” of -0.2 and 1.197, which make no sense. A straight line also lets extreme points pull the fit, and treats a prediction of 2 for a positive example as an error. The sigmoid fixes all of this by bending the line into an S shape that never leaves the 0 to 1 range.

The sigmoid function

def sigmoid(z): return 1 / (1 + np.exp(-z))
sigmoid at -5, -1, 0, 1, 5: [0.0067, 0.2689, 0.5, 0.7311, 0.9933]

A score of 0 maps to exactly 0.5, large positive scores approach 1 and large negative ones approach 0. The function is symmetric: sigmoid(1) and sigmoid(-1) add up to 1. For more than two classes the same role is played by the softmax function, of which sigmoid is the two-class special case.

Our dataset: study hours and exam results

rng = np.random.default_rng(42)
hours = rng.uniform(0, 10, 200)
true_p = 1 / (1 + np.exp(-(1.2 * hours - 6)))      # true boundary at 5 hours
passed = (rng.uniform(size=200) < true_p).astype(int)
# pass rate 0.49

200 students, each with a pass probability that rises smoothly with study hours. The true model has slope 1.2 and intercept -6, so the 50 percent point is exactly 5 hours. Because outcomes are random draws, some students who studied 8 hours still failed, just as in real data.

Log loss: how wrong is a probability?

def logloss(w):
    p = np.clip(sigmoid(X @ w), 1e-12, 1 - 1e-12)
    return -np.mean(passed * np.log(p) + (1 - passed) * np.log(1 - p))
# loss at start (all weights 0): 0.6931

Log loss, also called binary cross-entropy, punishes confident wrong answers heavily: predicting 0.99 for a student who failed costs far more than predicting 0.6. With all weights at zero, every prediction is 0.5 and the loss is ln 2, which is 0.6931, the value we got. Squared error is not used here because combined with the sigmoid it creates a non-convex surface; log loss is convex, so gradient descent cannot get stuck in a local minimum. The np.clip keeps log(0) from producing infinity.

Training with gradient descent

w = np.zeros(2)
for i in range(20001):
    p = sigmoid(X @ w)
    g = X.T @ (p - passed) / n      # gradient of log loss
    w -= 0.1 * g

step     0  loss 0.6228  w [-0.001  0.108]
step   100  loss 0.4325  w [-1.329  0.363]
step  1000  loss 0.2674  w [-4.62   0.945]
step  5000  loss 0.2543  w [-6.379  1.268]
step 20000  loss 0.2542  w [-6.557  1.301]

The gradient has a neat form: the inputs times the prediction errors p - y, the same shape as in linear regression. The loss fell quickly, then flattened: between step 5,000 and 20,000 it improved by only 0.0001. The final weights, intercept -6.557 and slope 1.301, are close to the true -6 and 1.2; with only 200 noisy examples, an exact match is not expected. Feature scaling would make convergence faster, as we showed with gradient descent in the linear regression article.

The decision boundary

decision boundary hours 5.04   (true 5.0)
P(pass | 2 h) = 0.019
P(pass | 4 h) = 0.205
P(pass | 5 h) = 0.487
P(pass | 6 h) = 0.777
P(pass | 8 h) = 0.979

The boundary is where the score is zero, so the probability is 0.5: hours = -b0 / b1 = 5.04, almost exactly the true value. The probabilities are the real value of the model: two hours of study gives about a 2 percent chance of passing, eight hours about 98 percent. A teacher can act on “this student has a 20 percent chance” in a way they cannot act on a bare “fail” label.

Accuracy, precision and recall

accuracy 0.905
tp 87  tn 94  fp 8  fn 11
precision 0.916  recall 0.888

The model classified 181 of 200 students correctly. Precision says that of those predicted to pass, 91.6 percent did; recall says that of those who passed, the model found 88.8 percent. Because the data is random around the boundary, about 90 percent is close to the best any model could do here. Always report precision and recall as well as accuracy, especially when one class is rare.

Reading coefficients as odds

odds ratio per hour = exp(1.301) = 3.67

This is why logistic regression is popular in medicine, finance and social science. The odds of passing are p / (1 – p). Each additional hour of study multiplied those odds by 3.67. A coefficient of 0 would mean an odds ratio of 1, no effect; a negative coefficient would shrink the odds. That interpretability is something tree ensembles and neural networks cannot offer as directly.

Choosing the decision threshold

Threshold Precision Recall
0.3 0.812 0.929
0.5 0.916 0.888
0.7 0.929 0.796

0.5 is only a default. Lowering the threshold catches more positives (higher recall) at the cost of more false alarms (lower precision); raising it does the opposite. For fraud or disease screening, where missing a case is expensive, a low threshold makes sense. For sending a costly intervention, a high one does. Pick the threshold from the business cost of each error type, not from habit.

Pitfall 1: sigmoid overflow

naive sigmoid(-1000) -> [0.]  with RuntimeWarning (overflow in exp)
stable sigmoid(-1000) -> [0.]   stable sigmoid(1000) -> [1.]

For very negative scores, np.exp(-z) becomes exp(1000), which overflows to infinity. NumPy still returned the right answer, 0, but raised a RuntimeWarning, and in other code paths the overflow can turn into NaN. The stable version only ever computes exp(-abs(z)), which cannot overflow, and handles positive and negative scores separately. Large scores are common with unscaled features, so use the stable form, or scipy.special.expit, in real code.

Pitfall 2: perfectly separable data

x = [1, 2, 3, 4, 6, 7, 8, 9]   y = [0, 0, 0, 0, 1, 1, 1, 1]
separable step   1000  w [-10.47  2.14]
separable step  10000  w [-19.8   4.  ]
separable step 100000  w [-30.85  6.21]
with L2 0.1          w [-6.12   1.22]

When one value of x splits the classes perfectly, there is no best answer: making the sigmoid steeper always reduces the loss a little further. The weights grew without limit, from slope 2.14 to 6.21 and still climbing, and the model became absurdly overconfident. Adding an L2 penalty on the slope, which discourages large weights, stopped the growth at a sensible 1.22. That is why scikit-learn applies L2 regularisation by default, and why tiny or clean datasets need it most.

Tip: if your coefficients look huge or keep changing as you train longer, check for a feature that perfectly separates the classes, often a leaked label.

More features and more than two classes

With several inputs, the score becomes b0 + b1 x1 + b2 x2 + ..., and nothing else changes: add columns to X and the same gradient descent code trains it. Scale the features so that gradient descent converges and the coefficients are comparable. For three or more classes, multinomial logistic regression replaces sigmoid with softmax, giving one probability per class. Compared with naive Bayes, logistic regression does not assume features are independent, and compared with k nearest neighbors it produces a compact model that predicts instantly. A single logistic unit is also exactly one neuron of a neural network, the building block behind large language models.

Final thoughts

Logistic regression is a linear model wearing a sigmoid. Train it on log loss with gradient descent, read its probabilities rather than just its labels, interpret coefficients as odds ratios, and choose the threshold from the cost of each kind of mistake. Guard against the two failures we reproduced, overflow in a naive sigmoid and runaway weights on separable data, with a stable sigmoid and regularisation. It remains the baseline every classifier should be compared against.

Frequently asked questions

What is logistic regression used for?

Predicting the probability of a binary outcome, such as pass or fail, churn or stay, fraud or genuine. It is also widely used to explain how each input changes the odds of the outcome.

Why is it called regression if it is used for classification?

It regresses the log odds of the outcome on the inputs with a linear model. The sigmoid then turns those log odds into a probability, and a threshold turns the probability into a class.

What loss function does logistic regression use?

Log loss, also called binary cross-entropy. It is convex when combined with the sigmoid, so gradient descent reliably finds the best weights.

How do I interpret logistic regression coefficients?

Take the exponential of a coefficient to get an odds ratio. In our example the slope of 1.301 gave 3.67, so each extra study hour multiplied the odds of passing by 3.67.

Why do my logistic regression weights keep growing?

Usually because the data is perfectly separable, so a steeper curve always lowers the loss. Add L2 regularisation, which kept our slope at 1.22 instead of growing past 6.

Keep learning

Work with Stack Quora

Need help with testing, WordPress or SEO?

We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.

Keep reading

More guides on testing, AI & modern development

Enjoyed this article? Pick a topic and jump straight into our newest tutorials.

Artificial Intelligence

Confusion Matrix Explained: Precision, Recall, F1 and Thresholds in Python

Confusion matrix explained with Python: TP, FP, FN, TN, precision, recall, F1, the accuracy trap, threshold choice by cost and…

Read article →
Software Development

API Pagination: Offset vs Cursor vs Keyset, Tested on 1 Million Rows

API pagination explained: offset vs keyset and cursor pagination timed on 1 million rows, duplicates when data changes, the ties…

Read article →
DevOps

Git Submodule Explained: Add, Clone, Update and Remove Safely

Git submodule explained: add, clone, update –init, update –remote, the detached HEAD trap, foreach and clean removal, tested step by…

Read article →
Test Automation

Pytest tmp path Explained: tmp_path, tmp_path_factory and Temp Files

Pytest tmp_path explained: per-test temporary folders, tmp_path_factory for shared files, monkeypatch.chdir, legacy tmpdir, retention and basetemp.

Read article →
Software Testing

Test Case Prioritization: 6 Techniques Compared With APFD

Test case prioritization compared: risk, coverage, cost, history and random orders scored with APFD and time to find all faults…

Read article →
Databases & SQL

Full Text Search in SQL: FTS5, Ranking, Phrases and Speed

Full text search in SQL with SQLite FTS5: MATCH queries, phrases, prefixes, NEAR, stemming, BM25 ranking, snippets, two gotchas and…

Read article →
Databases & SQL

Generated Columns in SQL: Virtual vs Stored, Indexes and Limits

Generated columns in SQL explained: virtual vs stored, automatic updates, write errors, storage cost, indexing, JSON fields and support in…

Read article →
Software Development

Object Freeze in JavaScript: Shallow vs Deep, Seal and const

Object freeze in JavaScript explained: freeze vs seal vs const, strict mode errors, shallow vs deep freeze, frozen arrays, Map…

Read article →
Software Development

Python hashlib: SHA-256, File Hashes, HMAC and Password Hashing

Python hashlib explained: SHA-256 for strings and files, algorithm speeds, why passwords need PBKDF2 or scrypt, safe verification and HMAC…

Read article →
Software Development

Python UUID Explained: uuid1, uuid4, uuid5, uuid7 and Database Keys

Python UUID explained: uuid4, uuid5, uuid1 privacy, time-ordered uuid7, parsing, storage formats and UUIDs as database keys, tested on Python…

Read article →
Artificial Intelligence

Random Forest in Python: Bagging, Feature Sampling and OOB From Scratch

Random forest in Python from scratch: bootstrap sampling, feature sampling, number of trees, out-of-bag evaluation and impurity vs permutation importance.

Read article →
Software Development

MIME Types Explained: Content-Type, nosniff and a Common Types List

MIME types explained and tested in a browser: Content-Type, nosniff, scripts and stylesheets with wrong types, a common MIME types…

Read article →
DevOps

Git Clean Explained: Remove Untracked Files Without Regret

Git clean explained with dry runs: -n, -f, -d, -x, -X, exclusions, paths, nested repositories and interactive mode, plus a…

Read article →
Test Automation

Pytest approx: Comparing Floats With rel, abs and Collections

Pytest approx explained: default tolerances, the near-zero trap, rel and abs, lists and dicts, NaN, NumPy, and how it differs…

Read article →
Software Testing

Accessibility Testing: WCAG Checklist, Automated Tools and Manual Checks

Accessibility testing explained: WCAG basics, an axe-core scan of a broken and a fixed page, what automation missed, contrast checks…

Read article →
Databases & SQL

SQL UNPIVOT: Turn Columns Into Rows, 4 Ways Tested

SQL UNPIVOT explained: turn columns into rows with UNION ALL, CROSS JOIN with CASE, JSON functions or native UNPIVOT, with…

Read article →
Software Development

Intl NumberFormat in JavaScript: Currency, Compact, Percent and Units

Intl.NumberFormat in JavaScript: locales, currency, compact notation, percent, units, rounding, formatToParts, the non-breaking space trap and formatter reuse.

Read article →
Databases & SQL

SQL EXPLAIN Plan: How to Read Query Plans and Fix Slow Queries

SQL EXPLAIN plan explained: read scans, index searches, covering indexes and temporary sorts, with before and after timings for five…

Read article →
Artificial Intelligence

Decision Tree in Python: Gini, Entropy and Overfitting From Scratch

Decision tree in Python from scratch: Gini impurity, entropy, information gain, best splits, readable rules, and how depth and leaf…

Read article →
DevOps

Git Blame Explained: Find Who Changed a Line, and Why

Git blame explained with a real repository: ignore whitespace, line ranges and functions, ignore-revs files for formatting commits, moved code…

Read article →
Test Automation

Pytest Benchmark: Measure Speed and Catch Performance Regressions

Pytest benchmark tutorial: the benchmark fixture, reading results, parametrized and pedantic benchmarks, saving baselines and failing the build on regressions.

Read article →
Software Testing

Defect Density: Formula, Worked Examples and How to Use It

Defect density explained: the formula, a five-module worked example, per KLOC vs per function point, severity weighting, defect removal efficiency…

Read article →
Databases & SQL

Case Insensitive Search in SQL: LOWER, LIKE, COLLATE and Indexes

Case insensitive search in SQL: LOWER, LIKE, COLLATE NOCASE and expression indexes compared with query plans, plus accented letters and…

Read article →
Databases & SQL

ON DELETE CASCADE in SQL: CASCADE vs SET NULL vs RESTRICT

ON DELETE CASCADE in SQL tested: CASCADE vs SET NULL vs RESTRICT, multi-level cascades, the SQLite foreign key pragma, row…

Read article →
Artificial Intelligence

Logistic Regression in Python: Sigmoid, Log Loss and Gradient Descent From Scratch

Logistic regression in Python from scratch: sigmoid, log loss, gradient descent, odds ratios, thresholds, sigmoid overflow and separable data, tested…

Read article →
DevOps

Git Worktree Explained: Work on Two Branches at Once

Git worktree explained: add, list, remove, prune, lock and detached worktrees, a hotfix workflow, and the errors you will hit,…

Read article →
Test Automation

Pytest ini Explained: addopts, markers, testpaths and Precedence

Pytest ini explained: testpaths, addopts, strict markers, xfail_strict, filterwarnings, minversion, config file precedence and command-line overrides, tested.

Read article →
Software Testing

Test Estimation Techniques: WBS, Three Point PERT and Monte Carlo

Test estimation techniques worked through: WBS, three point PERT, total uncertainty, a Monte Carlo check, percentage rules and Wideband Delphi.

Read article →
Artificial Intelligence

Linear Regression in Python: Least Squares From Scratch, With R Squared

Linear regression in Python from scratch: closed form, NumPy least squares and gradient descent, R squared, RMSE, outliers and curved…

Read article →
DevOps

Git Remote Explained: Add, Set-URL, Rename, Remove and Push URLs

Git remote explained: add, set-url, separate push URLs, rename, remove and prune, plus how to avoid leaking tokens in remote…

Read article →
Test Automation

Pytest Mock: mocker.patch, spy, side_effect and autospec Explained

Pytest mock explained: mocker.patch, return_value, side_effect, spy, autospec and PropertyMock, with tested examples for APIs, clocks and retries.

Read article →
Software Testing

System Testing Explained: Types, Process and a Real End-to-End Example

System testing explained: types, entry and exit criteria, the process, and a real end-to-end test suite run against a small…

Read article →
Artificial Intelligence

Naive Bayes Classifier Explained: Build a Spam Filter in Python

Naive Bayes classifier explained by building a spam filter in Python: Bayes rule, the naive assumption, Laplace smoothing, log probabilities…

Read article →
DevOps

Git Branch Explained: Create, Switch, Rename, Delete and Recover

Git branch explained with tested commands: list, create, switch, rename, delete, merged checks, recovering deleted branches, remote tracking and naming.

Read article →
Test Automation

Pytest raises: Testing Exceptions, match and excinfo

Pytest raises explained with tested examples: match regexes, excinfo, custom and chained exceptions, parametrized error cases, pytest.warns and common mistakes.

Read article →
Software Testing

Non Functional Testing: Types, Metrics and a Real Load Test

Non functional testing explained: 10 key types, measurable requirements, latency percentiles and what a small real load test revealed about…

Read article →