1. Home
  2. Blog
  3. Artificial Intelligence

Artificial Intelligence

Logistic Regression in Python: Sigmoid, Log Loss and Gradient Descent From Scratch

Logistic regression predicts the probability of a yes or no outcome: will a student pass, will a customer churn, is this email spam? We build it from scratch in NumPy, train it with gradient descent on log loss, read the coefficients as odds, tune the decision threshold, and run into the two classic failures: sigmoid overflow and perfectly separable data.

9 min read
Logistic regression in Python: sigmoid, log loss and gradient descent from scratch

Key takeaways

  • Logistic regression passes a linear score through the sigmoid function to get a probability between 0 and 1.
  • Trained with gradient descent on log loss, our model found the true 5-hour pass boundary at 5.04 hours and scored 90.5 percent accuracy.
  • Each extra hour of study multiplied the odds of passing by 3.67.
  • On perfectly separable data the weights kept growing (slope 2.14, then 4.0, then 6.21); L2 regularisation held it at 1.22.

Many real questions have a yes or no answer: pass or fail, fraud or genuine, click or no click. Logistic regression is the standard first model for them. It is fast, easy to explain to non-specialists, gives calibrated probabilities rather than bare labels, and often performs surprisingly close to far more complex models on tabular data.

We implemented it from scratch with NumPy 2.5.5 on Python 3.13.6, using a synthetic, seeded dataset so we know the true answer. Every number below is from those runs. We did not use scikit-learn, whose LogisticRegression does the same job with extra solvers and regularisation by default.

What is logistic regression?

Despite the name, logistic regression is a classification model. It computes a linear score, exactly as linear regression does, then squashes it into a probability:

z = b0 + b1 * x                 # linear score, any real number
p = 1 / (1 + exp(-z))           # sigmoid: probability between 0 and 1
predict 1 if p >= 0.5 else 0

Training finds the weights b0 and b1 that make the predicted probabilities match the observed outcomes as closely as possible.

Why not just use linear regression on 0 and 1?

b = np.linalg.lstsq(X, passed, rcond=None)[0]
linear reg on labels: min -0.2  max 1.197  -> outside 0..1

Fitting a straight line to 0/1 labels produced “probabilities” of -0.2 and 1.197, which make no sense. A straight line also lets extreme points pull the fit, and treats a prediction of 2 for a positive example as an error. The sigmoid fixes all of this by bending the line into an S shape that never leaves the 0 to 1 range.

The sigmoid function

def sigmoid(z): return 1 / (1 + np.exp(-z))
sigmoid at -5, -1, 0, 1, 5: [0.0067, 0.2689, 0.5, 0.7311, 0.9933]

A score of 0 maps to exactly 0.5, large positive scores approach 1 and large negative ones approach 0. The function is symmetric: sigmoid(1) and sigmoid(-1) add up to 1. For more than two classes the same role is played by the softmax function, of which sigmoid is the two-class special case.

Our dataset: study hours and exam results

rng = np.random.default_rng(42)
hours = rng.uniform(0, 10, 200)
true_p = 1 / (1 + np.exp(-(1.2 * hours - 6)))      # true boundary at 5 hours
passed = (rng.uniform(size=200) < true_p).astype(int)
# pass rate 0.49

200 students, each with a pass probability that rises smoothly with study hours. The true model has slope 1.2 and intercept -6, so the 50 percent point is exactly 5 hours. Because outcomes are random draws, some students who studied 8 hours still failed, just as in real data.

Log loss: how wrong is a probability?

def logloss(w):
    p = np.clip(sigmoid(X @ w), 1e-12, 1 - 1e-12)
    return -np.mean(passed * np.log(p) + (1 - passed) * np.log(1 - p))
# loss at start (all weights 0): 0.6931

Log loss, also called binary cross-entropy, punishes confident wrong answers heavily: predicting 0.99 for a student who failed costs far more than predicting 0.6. With all weights at zero, every prediction is 0.5 and the loss is ln 2, which is 0.6931, the value we got. Squared error is not used here because combined with the sigmoid it creates a non-convex surface; log loss is convex, so gradient descent cannot get stuck in a local minimum. The np.clip keeps log(0) from producing infinity.

Training with gradient descent

w = np.zeros(2)
for i in range(20001):
    p = sigmoid(X @ w)
    g = X.T @ (p - passed) / n      # gradient of log loss
    w -= 0.1 * g

step     0  loss 0.6228  w [-0.001  0.108]
step   100  loss 0.4325  w [-1.329  0.363]
step  1000  loss 0.2674  w [-4.62   0.945]
step  5000  loss 0.2543  w [-6.379  1.268]
step 20000  loss 0.2542  w [-6.557  1.301]

The gradient has a neat form: the inputs times the prediction errors p - y, the same shape as in linear regression. The loss fell quickly, then flattened: between step 5,000 and 20,000 it improved by only 0.0001. The final weights, intercept -6.557 and slope 1.301, are close to the true -6 and 1.2; with only 200 noisy examples, an exact match is not expected. Feature scaling would make convergence faster, as we showed with gradient descent in the linear regression article.

The decision boundary

decision boundary hours 5.04   (true 5.0)
P(pass | 2 h) = 0.019
P(pass | 4 h) = 0.205
P(pass | 5 h) = 0.487
P(pass | 6 h) = 0.777
P(pass | 8 h) = 0.979

The boundary is where the score is zero, so the probability is 0.5: hours = -b0 / b1 = 5.04, almost exactly the true value. The probabilities are the real value of the model: two hours of study gives about a 2 percent chance of passing, eight hours about 98 percent. A teacher can act on “this student has a 20 percent chance” in a way they cannot act on a bare “fail” label.

Accuracy, precision and recall

accuracy 0.905
tp 87  tn 94  fp 8  fn 11
precision 0.916  recall 0.888

The model classified 181 of 200 students correctly. Precision says that of those predicted to pass, 91.6 percent did; recall says that of those who passed, the model found 88.8 percent. Because the data is random around the boundary, about 90 percent is close to the best any model could do here. Always report precision and recall as well as accuracy, especially when one class is rare.

Reading coefficients as odds

odds ratio per hour = exp(1.301) = 3.67

This is why logistic regression is popular in medicine, finance and social science. The odds of passing are p / (1 – p). Each additional hour of study multiplied those odds by 3.67. A coefficient of 0 would mean an odds ratio of 1, no effect; a negative coefficient would shrink the odds. That interpretability is something tree ensembles and neural networks cannot offer as directly.

Choosing the decision threshold

Threshold Precision Recall
0.3 0.812 0.929
0.5 0.916 0.888
0.7 0.929 0.796

0.5 is only a default. Lowering the threshold catches more positives (higher recall) at the cost of more false alarms (lower precision); raising it does the opposite. For fraud or disease screening, where missing a case is expensive, a low threshold makes sense. For sending a costly intervention, a high one does. Pick the threshold from the business cost of each error type, not from habit.

Pitfall 1: sigmoid overflow

naive sigmoid(-1000) -> [0.]  with RuntimeWarning (overflow in exp)
stable sigmoid(-1000) -> [0.]   stable sigmoid(1000) -> [1.]

For very negative scores, np.exp(-z) becomes exp(1000), which overflows to infinity. NumPy still returned the right answer, 0, but raised a RuntimeWarning, and in other code paths the overflow can turn into NaN. The stable version only ever computes exp(-abs(z)), which cannot overflow, and handles positive and negative scores separately. Large scores are common with unscaled features, so use the stable form, or scipy.special.expit, in real code.

Pitfall 2: perfectly separable data

x = [1, 2, 3, 4, 6, 7, 8, 9]   y = [0, 0, 0, 0, 1, 1, 1, 1]
separable step   1000  w [-10.47  2.14]
separable step  10000  w [-19.8   4.  ]
separable step 100000  w [-30.85  6.21]
with L2 0.1          w [-6.12   1.22]

When one value of x splits the classes perfectly, there is no best answer: making the sigmoid steeper always reduces the loss a little further. The weights grew without limit, from slope 2.14 to 6.21 and still climbing, and the model became absurdly overconfident. Adding an L2 penalty on the slope, which discourages large weights, stopped the growth at a sensible 1.22. That is why scikit-learn applies L2 regularisation by default, and why tiny or clean datasets need it most.

Tip: if your coefficients look huge or keep changing as you train longer, check for a feature that perfectly separates the classes, often a leaked label.

More features and more than two classes

With several inputs, the score becomes b0 + b1 x1 + b2 x2 + ..., and nothing else changes: add columns to X and the same gradient descent code trains it. Scale the features so that gradient descent converges and the coefficients are comparable. For three or more classes, multinomial logistic regression replaces sigmoid with softmax, giving one probability per class. Compared with naive Bayes, logistic regression does not assume features are independent, and compared with k nearest neighbors it produces a compact model that predicts instantly. A single logistic unit is also exactly one neuron of a neural network, the building block behind large language models.

Final thoughts

Logistic regression is a linear model wearing a sigmoid. Train it on log loss with gradient descent, read its probabilities rather than just its labels, interpret coefficients as odds ratios, and choose the threshold from the cost of each kind of mistake. Guard against the two failures we reproduced, overflow in a naive sigmoid and runaway weights on separable data, with a stable sigmoid and regularisation. It remains the baseline every classifier should be compared against.

Frequently asked questions

What is logistic regression used for?

Predicting the probability of a binary outcome, such as pass or fail, churn or stay, fraud or genuine. It is also widely used to explain how each input changes the odds of the outcome.

Why is it called regression if it is used for classification?

It regresses the log odds of the outcome on the inputs with a linear model. The sigmoid then turns those log odds into a probability, and a threshold turns the probability into a class.

What loss function does logistic regression use?

Log loss, also called binary cross-entropy. It is convex when combined with the sigmoid, so gradient descent reliably finds the best weights.

How do I interpret logistic regression coefficients?

Take the exponential of a coefficient to get an odds ratio. In our example the slope of 1.301 gave 3.67, so each extra study hour multiplied the odds of passing by 3.67.

Why do my logistic regression weights keep growing?

Usually because the data is perfectly separable, so a steeper curve always lowers the loss. Add L2 regularisation, which kept our slope at 1.22 instead of growing past 6.

Keep learning

Work with Stack Quora

Need help with testing, WordPress or SEO?

We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.

Keep reading

More guides on testing, AI & modern development

Enjoyed this article? Pick a topic and jump straight into our newest tutorials.

Artificial Intelligence

Decision Tree in Python: Gini, Entropy and Overfitting From Scratch

Decision tree in Python from scratch: Gini impurity, entropy, information gain, best splits, readable rules, and how depth and leaf…

Read article →
Software Development

URL Query Parameters: Encoding, URLSearchParams and Common Traps

URL query parameters explained: reading and building them with URLSearchParams and urllib, percent encoding, plus signs, repeated keys and nine…

Read article →
DevOps

Git Blame Explained: Find Who Changed a Line, and Why

Git blame explained with a real repository: ignore whitespace, line ranges and functions, ignore-revs files for formatting commits, moved code…

Read article →
Test Automation

Pytest Benchmark: Measure Speed and Catch Performance Regressions

Pytest benchmark tutorial: the benchmark fixture, reading results, parametrized and pedantic benchmarks, saving baselines and failing the build on regressions.

Read article →
Software Testing

Defect Density: Formula, Worked Examples and How to Use It

Defect density explained: the formula, a five-module worked example, per KLOC vs per function point, severity weighting, defect removal efficiency…

Read article →
Databases & SQL

Case Insensitive Search in SQL: LOWER, LIKE, COLLATE and Indexes

Case insensitive search in SQL: LOWER, LIKE, COLLATE NOCASE and expression indexes compared with query plans, plus accented letters and…

Read article →
Databases & SQL

ON DELETE CASCADE in SQL: CASCADE vs SET NULL vs RESTRICT

ON DELETE CASCADE in SQL tested: CASCADE vs SET NULL vs RESTRICT, multi-level cascades, the SQLite foreign key pragma, row…

Read article →
Software Development

JavaScript BigInt Explained: Big Numbers, Division and JSON

JavaScript BigInt explained: the 2^53 limit, creating and converting BigInt, division, mixing types, JSON stringify and parse without losing digits,…

Read article →
Software Development

Python Multiprocessing: Pool, Processes, Queues and Real Timings

Python multiprocessing timed: Pool vs a loop, start-up overhead, chunksize, the main guard, exceptions, shared values with locks, queues and…

Read article →
Software Development

Python __slots__ Explained: Memory Savings, Speed and 6 Traps, Tested

Python __slots__ explained and measured on Python 3.13: memory savings, attribute speed, dataclass slots=True, and six traps including inheritance, weakref…

Read article →
Artificial Intelligence

Logistic Regression in Python: Sigmoid, Log Loss and Gradient Descent From Scratch

Logistic regression in Python from scratch: sigmoid, log loss, gradient descent, odds ratios, thresholds, sigmoid overflow and separable data, tested…

Read article →
Software Development

Content Security Policy Explained: Nonces, strict-dynamic and Report-Only

Content security policy explained: nonces, strict-dynamic, Report-Only, eval, image and style rules, tested in a real browser, plus a strict…

Read article →
DevOps

Git Worktree Explained: Work on Two Branches at Once

Git worktree explained: add, list, remove, prune, lock and detached worktrees, a hotfix workflow, and the errors you will hit,…

Read article →
Test Automation

Pytest ini Explained: addopts, markers, testpaths and Precedence

Pytest ini explained: testpaths, addopts, strict markers, xfail_strict, filterwarnings, minversion, config file precedence and command-line overrides, tested.

Read article →
Software Testing

Test Estimation Techniques: WBS, Three Point PERT and Monte Carlo

Test estimation techniques worked through: WBS, three point PERT, total uncertainty, a Monte Carlo check, percentage rules and Wideband Delphi.

Read article →
Databases & SQL

Gaps and Islands in SQL: Find Streaks and Missing Ranges

Gaps and islands in SQL: find login streaks and missing ranges with ROW_NUMBER, DENSE_RANK, LAG and LEAD, plus the duplicate…

Read article →
Databases & SQL

INSERT INTO SELECT in SQL: Copy Rows Between Tables Safely

INSERT INTO SELECT in SQL: copy rows between tables with chosen columns, joins and summaries, avoid duplicates on reruns, upsert,…

Read article →
Software Development

JavaScript Proxy Explained: All 13 Traps, Reflect and Real Use Cases

JavaScript Proxy explained: all 13 traps, Reflect, validation, defaults, hidden keys, revocable proxies, private field problems and the speed cost,…

Read article →
Artificial Intelligence

Linear Regression in Python: Least Squares From Scratch, With R Squared

Linear regression in Python from scratch: closed form, NumPy least squares and gradient descent, R squared, RMSE, outliers and curved…

Read article →
DevOps

Git Remote Explained: Add, Set-URL, Rename, Remove and Push URLs

Git remote explained: add, set-url, separate push URLs, rename, remove and prune, plus how to avoid leaking tokens in remote…

Read article →
Test Automation

Pytest Mock: mocker.patch, spy, side_effect and autospec Explained

Pytest mock explained: mocker.patch, return_value, side_effect, spy, autospec and PropertyMock, with tested examples for APIs, clocks and retries.

Read article →
Software Testing

System Testing Explained: Types, Process and a Real End-to-End Example

System testing explained: types, entry and exit criteria, the process, and a real end-to-end test suite run against a small…

Read article →
Databases & SQL

SQL Random Rows: How to Select Random Rows Fast and Fairly

SQL random rows: how to select random rows fast and fairly, timing ORDER BY RANDOM, offsets and ID jumps, and…

Read article →
Databases & SQL

SQL Median: 4 Ways to Calculate the Median, With Tested Queries

SQL median explained: four tested ways to calculate the median, including percentile functions, window ranks, offsets and per group medians.

Read article →
Artificial Intelligence

Naive Bayes Classifier Explained: Build a Spam Filter in Python

Naive Bayes classifier explained by building a spam filter in Python: Bayes rule, the naive assumption, Laplace smoothing, log probabilities…

Read article →
DevOps

Git Branch Explained: Create, Switch, Rename, Delete and Recover

Git branch explained with tested commands: list, create, switch, rename, delete, merged checks, recovering deleted branches, remote tracking and naming.

Read article →
Test Automation

Pytest raises: Testing Exceptions, match and excinfo

Pytest raises explained with tested examples: match regexes, excinfo, custom and chained exceptions, parametrized error cases, pytest.warns and common mistakes.

Read article →
Software Testing

Non Functional Testing: Types, Metrics and a Real Load Test

Non functional testing explained: 10 key types, measurable requirements, latency percentiles and what a small real load test revealed about…

Read article →
Artificial Intelligence

K Means Clustering Explained: How It Works, With Python Code

K means clustering explained: the algorithm, a from-scratch NumPy version, choosing k with the elbow and silhouette methods, k-means++, scaling…

Read article →
DevOps

Git Log Explained: 15 Useful Commands and Pretty Formats

Git log explained with 15 tested commands: oneline, graph, filters by author, date, message and code, file history, stats, pretty…

Read article →
Test Automation

Pytest conftest.py Explained: Shared Fixtures, Hooks and Overrides

Pytest conftest.py explained with a tested suite: shared fixtures, scope and teardown, folder overrides, autouse, custom options, markers and useful…

Read article →
Software Testing

Usability Testing: Methods, Metrics, SUS Score and Checklist

Usability testing explained: methods, planning, writing tasks, running sessions, metrics, SUS scoring, how many users you need and a practical…

Read article →
Artificial Intelligence

K Nearest Neighbors (KNN) Explained: How It Works, With Python Code

K nearest neighbors explained: the KNN algorithm, a from-scratch NumPy version, choosing k, feature scaling, distance metrics, regression and vector…

Read article →
DevOps

Git Rebase Interactive: Squash, Fixup, Reword and Autosquash

Git rebase interactive explained step by step: pick, reword, squash, fixup, edit, drop, autosquash, conflicts and undoing a rebase with…

Read article →
Software Testing

Alpha vs Beta Testing: Key Differences, Exit Criteria and Checklist

Alpha vs beta testing compared: who tests, what each phase finds, entry and exit criteria, how to run a beta,…

Read article →
Test Automation

Pytest Monkeypatch: 10 Tested Examples for Env Vars, Time and APIs

Pytest monkeypatch explained with 10 passing tests: environment variables, functions, config, time, sleeps, paths, blocking the network and the wrong-target…

Read article →