Logistic Regression in Python: Sigmoid, Log Loss and Gradient Descent From Scratch
Logistic regression predicts the probability of a yes or no outcome: will a student pass, will a customer churn, is this email spam? We build it from scratch in NumPy, train it with gradient descent on log loss, read the coefficients as odds, tune the decision threshold, and run into the two classic failures: sigmoid overflow and perfectly separable data.
- Logistic regression passes a linear score through the sigmoid function to get a probability between 0 and 1.
- Trained with gradient descent on log loss, our model found the true 5-hour pass boundary at 5.04 hours and scored 90.5 percent accuracy.
- Each extra hour of study multiplied the odds of passing by 3.67.
- On perfectly separable data the weights kept growing (slope 2.14, then 4.0, then 6.21); L2 regularisation held it at 1.22.
Many real questions have a yes or no answer: pass or fail, fraud or genuine, click or no click. Logistic regression is the standard first model for them. It is fast, easy to explain to non-specialists, gives calibrated probabilities rather than bare labels, and often performs surprisingly close to far more complex models on tabular data.
We implemented it from scratch with NumPy 2.5.5 on Python 3.13.6, using a synthetic, seeded dataset so we know the true answer. Every number below is from those runs. We did not use scikit-learn, whose LogisticRegression does the same job with extra solvers and regularisation by default.
What is logistic regression?
Despite the name, logistic regression is a classification model. It computes a linear score, exactly as linear regression does, then squashes it into a probability:
z = b0 + b1 * x # linear score, any real number
p = 1 / (1 + exp(-z)) # sigmoid: probability between 0 and 1
predict 1 if p >= 0.5 else 0
Training finds the weights b0 and b1 that make the predicted probabilities match the observed outcomes as closely as possible.
Why not just use linear regression on 0 and 1?
b = np.linalg.lstsq(X, passed, rcond=None)[0]
linear reg on labels: min -0.2 max 1.197 -> outside 0..1
Fitting a straight line to 0/1 labels produced “probabilities” of -0.2 and 1.197, which make no sense. A straight line also lets extreme points pull the fit, and treats a prediction of 2 for a positive example as an error. The sigmoid fixes all of this by bending the line into an S shape that never leaves the 0 to 1 range.
The sigmoid function
def sigmoid(z): return 1 / (1 + np.exp(-z))
sigmoid at -5, -1, 0, 1, 5: [0.0067, 0.2689, 0.5, 0.7311, 0.9933]
A score of 0 maps to exactly 0.5, large positive scores approach 1 and large negative ones approach 0. The function is symmetric: sigmoid(1) and sigmoid(-1) add up to 1. For more than two classes the same role is played by the softmax function, of which sigmoid is the two-class special case.
Our dataset: study hours and exam results
rng = np.random.default_rng(42)
hours = rng.uniform(0, 10, 200)
true_p = 1 / (1 + np.exp(-(1.2 * hours - 6))) # true boundary at 5 hours
passed = (rng.uniform(size=200) < true_p).astype(int)
# pass rate 0.49
200 students, each with a pass probability that rises smoothly with study hours. The true model has slope 1.2 and intercept -6, so the 50 percent point is exactly 5 hours. Because outcomes are random draws, some students who studied 8 hours still failed, just as in real data.
Log loss: how wrong is a probability?
def logloss(w):
p = np.clip(sigmoid(X @ w), 1e-12, 1 - 1e-12)
return -np.mean(passed * np.log(p) + (1 - passed) * np.log(1 - p))
# loss at start (all weights 0): 0.6931
Log loss, also called binary cross-entropy, punishes confident wrong answers heavily: predicting 0.99 for a student who failed costs far more than predicting 0.6. With all weights at zero, every prediction is 0.5 and the loss is ln 2, which is 0.6931, the value we got. Squared error is not used here because combined with the sigmoid it creates a non-convex surface; log loss is convex, so gradient descent cannot get stuck in a local minimum. The np.clip keeps log(0) from producing infinity.
Training with gradient descent
w = np.zeros(2)
for i in range(20001):
p = sigmoid(X @ w)
g = X.T @ (p - passed) / n # gradient of log loss
w -= 0.1 * g
step 0 loss 0.6228 w [-0.001 0.108]
step 100 loss 0.4325 w [-1.329 0.363]
step 1000 loss 0.2674 w [-4.62 0.945]
step 5000 loss 0.2543 w [-6.379 1.268]
step 20000 loss 0.2542 w [-6.557 1.301]
The gradient has a neat form: the inputs times the prediction errors p - y, the same shape as in linear regression. The loss fell quickly, then flattened: between step 5,000 and 20,000 it improved by only 0.0001. The final weights, intercept -6.557 and slope 1.301, are close to the true -6 and 1.2; with only 200 noisy examples, an exact match is not expected. Feature scaling would make convergence faster, as we showed with gradient descent in the linear regression article.
The decision boundary
decision boundary hours 5.04 (true 5.0)
P(pass | 2 h) = 0.019
P(pass | 4 h) = 0.205
P(pass | 5 h) = 0.487
P(pass | 6 h) = 0.777
P(pass | 8 h) = 0.979
The boundary is where the score is zero, so the probability is 0.5: hours = -b0 / b1 = 5.04, almost exactly the true value. The probabilities are the real value of the model: two hours of study gives about a 2 percent chance of passing, eight hours about 98 percent. A teacher can act on “this student has a 20 percent chance” in a way they cannot act on a bare “fail” label.
Accuracy, precision and recall
accuracy 0.905
tp 87 tn 94 fp 8 fn 11
precision 0.916 recall 0.888
The model classified 181 of 200 students correctly. Precision says that of those predicted to pass, 91.6 percent did; recall says that of those who passed, the model found 88.8 percent. Because the data is random around the boundary, about 90 percent is close to the best any model could do here. Always report precision and recall as well as accuracy, especially when one class is rare.
Reading coefficients as odds
odds ratio per hour = exp(1.301) = 3.67
This is why logistic regression is popular in medicine, finance and social science. The odds of passing are p / (1 – p). Each additional hour of study multiplied those odds by 3.67. A coefficient of 0 would mean an odds ratio of 1, no effect; a negative coefficient would shrink the odds. That interpretability is something tree ensembles and neural networks cannot offer as directly.
Choosing the decision threshold
| Threshold | Precision | Recall |
|---|---|---|
| 0.3 | 0.812 | 0.929 |
| 0.5 | 0.916 | 0.888 |
| 0.7 | 0.929 | 0.796 |
0.5 is only a default. Lowering the threshold catches more positives (higher recall) at the cost of more false alarms (lower precision); raising it does the opposite. For fraud or disease screening, where missing a case is expensive, a low threshold makes sense. For sending a costly intervention, a high one does. Pick the threshold from the business cost of each error type, not from habit.
Pitfall 1: sigmoid overflow
naive sigmoid(-1000) -> [0.] with RuntimeWarning (overflow in exp)
stable sigmoid(-1000) -> [0.] stable sigmoid(1000) -> [1.]
For very negative scores, np.exp(-z) becomes exp(1000), which overflows to infinity. NumPy still returned the right answer, 0, but raised a RuntimeWarning, and in other code paths the overflow can turn into NaN. The stable version only ever computes exp(-abs(z)), which cannot overflow, and handles positive and negative scores separately. Large scores are common with unscaled features, so use the stable form, or scipy.special.expit, in real code.
Pitfall 2: perfectly separable data
x = [1, 2, 3, 4, 6, 7, 8, 9] y = [0, 0, 0, 0, 1, 1, 1, 1]
separable step 1000 w [-10.47 2.14]
separable step 10000 w [-19.8 4. ]
separable step 100000 w [-30.85 6.21]
with L2 0.1 w [-6.12 1.22]
When one value of x splits the classes perfectly, there is no best answer: making the sigmoid steeper always reduces the loss a little further. The weights grew without limit, from slope 2.14 to 6.21 and still climbing, and the model became absurdly overconfident. Adding an L2 penalty on the slope, which discourages large weights, stopped the growth at a sensible 1.22. That is why scikit-learn applies L2 regularisation by default, and why tiny or clean datasets need it most.
Tip: if your coefficients look huge or keep changing as you train longer, check for a feature that perfectly separates the classes, often a leaked label.
More features and more than two classes
With several inputs, the score becomes b0 + b1 x1 + b2 x2 + ..., and nothing else changes: add columns to X and the same gradient descent code trains it. Scale the features so that gradient descent converges and the coefficients are comparable. For three or more classes, multinomial logistic regression replaces sigmoid with softmax, giving one probability per class. Compared with naive Bayes, logistic regression does not assume features are independent, and compared with k nearest neighbors it produces a compact model that predicts instantly. A single logistic unit is also exactly one neuron of a neural network, the building block behind large language models.
Final thoughts
Logistic regression is a linear model wearing a sigmoid. Train it on log loss with gradient descent, read its probabilities rather than just its labels, interpret coefficients as odds ratios, and choose the threshold from the cost of each kind of mistake. Guard against the two failures we reproduced, overflow in a naive sigmoid and runaway weights on separable data, with a stable sigmoid and regularisation. It remains the baseline every classifier should be compared against.
Frequently asked questions
What is logistic regression used for?
Predicting the probability of a binary outcome, such as pass or fail, churn or stay, fraud or genuine. It is also widely used to explain how each input changes the odds of the outcome.
Why is it called regression if it is used for classification?
It regresses the log odds of the outcome on the inputs with a linear model. The sigmoid then turns those log odds into a probability, and a threshold turns the probability into a class.
What loss function does logistic regression use?
Log loss, also called binary cross-entropy. It is convex when combined with the sigmoid, so gradient descent reliably finds the best weights.
How do I interpret logistic regression coefficients?
Take the exponential of a coefficient to get an odds ratio. In our example the slope of 1.301 gave 3.67, so each extra study hour multiplied the odds of passing by 3.67.
Why do my logistic regression weights keep growing?
Usually because the data is perfectly separable, so a steeper curve always lowers the loss. Add L2 regularisation, which kept our slope at 1.22 instead of growing past 6.
Keep learning
Need help with testing, WordPress or SEO?
We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.