Linear Regression in Python: Least Squares From Scratch, With R Squared
Linear regression fits a straight line through data to predict a number from one or more inputs. Learn what the slope and intercept mean, three ways to compute them in NumPy (closed form, least squares and gradient descent), how to judge the fit with R squared, RMSE and residuals, and where linear regression breaks, with every number from code we ran.
- Linear regression finds the line
y = intercept + slope xthat minimises the sum of squared errors. - Our closed form,
np.linalg.lstsqand the normal equation all gave intercept 45.645 and slope 3.186. - Gradient descent needed 200,000 steps on raw data but only 1,000 after scaling the input.
- One extreme point moved the slope from 3.186 to 4.742, and a straight line on curved data scored R squared 0.942 versus 0.993 with an x squared feature.
How much will sales rise if we spend more on ads? How does delivery time grow with distance? How does price change with floor area? Linear regression answers questions like these by fitting the best straight line through observed data. It is the simplest predictive model, the foundation of much of statistics and machine learning, and still the first model to try when the target is a number.
We implemented it three ways in Python 3.13.6 with NumPy 2.5.5 on a synthetic, seeded dataset, and every number below comes from those runs. We did not use scikit-learn, though it gives the same answers in one line.
What is linear regression?
Simple linear regression models a target y as a straight-line function of one input x:
y = b0 + b1 * x + error
b0 is the intercept, the predicted y when x is zero, and b1 is the slope, how much y changes per unit of x. Ordinary least squares picks the b0 and b1 that make the sum of squared vertical distances between the points and the line as small as possible. With several inputs it becomes multiple linear regression, the same idea in more dimensions.
Our dataset: ad spend and sales
rng = np.random.default_rng(42)
x = rng.uniform(10, 100, 60) # monthly ad spend, thousand rupees
y = 50 + 3.2 * x + rng.normal(0, 25, 60) # sales with random noise
We generated 60 months of synthetic data from a known truth: intercept 50 and slope 3.2, plus noise. Knowing the true values lets us check how well each method recovers them.
Method 1: the closed-form formula
xm, ym = x.mean(), y.mean()
b1 = ((x - xm) * (y - ym)).sum() / ((x - xm) ** 2).sum()
b0 = ym - b1 * xm
# closed form: intercept 45.64, slope 3.186
For one input, least squares has an exact answer: the slope is the covariance of x and y divided by the variance of x, and the line always passes through the point of means. The result, slope 3.186 and intercept 45.64, is close to the true 3.2 and 50; the gap comes from the random noise in only 60 points.
Method 2: least squares with NumPy
X = np.c_[np.ones_like(x), x] # add a column of ones for the intercept
np.linalg.lstsq(X, y, rcond=None)[0] # [45.645 3.186]
np.linalg.solve(X.T @ X, X.T @ y) # normal equation: [45.645 3.186]
Writing the problem as matrices generalises to any number of inputs: add one column per feature. lstsq uses a numerically stable decomposition, while the normal equation solves (X^T X) b = X^T y directly. Both matched the closed form exactly. Prefer lstsq; the normal equation can lose precision when features are highly correlated.
Method 3: gradient descent
w = np.zeros(2); lr = 0.0001
for i in range(200000):
g = 2 / len(y) * X.T @ (X @ w - y) # gradient of mean squared error
w -= lr * g
# gradient descent after 200k steps: [45.584 3.186]
Gradient descent starts from zero and repeatedly steps downhill on the error surface. It is how neural networks are trained, including the output layers that end in a softmax function, and it scales to huge datasets where matrix methods do not fit in memory. Here, after 200,000 steps the slope matched, but the intercept, 45.584, still had not quite converged.
Why scaling the input speeds up gradient descent
mu, sd = x.mean(), x.std()
Xz = np.c_[np.ones_like(x), (x - mu) / sd] # standardised input
w2 = np.zeros(2)
for i in range(1000):
w2 -= 0.1 * (2 / len(y) * Xz.T @ (Xz @ w2 - y))
# scaled GD 1000 steps -> slope 3.186 intercept 45.64
On the raw data, ad spend values of 10 to 100 made the error surface long and narrow, so the learning rate had to be tiny and progress along the intercept direction was slow. Standardising the input made the surface round, allowing a learning rate 1,000 times larger: 1,000 steps converged fully, versus 200,000 that did not. Converting back gave the same slope and intercept as the exact methods. Scaling is equally important for distance-based models such as k nearest neighbors.
Evaluating the fit: R squared, RMSE and MAE
R^2 0.949 RMSE 18.70 MAE 15.53
residual mean ~0: -0.0
- R squared (0.949): the share of variation in sales explained by ad spend. 1 is perfect, 0 means no better than predicting the mean.
- RMSE (18.70): typical prediction error in the target’s units, penalising large errors more. It came out close to the noise level of 25 we added, minus what the fit absorbed.
- MAE (15.53): the average absolute error, easier to explain and less sensitive to outliers.
Least squares residuals always average to zero when an intercept is included, as ours did. Plotting residuals against x is the best quick check: a random cloud is good, a curve or funnel shape means the model is missing something.
Making predictions, and the danger of extrapolation
predict spend 80 -> 300.5
predict spend 300 -> 1001.3 (extrapolation)
Within the range of the data (10 to 100), predictions are reasonable. At 300, three times the largest spend we ever observed, the line confidently predicts 1,001, but nothing in the data supports that: ad returns usually flatten as markets saturate. Linear regression has no idea where its data ends. Only predict within, or close to, the range you trained on.
Tip: always report the input range of your training data next to a regression model, so nobody uses it far outside that range.
Outliers pull the line
one outlier changes slope from 3.186 to 4.742
We added a single bad point, a month with sales of 2,000 at spend 95, perhaps a data entry error. The slope jumped by about 49 percent. Because errors are squared, one distant point can dominate the fit. Check for outliers before fitting, investigate them rather than deleting them blindly, and consider robust alternatives such as Huber regression or modelling the median when outliers are real.
When the relationship is curved
straight line on curved data R^2: 0.942
with x^2 feature R^2: 0.993 coefs [-2.62 1.49 1.85]
On data generated from y = 2 x^2 + noise, a straight line still scored 0.942, which looks good, but its residuals would show a clear curve. Adding x^2 as a second feature raised R squared to 0.993. This is still linear regression, because the model is linear in its coefficients, and it is how polynomial and many feature-engineered models work. The squared-term coefficient, 1.85, sits near the true 2; the other terms absorbed noise.
Assumptions behind linear regression
- Linearity: the relationship is a straight line in the chosen features.
- Independent errors: one month’s error does not predict the next; time series often break this.
- Constant variance: errors are similar in size across the range of x.
- No extreme outliers dominating the fit.
- For multiple regression: inputs are not near-duplicates of each other, which makes coefficients unstable.
Predictions can still be useful when assumptions are bent, but confidence intervals and p-values depend on them.
Where linear regression is used
Forecasting sales from marketing spend, estimating prices from size and location, capacity planning from traffic growth, and as a baseline before trying complex models. When the target is a category rather than a number, logistic regression and classifiers such as naive Bayes take over; for unlabelled grouping see k means clustering. Similar linear algebra also powers vector similarity, as in cosine similarity between embeddings.
Final thoughts
Linear regression is a straight line with a lot of power: a closed-form solution, a matrix solution that scales to many features and a gradient-descent version that scales to huge data all gave us the same slope of 3.186. Judge the fit with R squared, RMSE and residual plots, scale inputs for gradient descent, watch for outliers, add features for curves, and never trust predictions far outside your data. Master it, and every more advanced model becomes easier to understand.
Frequently asked questions
What is linear regression in simple terms?
It fits the straight line that best matches your data, so you can predict a number from one or more inputs. Best means the smallest total of squared differences between the line and the points.
How do I do linear regression in Python without libraries?
Compute the slope as the covariance of x and y divided by the variance of x, and the intercept as the mean of y minus the slope times the mean of x. NumPy makes this a few lines.
What is a good R squared value?
It depends on the field. Physical measurements often exceed 0.9, while human behaviour data may be useful at 0.3. Always check residuals too, since a high R squared can hide a curved pattern.
Why use gradient descent if there is an exact formula?
The exact solution becomes expensive or impossible with very large data or many features. Gradient descent scales well and is the same method used to train neural networks.
Is polynomial regression still linear regression?
Yes. Adding x squared or other transformed features keeps the model linear in its coefficients, so the same least squares methods fit it.
Keep learning
Need help with testing, WordPress or SEO?
We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.