1. Home
  2. Blog
  3. Artificial Intelligence

Artificial Intelligence

Linear Regression in Python: Least Squares From Scratch, With R Squared

Linear regression fits a straight line through data to predict a number from one or more inputs. Learn what the slope and intercept mean, three ways to compute them in NumPy (closed form, least squares and gradient descent), how to judge the fit with R squared, RMSE and residuals, and where linear regression breaks, with every number from code we ran.

9 min read
Linear regression in Python: least squares from scratch, with R squared

Key takeaways

  • Linear regression finds the line y = intercept + slope x that minimises the sum of squared errors.
  • Our closed form, np.linalg.lstsq and the normal equation all gave intercept 45.645 and slope 3.186.
  • Gradient descent needed 200,000 steps on raw data but only 1,000 after scaling the input.
  • One extreme point moved the slope from 3.186 to 4.742, and a straight line on curved data scored R squared 0.942 versus 0.993 with an x squared feature.

How much will sales rise if we spend more on ads? How does delivery time grow with distance? How does price change with floor area? Linear regression answers questions like these by fitting the best straight line through observed data. It is the simplest predictive model, the foundation of much of statistics and machine learning, and still the first model to try when the target is a number.

We implemented it three ways in Python 3.13.6 with NumPy 2.5.5 on a synthetic, seeded dataset, and every number below comes from those runs. We did not use scikit-learn, though it gives the same answers in one line.

What is linear regression?

Simple linear regression models a target y as a straight-line function of one input x:

y = b0 + b1 * x + error

b0 is the intercept, the predicted y when x is zero, and b1 is the slope, how much y changes per unit of x. Ordinary least squares picks the b0 and b1 that make the sum of squared vertical distances between the points and the line as small as possible. With several inputs it becomes multiple linear regression, the same idea in more dimensions.

Our dataset: ad spend and sales

rng = np.random.default_rng(42)
x = rng.uniform(10, 100, 60)                  # monthly ad spend, thousand rupees
y = 50 + 3.2 * x + rng.normal(0, 25, 60)      # sales with random noise

We generated 60 months of synthetic data from a known truth: intercept 50 and slope 3.2, plus noise. Knowing the true values lets us check how well each method recovers them.

Method 1: the closed-form formula

xm, ym = x.mean(), y.mean()
b1 = ((x - xm) * (y - ym)).sum() / ((x - xm) ** 2).sum()
b0 = ym - b1 * xm
# closed form: intercept 45.64, slope 3.186

For one input, least squares has an exact answer: the slope is the covariance of x and y divided by the variance of x, and the line always passes through the point of means. The result, slope 3.186 and intercept 45.64, is close to the true 3.2 and 50; the gap comes from the random noise in only 60 points.

Method 2: least squares with NumPy

X = np.c_[np.ones_like(x), x]                       # add a column of ones for the intercept
np.linalg.lstsq(X, y, rcond=None)[0]                # [45.645  3.186]
np.linalg.solve(X.T @ X, X.T @ y)                   # normal equation: [45.645  3.186]

Writing the problem as matrices generalises to any number of inputs: add one column per feature. lstsq uses a numerically stable decomposition, while the normal equation solves (X^T X) b = X^T y directly. Both matched the closed form exactly. Prefer lstsq; the normal equation can lose precision when features are highly correlated.

Method 3: gradient descent

w = np.zeros(2); lr = 0.0001
for i in range(200000):
    g = 2 / len(y) * X.T @ (X @ w - y)     # gradient of mean squared error
    w -= lr * g
# gradient descent after 200k steps: [45.584  3.186]

Gradient descent starts from zero and repeatedly steps downhill on the error surface. It is how neural networks are trained, including the output layers that end in a softmax function, and it scales to huge datasets where matrix methods do not fit in memory. Here, after 200,000 steps the slope matched, but the intercept, 45.584, still had not quite converged.

Why scaling the input speeds up gradient descent

mu, sd = x.mean(), x.std()
Xz = np.c_[np.ones_like(x), (x - mu) / sd]          # standardised input
w2 = np.zeros(2)
for i in range(1000):
    w2 -= 0.1 * (2 / len(y) * Xz.T @ (Xz @ w2 - y))
# scaled GD 1000 steps -> slope 3.186 intercept 45.64

On the raw data, ad spend values of 10 to 100 made the error surface long and narrow, so the learning rate had to be tiny and progress along the intercept direction was slow. Standardising the input made the surface round, allowing a learning rate 1,000 times larger: 1,000 steps converged fully, versus 200,000 that did not. Converting back gave the same slope and intercept as the exact methods. Scaling is equally important for distance-based models such as k nearest neighbors.

Evaluating the fit: R squared, RMSE and MAE

R^2 0.949   RMSE 18.70   MAE 15.53
residual mean ~0: -0.0
  • R squared (0.949): the share of variation in sales explained by ad spend. 1 is perfect, 0 means no better than predicting the mean.
  • RMSE (18.70): typical prediction error in the target’s units, penalising large errors more. It came out close to the noise level of 25 we added, minus what the fit absorbed.
  • MAE (15.53): the average absolute error, easier to explain and less sensitive to outliers.

Least squares residuals always average to zero when an intercept is included, as ours did. Plotting residuals against x is the best quick check: a random cloud is good, a curve or funnel shape means the model is missing something.

Making predictions, and the danger of extrapolation

predict spend 80  -> 300.5
predict spend 300 -> 1001.3   (extrapolation)

Within the range of the data (10 to 100), predictions are reasonable. At 300, three times the largest spend we ever observed, the line confidently predicts 1,001, but nothing in the data supports that: ad returns usually flatten as markets saturate. Linear regression has no idea where its data ends. Only predict within, or close to, the range you trained on.

Tip: always report the input range of your training data next to a regression model, so nobody uses it far outside that range.

Outliers pull the line

one outlier changes slope from 3.186 to 4.742

We added a single bad point, a month with sales of 2,000 at spend 95, perhaps a data entry error. The slope jumped by about 49 percent. Because errors are squared, one distant point can dominate the fit. Check for outliers before fitting, investigate them rather than deleting them blindly, and consider robust alternatives such as Huber regression or modelling the median when outliers are real.

When the relationship is curved

straight line on curved data R^2: 0.942
with x^2 feature R^2: 0.993   coefs [-2.62  1.49  1.85]

On data generated from y = 2 x^2 + noise, a straight line still scored 0.942, which looks good, but its residuals would show a clear curve. Adding x^2 as a second feature raised R squared to 0.993. This is still linear regression, because the model is linear in its coefficients, and it is how polynomial and many feature-engineered models work. The squared-term coefficient, 1.85, sits near the true 2; the other terms absorbed noise.

Assumptions behind linear regression

  • Linearity: the relationship is a straight line in the chosen features.
  • Independent errors: one month’s error does not predict the next; time series often break this.
  • Constant variance: errors are similar in size across the range of x.
  • No extreme outliers dominating the fit.
  • For multiple regression: inputs are not near-duplicates of each other, which makes coefficients unstable.

Predictions can still be useful when assumptions are bent, but confidence intervals and p-values depend on them.

Where linear regression is used

Forecasting sales from marketing spend, estimating prices from size and location, capacity planning from traffic growth, and as a baseline before trying complex models. When the target is a category rather than a number, logistic regression and classifiers such as naive Bayes take over; for unlabelled grouping see k means clustering. Similar linear algebra also powers vector similarity, as in cosine similarity between embeddings.

Final thoughts

Linear regression is a straight line with a lot of power: a closed-form solution, a matrix solution that scales to many features and a gradient-descent version that scales to huge data all gave us the same slope of 3.186. Judge the fit with R squared, RMSE and residual plots, scale inputs for gradient descent, watch for outliers, add features for curves, and never trust predictions far outside your data. Master it, and every more advanced model becomes easier to understand.

Frequently asked questions

What is linear regression in simple terms?

It fits the straight line that best matches your data, so you can predict a number from one or more inputs. Best means the smallest total of squared differences between the line and the points.

How do I do linear regression in Python without libraries?

Compute the slope as the covariance of x and y divided by the variance of x, and the intercept as the mean of y minus the slope times the mean of x. NumPy makes this a few lines.

What is a good R squared value?

It depends on the field. Physical measurements often exceed 0.9, while human behaviour data may be useful at 0.3. Always check residuals too, since a high R squared can hide a curved pattern.

Why use gradient descent if there is an exact formula?

The exact solution becomes expensive or impossible with very large data or many features. Gradient descent scales well and is the same method used to train neural networks.

Is polynomial regression still linear regression?

Yes. Adding x squared or other transformed features keeps the model linear in its coefficients, so the same least squares methods fit it.

Keep learning

Work with Stack Quora

Need help with testing, WordPress or SEO?

We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.

Keep reading

More guides on testing, AI & modern development

Enjoyed this article? Pick a topic and jump straight into our newest tutorials.

Artificial Intelligence

Confusion Matrix Explained: Precision, Recall, F1 and Thresholds in Python

Confusion matrix explained with Python: TP, FP, FN, TN, precision, recall, F1, the accuracy trap, threshold choice by cost and…

Read article →
Software Development

API Pagination: Offset vs Cursor vs Keyset, Tested on 1 Million Rows

API pagination explained: offset vs keyset and cursor pagination timed on 1 million rows, duplicates when data changes, the ties…

Read article →
DevOps

Git Submodule Explained: Add, Clone, Update and Remove Safely

Git submodule explained: add, clone, update –init, update –remote, the detached HEAD trap, foreach and clean removal, tested step by…

Read article →
Test Automation

Pytest tmp path Explained: tmp_path, tmp_path_factory and Temp Files

Pytest tmp_path explained: per-test temporary folders, tmp_path_factory for shared files, monkeypatch.chdir, legacy tmpdir, retention and basetemp.

Read article →
Software Testing

Test Case Prioritization: 6 Techniques Compared With APFD

Test case prioritization compared: risk, coverage, cost, history and random orders scored with APFD and time to find all faults…

Read article →
Databases & SQL

Full Text Search in SQL: FTS5, Ranking, Phrases and Speed

Full text search in SQL with SQLite FTS5: MATCH queries, phrases, prefixes, NEAR, stemming, BM25 ranking, snippets, two gotchas and…

Read article →
Databases & SQL

Generated Columns in SQL: Virtual vs Stored, Indexes and Limits

Generated columns in SQL explained: virtual vs stored, automatic updates, write errors, storage cost, indexing, JSON fields and support in…

Read article →
Software Development

Object Freeze in JavaScript: Shallow vs Deep, Seal and const

Object freeze in JavaScript explained: freeze vs seal vs const, strict mode errors, shallow vs deep freeze, frozen arrays, Map…

Read article →
Software Development

Python hashlib: SHA-256, File Hashes, HMAC and Password Hashing

Python hashlib explained: SHA-256 for strings and files, algorithm speeds, why passwords need PBKDF2 or scrypt, safe verification and HMAC…

Read article →
Software Development

Python UUID Explained: uuid1, uuid4, uuid5, uuid7 and Database Keys

Python UUID explained: uuid4, uuid5, uuid1 privacy, time-ordered uuid7, parsing, storage formats and UUIDs as database keys, tested on Python…

Read article →
Artificial Intelligence

Random Forest in Python: Bagging, Feature Sampling and OOB From Scratch

Random forest in Python from scratch: bootstrap sampling, feature sampling, number of trees, out-of-bag evaluation and impurity vs permutation importance.

Read article →
Software Development

MIME Types Explained: Content-Type, nosniff and a Common Types List

MIME types explained and tested in a browser: Content-Type, nosniff, scripts and stylesheets with wrong types, a common MIME types…

Read article →
DevOps

Git Clean Explained: Remove Untracked Files Without Regret

Git clean explained with dry runs: -n, -f, -d, -x, -X, exclusions, paths, nested repositories and interactive mode, plus a…

Read article →
Test Automation

Pytest approx: Comparing Floats With rel, abs and Collections

Pytest approx explained: default tolerances, the near-zero trap, rel and abs, lists and dicts, NaN, NumPy, and how it differs…

Read article →
Software Testing

Accessibility Testing: WCAG Checklist, Automated Tools and Manual Checks

Accessibility testing explained: WCAG basics, an axe-core scan of a broken and a fixed page, what automation missed, contrast checks…

Read article →
Databases & SQL

SQL UNPIVOT: Turn Columns Into Rows, 4 Ways Tested

SQL UNPIVOT explained: turn columns into rows with UNION ALL, CROSS JOIN with CASE, JSON functions or native UNPIVOT, with…

Read article →
Software Development

Intl NumberFormat in JavaScript: Currency, Compact, Percent and Units

Intl.NumberFormat in JavaScript: locales, currency, compact notation, percent, units, rounding, formatToParts, the non-breaking space trap and formatter reuse.

Read article →
Databases & SQL

SQL EXPLAIN Plan: How to Read Query Plans and Fix Slow Queries

SQL EXPLAIN plan explained: read scans, index searches, covering indexes and temporary sorts, with before and after timings for five…

Read article →
Artificial Intelligence

Decision Tree in Python: Gini, Entropy and Overfitting From Scratch

Decision tree in Python from scratch: Gini impurity, entropy, information gain, best splits, readable rules, and how depth and leaf…

Read article →
DevOps

Git Blame Explained: Find Who Changed a Line, and Why

Git blame explained with a real repository: ignore whitespace, line ranges and functions, ignore-revs files for formatting commits, moved code…

Read article →
Test Automation

Pytest Benchmark: Measure Speed and Catch Performance Regressions

Pytest benchmark tutorial: the benchmark fixture, reading results, parametrized and pedantic benchmarks, saving baselines and failing the build on regressions.

Read article →
Software Testing

Defect Density: Formula, Worked Examples and How to Use It

Defect density explained: the formula, a five-module worked example, per KLOC vs per function point, severity weighting, defect removal efficiency…

Read article →
Databases & SQL

Case Insensitive Search in SQL: LOWER, LIKE, COLLATE and Indexes

Case insensitive search in SQL: LOWER, LIKE, COLLATE NOCASE and expression indexes compared with query plans, plus accented letters and…

Read article →
Databases & SQL

ON DELETE CASCADE in SQL: CASCADE vs SET NULL vs RESTRICT

ON DELETE CASCADE in SQL tested: CASCADE vs SET NULL vs RESTRICT, multi-level cascades, the SQLite foreign key pragma, row…

Read article →
Artificial Intelligence

Logistic Regression in Python: Sigmoid, Log Loss and Gradient Descent From Scratch

Logistic regression in Python from scratch: sigmoid, log loss, gradient descent, odds ratios, thresholds, sigmoid overflow and separable data, tested…

Read article →
DevOps

Git Worktree Explained: Work on Two Branches at Once

Git worktree explained: add, list, remove, prune, lock and detached worktrees, a hotfix workflow, and the errors you will hit,…

Read article →
Test Automation

Pytest ini Explained: addopts, markers, testpaths and Precedence

Pytest ini explained: testpaths, addopts, strict markers, xfail_strict, filterwarnings, minversion, config file precedence and command-line overrides, tested.

Read article →
Software Testing

Test Estimation Techniques: WBS, Three Point PERT and Monte Carlo

Test estimation techniques worked through: WBS, three point PERT, total uncertainty, a Monte Carlo check, percentage rules and Wideband Delphi.

Read article →
Artificial Intelligence

Linear Regression in Python: Least Squares From Scratch, With R Squared

Linear regression in Python from scratch: closed form, NumPy least squares and gradient descent, R squared, RMSE, outliers and curved…

Read article →
DevOps

Git Remote Explained: Add, Set-URL, Rename, Remove and Push URLs

Git remote explained: add, set-url, separate push URLs, rename, remove and prune, plus how to avoid leaking tokens in remote…

Read article →
Test Automation

Pytest Mock: mocker.patch, spy, side_effect and autospec Explained

Pytest mock explained: mocker.patch, return_value, side_effect, spy, autospec and PropertyMock, with tested examples for APIs, clocks and retries.

Read article →
Software Testing

System Testing Explained: Types, Process and a Real End-to-End Example

System testing explained: types, entry and exit criteria, the process, and a real end-to-end test suite run against a small…

Read article →
Artificial Intelligence

Naive Bayes Classifier Explained: Build a Spam Filter in Python

Naive Bayes classifier explained by building a spam filter in Python: Bayes rule, the naive assumption, Laplace smoothing, log probabilities…

Read article →
DevOps

Git Branch Explained: Create, Switch, Rename, Delete and Recover

Git branch explained with tested commands: list, create, switch, rename, delete, merged checks, recovering deleted branches, remote tracking and naming.

Read article →
Test Automation

Pytest raises: Testing Exceptions, match and excinfo

Pytest raises explained with tested examples: match regexes, excinfo, custom and chained exceptions, parametrized error cases, pytest.warns and common mistakes.

Read article →
Software Testing

Non Functional Testing: Types, Metrics and a Real Load Test

Non functional testing explained: 10 key types, measurable requirements, latency percentiles and what a small real load test revealed about…

Read article →