Least Squares: How to Draw the "Best" Straight Line Through Scattered Points

⓪ The Problem: A Straight Line Through Scattered Points

Suppose you ran 5 experiments, each time recording input x and output y, obtaining these 5 data points:

You want to find a straight line y = ax + b, to make it "best" pass through these points. But what does "best" mean? Pass through which points? Stay as close as possible to all of them?

⚠️ Here's the Problem

With three points, it's already impossible for one straight line to pass through all of them — unless they happen to be collinear. So "pass through" isn't literal; it means "as close as possible." But how do you quantify "closeness"? This entire article is about computing ? that.

🤔 Notation

We have n data points:(x₁, y₁), (x₂, y₂), …, (xₙ, yₙ).
The line we seek is ŷ = ax + b (read "y-hat"), a is theSlope, b is theIntercept.
For each xᵢ, the predicted value from the line is ŷᵢ = a·xᵢ + b.

① Exploration: The Two-Point World

Let's start with the simplest case:only two data points.

ŷ = a·x + b
Passes exactly through both points

Two points on a plane determine a unique line — this is middle-school geometry. The solution to the system of equations isexact, with zero residual.

💡 Key Fact ①

Two points → two equations → two unknowns (a, b) → exactly solvable. The line passes perfectly through both points.

In mathematics this is called a "well-posed problem" (number of equations = number of unknowns). But real data rarely has only two points.

② The Third Point Arrives — Overdetermined!

Now add a third point (3, 3), along with the first two points (1, 2), (2, 3.5), all together:

Look:no single line can pass through all three points simultaneously. The number of equations (3) exceeds the number of unknowns (2) — this is called an "overdetermined system" — usually unsolvable.

⚠️ Exact Solutions Won't Work

3 equations, 2 unknowns → no solution. We must accept that "the line won't pass through all points" and instead seek a compromise that is "best overall."

🤔 The Core Question

What is the criterion for "best overall"? The answer lies in the diagram below —vertical dashed lines.

③ Discovery 1: Error Is the "Vertical Distance"

For any candidate line ŷ = ax + b, define theresidual (residual):

eᵢ = yᵢ − ŷᵢ = yᵢ − (a·xᵢ + b)

The residual is the length of thevertical dashed lines — thevertical distance (not the perpendicular distance! Because x is a precise input, error is only in the y-direction).

💡 Key Fact ②

Each candidate line corresponds to a set of residuals {e₁, e₂, …, eₙ}. The "better" the line, the smaller the residuals overall. So finding the "best line" = finding (a, b) that makes the residuals "smallest overall."

④ Discovery 2: Why Square?

Now combine the residuals into a single number representing "total error." Here are 4 natural candidates:

Method	Expression	Problem
① Direct sum	Σ eᵢ	Signs cancel out: +3 and −3 sum to 0, appearing "perfect"
② Sum of absolute values	Σ \|eᵢ\|	Not everywhere differentiable, mathematically awkward
③ ③ Sum of squares	Σ eᵢ²	Solves all the above problems
④ Sum of fourth powers	Σ eᵢ⁴	Overly sensitive to outliers, plus heavier computation

🤔 Why squares in particular? Three reasons

① Signs can't cancel: after squaring, all errors become positive, and large errors are amplified and penalized.

② Differentiable everywhere: quadratic functions are smooth — we can confidently use calculus to find extrema.

③ Mathematically optimal: if errors follow a normal distribution, minimizing the sum of squares is equivalent to maximum likelihood estimation — proved by Gauss in 1809.

💡 Key Fact ③

Define SSE (Sum of Squared Errors):

SSE(a, b) = Σ(yᵢ − a·xᵢ − b)²

The task of the least squares method is: find a and b that minimize SSE.

⑤ Discovery 3: Translating Intuition into Math

SSE(a, b) is abivariate quadratic function. Our task is:

Find a and b, such that SSE(a, b) = Σ(yᵢ − a·xᵢ − b)² reaches its minimum

💡 The Key Mindset Shift

"Drawing a good line" = "Minimizing the sum of squared errors" = "Finding the minimum of a bivariate function." The latter two are precise mathematical problems — with standard solution methods!

How do you find the minimum of a bivariate quadratic function? Calculus tells us:Partial derivatives = 0.

SSE(a,b) is an upward-opening paraboloid; the minimum is at the bottom of the valley.

⑥ Derivation: Partial Derivatives → Normal Equations

Expand SSE, then take partial derivatives with respect to a and b:

①

Expand SSE:

SSE = Σ(yᵢ² − 2a xᵢyᵢ − 2b yᵢ + a²xᵢ² + 2ab xᵢ + b²)

②

Partial derivative w.r.t. b, set = 0:

∂SSE/∂b = Σ(−2yᵢ + 2a xᵢ + 2b) = 0
⇒ a·Σxᵢ + n·b = Σyᵢ

③

Partial derivative w.r.t. a, set = 0:

∂SSE/∂a = Σ(−2xᵢyᵢ + 2a xᵢ² + 2b xᵢ) = 0
⇒ a·Σxᵢ² + b·Σxᵢ = Σxᵢyᵢ

✅ Normal Equations

a·Σxᵢ² + b·Σxᵢ = Σxᵢyᵢ
a·Σxᵢ + b·n = Σyᵢ

Two equations, two unknowns — exactly solvable! Starting from an "unsolvable" overdetermined system, we've found a definite solution that is "optimal in the least-squares sense."

Solve this system of two linear equations to get the classic formulas:

a = n·Σxᵢyᵢ − (Σxᵢ)(Σyᵢ)n·Σxᵢ² − (Σxᵢ)² b = (Σxᵢ²)(Σyᵢ) − (Σxᵢ)(Σxᵢyᵢ)n·Σxᵢ² − (Σxᵢ)²

🤔 Equivalent Form (Easier to Remember)

Using means x̄ = Σxᵢ/n, ȳ = Σyᵢ/n:

a = Σ(xᵢ − x̄)(yᵢ − ȳ) / Σ(xᵢ − x̄)²
b = ȳ − a·x̄

Slope a = "covariance of x and y / variance of x." Intercept b ensures the line passes through the centroid (x̄, ȳ).

⑦ Compute It by Hand: 5 Data Points

Return to the 5 points from the beginning: (1, 2), (2, 3), (3, 5), (4, 4), (5, 6). Plug them into the formulas step by step:

i	xᵢ	yᵢ	xᵢ²	xᵢyᵢ
1	1	2	1	2
2	2	3	4	6
3	3	5	9	15
4	4	4	16	16
5	5	6	25	30
Σ	15	20	55	69

①

Plug into formula for a:

a = (5×69 − 15×20) / (5×55 − 15²)
= (345 − 300) / (275 − 225)
= 45 / 50 = 0.9

②

Plug in for b:

b = ȳ − a·x̄ = (20/5) − 0.9×(15/5)
= 4 − 0.9×3 = 4 − 2.7 = 1.3

③

Verify SSE:

i	yᵢ	ŷᵢ=0.9xᵢ+1.3	eᵢ	eᵢ²
1	2	2.2	−0.2	0.04
2	3	3.1	−0.1	0.01
3	5	4.0	+1.0	1.00
4	4	4.9	−0.9	0.81
5	6	5.8	+0.2	0.04
SSE = Σeᵢ² =			1.90

✅ Note the Result

Optimal line:ŷ = 0.9x + 1.3, SSE = 1.90.
No other line can produce a smaller sum of squared errors than 1.90.
(Don't believe it? Go to §⑩ Interactive Lab and drag the sliders — deviate from 0.9 or 1.3, and SSE will definitely increase.)

Part 2 Deep Dive & Generalization

From two-variable linear equations to matrices, from computation to geometry, from one line to all of machine learning.

⑧ Generalization: Matrix Form

Write the n equations yᵢ ≈ a·xᵢ + b in matrix form:

y ≈ Xβ

Observation vector

y = [y₁ y₂ … yₙ]ᵀ

Design matrix

X = [x₁ 1; x₂ 1; …; xₙ 1]

Parameter vector

β = [a b]ᵀ

The objective becomes:Minimize ‖y − Xβ‖²(squared vector length = sum of squared components = SSE).

①

Expand: ‖y − Xβ‖² = (y − Xβ)ᵀ(y − Xβ) = yᵀy − 2βᵀXᵀy + βᵀXᵀXβ

②

Take gradient w.r.t. β, set = 0:

∂/∂β [ yᵀy − 2βᵀXᵀy + βᵀXᵀXβ ] = −2Xᵀy + 2XᵀXβ = 0

③

Yields:

XᵀX β = Xᵀy

This is the same thing as the "Normal Equations" from §⑥ — just expressed in matrix language.

✅ Matrix-Form Solution (when XᵀX is invertible)

β̂ = (XᵀX)⁻¹ Xᵀy

This formula expresses the entire least squares method in just 4 matrix operations! It applies to any number of independent variables (multiple linear regression), not just a single line.

⑨ Geometric Insight: Projection onto Column Space

The matrix form reveals an elegant geometric interpretation that is deeper than the calculus derivation:

💡 Core Insight: Orthogonal Projection

ŷ = Xβ̂ is the observation vector y projected onto X'scolumn spaceas anorthogonal projection.
The residual vector e = y − ŷ isperpendicular (orthogonal) to the column space — this is the geometric essence of "minimum": the hypotenuse of a right triangle is the shortest path.

Orthogonality implies:Xᵀe = 0 ⇒ Xᵀ(y − Xβ̂) = 0 ⇒ XᵀXβ̂ = Xᵀy.

See — the Normal Equations didn't "happen" to come from calculus; they are a direct translation of the geometric fact that "the residual is orthogonal to the column space."

🤔 Why Is This the "Best"?

In n-dimensional space, the observation vector y is fixed. X's column space is a 2D plane (spanned by the column vectors). y'sshortest distanceto this plane is the perpendicular length. The foot of the perpendicular is β̂ — it is the point in the plane closest to y. No other point is at a smaller distance.

⑩ Interactive Lab: Feel Least Squares Yourself

Drag the sliders to change slope a and intercept b, and watch how SSE changes. Click "Optimal Solution" to return to the least-squares solution.

Slope a: 0.90 Intercept b: 1.30

SSE = 1.90

⚠️ Try It

Change a from 0.9 to 1.5 — how much does SSE increase from 1.90? Then change b from 1.3 to 0 — how much does it increase? You'll find that SSE at a=0.9, b=1.3 really is the smallest. That's the power of least squares: it finds the unique global minimum.

⑪ Cross-Domain Easter Eggs: Least Squares Is Everywhere

Least squares isn't just for statistics — it's the "universal language" of data science. With the same XᵀXβ = Xᵀy, you can:

Polynomial fitting
ŷ = a₀ + a₁x + a₂x² + …
Just add columns to the design matrix

Physics experiments
Spring: F = k·x
Measure multiple (x,F) pairs to fit k

Linear regression / Machine learning
Least squares = supervised learning with MSE loss

And many more equivalent problems
Every "optimal approximate solution to an overdetermined system" is fundamentally least squares

💡 This Is the Beauty of Mathematical "Reuse"

Same XᵀXβ = Xᵀy: spring constant = slope a, GDP growth rate = slope a, house-price-to-area coefficient = slope a. Physical laws, economic patterns, machine learning models — the kernel is the same mathematics.

⑫ Complete Map: You Derived Least Squares From Scratch

Look back at the question mark from Section ⓪ — that "?" has now become precise numbers:ŷ = 0.9x + 1.3, SSE = 1.90.

⓪ Problem: Scattered points → find the "best" line

①② Exploration: 2 points solvable → 3 points unsolvable, compromise needed

③④ Translation: Error = vertical distance → sum of squares SSE

⑤⑥ Solution: ∂SSE/∂a = ∂SSE/∂b = 0 → Normal Equations

⑦ Verification: Plug in 5 points → a=0.9, b=1.3, SSE=1.90 ✓

⑧ Generalization: Matrix form XᵀXβ = Xᵀy → β̂ = (XᵀX)⁻¹Xᵀy

⑨ Insight: Orthogonal projection → residual ⟂ column space

⑩⑪ Extension: Interactive verification + cross-domain mapping (physics · economics · ML)

✅ You Can Now…

Start from a scatter plot and write SSE(a, b) = Σ(yᵢ − axᵢ − b)²
Take partial derivatives w.r.t. a and b to derive the Normal Equations
Compute the optimal slope and intercept from concrete data
Understand the origin of the matrix form β̂ = (XᵀX)⁻¹Xᵀy
Explain in geometric terms: ŷ is the orthogonal projection of y onto the column space
Recognize that every "optimal approximation to an overdetermined system" = least squares

Part 1 From Intuition to Formulas

⓪ The Problem: A Straight Line Through Scattered Points

① Exploration: The Two-Point World

② The Third Point Arrives — Overdetermined!

③ Discovery 1: Error Is the "Vertical Distance"

④ Discovery 2: Why Square?

⑤ Discovery 3: Translating Intuition into Math

⑥ Derivation: Partial Derivatives → Normal Equations

⑦ Compute It by Hand: 5 Data Points

Part 2 Deep Dive & Generalization

⑧ Generalization: Matrix Form

⑨ Geometric Insight: Projection onto Column Space

⑩ Interactive Lab: Feel Least Squares Yourself

⑪ Cross-Domain Easter Eggs: Least Squares Is Everywhere

⑫ Complete Map: You Derived Least Squares From Scratch