The Math Behind GPUs — The Complete Journey from Triangles to Pixels

Part 1: From 3D Space to Screen Coordinates

How a triangle vertex flies from the virtual world to your monitor in microseconds

① The Problem: How Is a 3D Scene Actually Computed?

Open any 3D game — every frame you see, every character, building, and shadow — is backed byhundreds of thousands of triangles. The GPU processes these triangles every second, turning them into pixels on your screen.

But how does a triangle's position in the 3D world — say "a vertex at (x=3.7, y=1.2, z=-8.5)" — ultimately become "that pixel at row 847, column 523" on your monitor?

This entire article is about unraveling that ? — the entire GPU math pipeline

⚠ Here's the Problem Your monitor is 2D (only rows and columns), but the 3D world has three dimensions: x, y, z.How do you turn 3D coordinates into 2D pixel positions? And — distant objects look smaller. How does this "near-large, far-small" become mathematics?

② First Stop: The Mathematical Representation of a "Point"

In the 3D world, every object — characters, buildings, clouds — is made of triangle vertices. Let's focus on one vertex first.

The position of a point P in 3D space is represented by three numbers:P = (x, y, z). But a more elegant way is to view it as acolumn vector:

P = xyz

Vectors can represent not just "position," but alsodirection, displacement, normal — these are crucial in subsequent lighting calculations.

Point P(3, 2, 1) — three numbers, corresponding to projections onto the three axes

💡 Key Fact ① One vertex = one 3D column vector. The three vertices of a triangle are three such vectors. What the GPU does to them is fundamentallymath operations on vectors.

③ Discovery 1: Translation — Moving a Point to a New Location

You move a game character: press the right arrow, and the character instantly shifts from (3, 2, 1) to (9, 2, 1). Simple:

x'y'z' = xyz + ΔxΔyΔz

Add atranslation amount to each component, done.

Translation = add an offset to each coordinate component

✅ Note the Result Translation = vector addition.P' = P + T. Simple and intuitive — each of the three components gets its own addition.

⚠ But There's a Catch Translation is addition, but rotation and scaling are multiplication. Three operations, inconsistent forms — every transform requires asking "addition or multiplication this time?" Later we'll use a clever trick to unify them all asmatrix multiplication.

④ Discovery 2: Rotation — The Magic of Matrices

Rotation is more complex than translation. Rotating by θ around the z-axis: x and y coordinates get "mixed together," but z stays unchanged.

x'y'z' = cos θ−sin θ0 sin θcos θ0 001 xyz

This ismatrix × vector. Each row of the matrix pairs up with each component of the vector — multiply and add:

x' = cos θ · x − sin θ · y y' = sin θ · x + cos θ · y

Rotate by θ around the origin —matrix multiplicationdone in one step

💡 Key Fact ② Rotation = matrix × vector. Matrix multiplication is naturally suited for rotation: new coordinates becomelinear combinations (weighted sums) of old coordinates. Scaling is also matrix multiplication — a diagonal matrix.

🤔 Why Use Matrices? Matrix multiplication can "batch up" a pile of operations: rotate then translate? Multiply two matrices to get one new matrix, then one multiplication does it all. This is calledcomposition of transforms — and what GPUs do best is matrix multiplication.

⑤ Discovery 3: Scaling — Power on the Diagonal

Scaling is the simplest: multiply each axis by a factor. In matrix form:

s_x00 0s_y0 00s_z

The three scale factors sit on thediagonal. Matrix multiplication automatically becomes: x' = s_x·x, y' = s_y·y, z' = s_z·z.

Now all three basic transforms are written as matrices:

Transform	Form	Matrix Size
Scaling	matrix × vector	3×3
Rotation	matrix × vector	3×3
Translation	Vector addition	—

⚠ Dissonance Scaling and rotation are 3×3 matrix multiplication, but translation is addition. The three operations can't be merged into one matrix. This is the core problem we'll solve next.

⑥ Discovery 4: Homogeneous Coordinates — Representing 3D Points with 4 Numbers

The solution is surprisingly elegant:Add a "fourth component" to every 3D vector, called w.

P = xyz1

For ordinary 3D points, w = 1; for direction vectors (vectors that shouldn't be translated, like normals), w = 0.

Now translation can also be written as 4×4 matrix multiplication:

x'y'z'1 = 100Δx 010Δy 001Δz 0001 xyz1

Look! The fourth row carries w=1 straight through to the result; and the rightmost column (blue) holds the three translation amounts — during matrix multiplication they get multiplied by w=1 and added to x', y', z'.

💡 Key Fact ③ Homogeneous coordinates (x, y, z, w=1) maketranslation into a single matrix multiplication. From now on, scaling, rotation, and translation are all 4×4 matrix multiplication — unified!

✅ Note the Result The 4×4 homogeneous matrix = the GPU's "universal language." Every spatial transform, regardless of type, is matrix × 4D vector. GPU hardware is designed specifically for this operation — thousands of cores doing matrix multiplication in parallel.

⑦ Discovery 5: MVP — Three Steps Merged into One

A triangle vertex goes from "coordinates in the model file" to "position on screen" through three matrix transforms. The GPU abbreviates them as M·V·P:

① Model Matrix (Model Transform)
Moves vertices from "object-local coordinates" to "world coordinates"
Object → World: includes scaling, rotation, translation

② View Matrix (View Transform)
Moves world coordinates to "camera coordinates"
World → Eye: equivalent to moving the camera to the origin

③ Projection Matrix (Projection Transform)
"Flattens" the 3D scene seen by the camera into 2D clip space
Eye → Screen: near-large far-small is encoded here

Three things, but the GPU does onlyone matrix multiplication:

v_clip = P · V · M · v_local

The GPU first multiplies the three matrices into one big matrix MVP = P·V·M, then does only one 4×4 × 4×1 multiplication per vertex.

🤔 Why Multiply Matrices First? Matrix multiplication satisfiesassociativity: (P·V)·M = P·(V·M). A single frame has hundreds of thousands of vertices — doing three matrix multiplications per vertex is too slow. Multiply the three matrices first (computed once), then each vertex gets multiplied by this big matrix once — saving 67% of the computation.

⑧ Discovery 6: Perspective Projection — How "Near-Large, Far-Small" Becomes Math

This is the most ingenious part of GPU math. Your eye (or camera) is a point; the farther an object, the smaller it appears. How do you encode this effect in a matrix?

Frustum: the pyramid-shaped space seen by the eye

The core formula — the essence of the projection matrix is in its last row:

Projection matrix P = ………… ………… ………… 00−10 → It moves the z-coordinateinto the w component!

💡 Key Fact ④ The projection matrix doesn't directly divide — it hides the "depth information" in the w component. The farther a point is from the camera, the larger its w. The next step —perspective division — is what actually makes distant objects smaller.

⑨ Discovery 7: Perspective Division — The Magic of Dividing by w

After multiplying by the projection matrix, we get a 4D vector (x_c, y_c, z_c, w_c). The final step:

x_ndcy_ndcz_ndc = x_c / w_cy_c / w_cz_c / w_c

Because of the projection matrix's design,w_c ≈ the original z (depth). So:larger z → larger w → divide by larger number → x, y get smaller → distant objects shrink!

Perspective division: performed automatically by GPU hardware — the final step every vertex goes through

✅ Note the Result After the MVP matrix + perspective division, a 3D world vertex (x, y, z) becomesNormalized Device Coordinates (NDC): x, y are in [-1, 1], z is in [0, 1]. Finally mapping to screen pixels (e.g. 1920×1080) is calledviewport transform — a simple linear mapping.

🤔 Why Is NDC [-1, 1]? It makes the GPU resolution-independent. Whether your screen is 720p or 4K, clipping and back-face culling happen in the [-1, 1] NDC space — this is the hardware's fixed "standard ruler."

Part 2: Inside the Triangle — From Outline to Pixels

Three vertices determine positions on screen, but what about the thousands of pixels inside the triangle?

⑩ Discovery 8: Rasterization — Turning Triangles into Pixels

The three vertices have positions on screen (e.g. (100, 200), (350, 100), (280, 400)), but how do you fill the triangle's interior? This isRasterization.

The GPU usesedge functionsto determine whether a pixel is inside or outside the triangle:

E_AB(P) = (B_x − A_x)(P_y − A_y) − (B_y − A_y)(P_x − A_x)

This is the sign of the2D cross product. Compute once per edge; if all three results have the same sign — the pixel is inside the triangle.

Edge function: test all three edges; same sign = inside the triangle

💡 Key Fact ⑤ The edge function is the core formula of rasterization — it turns ageometry problem("is the point inside the triangle?") into analgebra problem("do three expressions have the same sign?"). GPU hardware has dedicated circuits to accelerate this computation.

⑪ Discovery 9: Barycentric Coordinates — The "Share" of Three Vertices

A pixel is inside the triangle, but how are its color, depth, and texture coordinates determined? Answer:Barycentric Coordinates.

P = α · A + β · B + γ · C where α + β + γ = 1, and α, β, γ ≥ 0

α, β, γ are the "weights" of the three vertices. The closer the pixel is to A, the larger α is; at point A, α = 1 and β = γ = 0.

Barycentric coordinates: the area formed by the pixel and the opposite vertex, as a fraction of total area = that vertex's weight

💡 Key Fact ⑥ Any vertex attribute (color, texture coordinates, normal direction) can besmoothly interpolated across the triangle using these three weights. This is why when a triangle has three different colors at its vertices, you see a gradient in the middle.

Barycentric Coordinate Calculator: A 0.33 B 0.33 C 0.34

P = 0.33·A + 0.33·B + 0.34·C (sum = 1.00 ✓)

⑫ Extension 1: Texture Mapping — "Stickers" for Triangles

Each vertex not only has 3D coordinates, but alsotexture coordinates (u, v). u and v are in [0, 1], representing a position on the texture image.

A pixel (x_screen, y_screen) with known weights α, β, γ naturally gets its texture coordinates as the weighted average of the three vertices' texture coordinates:

uv = α · u_Av_A + β · u_Bv_B + γ · u_Cv_C

But there's a trap: perspective transformation distorts linear interpolation of texture coordinates.Directly interpolating (u,v) using screen-space α,β,γ is wrong! The correct approach is to first interpolate (u/w, v/w, 1/w), then divide by 1/w to recover — this is calledperspective-correct interpolation.

🤔 Why Can't We Interpolate Directly? Perspective transformation isnonlinear. Far-away textures become dense, near textures become sparse. Without w-correction, textures appear "warped" — like the "wavy" effect in old PS1 games. Modern GPUs handle this automatically at the hardware level.

⑬ Extension 2: Lighting — Normals, Dot Products, and Color

A pixel's color doesn't just come from textures — there's also lighting. The core formula:

Brightness = N · L

where N is the surface normal (unit length),L is the light direction (unit length). Their dot product = |N|·|L|·cos(angle).

When N and L point in the same direction (facing the light, angle = 0°) → cos(0) = 1 → brightest. When the angle is 90° → cos(90°) = 0 → completely dark.

Dot product = the mathematical essence of lighting: how "aligned" two directions are

Angle θ	N·L = cos(θ)	Effect
0° (facing the light)	1.0	brightest
45°	≈ 0.71	Moderate
60°	0.5	Dimmer
90°	0	completely dark
> 90°	< 0 → clamp to 0	Light source behind

🤔 Do Normals Need Matrix Transforms Too? Yes — but with a different matrix. Normals use the inverse transpose of the Model matrix (i.e. M^−T) to transform, ensuring they remain unit vectors perpendicular to the surface after transformation. The GPU handles this automatically in the vertex shader.

⑭ Complete Map: The Life Story of a Vertex

Now look back — what does a vertex read from a 3D model file actually go through?

① Local Coordinates (x, y, z, 1)
Original position in the model file

② × Model Matrix → World Coordinates
Scale + Rotate + Translate = all in one go

③ × View Matrix → Camera Coordinates
World re-centered with camera at the origin

④ × Projection Matrix → Clip Space
Depth hidden in w, ready for perspective

⑤ Perspective Division (/w) → NDC [-1,1]
Near-large far-small takes effect here

⑥ Viewport Transform → Screen Coordinates (pixels)
[-1,1] linearly mapped to [0, 1920]×[0, 1080]

⑦ Rasterization → Pixels inside triangle
Edge function determines inside/outside

⑧ Barycentric Interpolation → Fragment Attributes
Color/Texture/Normal = weighted average of three vertices

⑨ Texture Sampling + Lighting → Final Pixel Color RGB
N·L dot product determines brightness, texture lookup determines hue

✅ One Formula to Rule Them All The complete journey from 3D vertex to screen pixel can be compressed into:

Screen Color = Texture(Interpolate(Viewport(/w(P·V·M · v_Local)))) · N·L

Core operations: matrix multiplication, perspective division, edge testing, weight interpolation, dot product — all linear algebra

⑮ Easter Egg: This Math Doesn't Just Belong to GPUs

The GPU pipeline's mathematical toolbox — matrices, homogeneous coordinates, barycentric coordinates, dot products — appears repeatedly in seemingly unrelated fields:

GPU Concept	Cross-Domain Equivalent	Field
4×4 Homogeneous Transform Matrix	DH Matrix in Robot Kinematics	Robotics
MVP = P·V·M Matrix Composition	Chained Coordinate Transforms	Physics/Engineering
Barycentric Interpolation	Shape Functions in FEM	Computational Mechanics
N·L Diffuse Dot Product	Cosine Similarity of Two Vectors	Machine Learning / NLP
Perspective Projection Matrix	Intrinsic Matrix in Camera Calibration	Computer Vision
Homogeneous Coordinates (x,y,z,w)	Homogeneous Representation in Projective Geometry	Pure Mathematics
Rasterization Edge Function	Point-in-Polygon Test in Computational Geometry	GIS/CAD

💡 The Star of the Show: Linear Algebra The math behind GPUs is fundamentallyLinear Algebraapplied to the specific problem of 3D rendering. What you've learned isn't just "how GPUs draw triangles" — it'show to use matrices and vectors to describe, transform, and project objects in space — this is the universal language of modern technology (graphics, robotics, AI, physics simulation).

✦ You Can Now Derive All of This From Scratch

Look back at the question mark from Section ①:3D Triangle Vertex → Screen Pixel.

You now know:

UseHomogeneous coordinates (x, y, z, 1) to represent vertices, and 4×4 matrices for translation, rotation, and scaling

P·V·M Merge three steps into one matrix; each vertex multiplied only once

The projection matrix hides depth in the w component, Divide by w to achieve near-large far-small

Three vertices on screen form a triangle;edge functions determine which pixels are inside

Barycentric coordinates assign weights to the three vertices; attributes smoothly transition inside the triangle

N·L dot product computes lighting brightness; texture sampling determines color

✅ Your Toolbox Matrix multiplication, homogeneous coordinates, perspective division, edge functions, barycentric coordinates, dot products — these six concepts are the entire mathematical foundation for thebillions of triangles that modern GPUs process every second. You don't need to memorize API names (OpenGL? Vulkan? DirectX? — they're just different dialects of the same math) because you already understand the math inside.