00 · Orientation — Math Relearn

Start here · Method

00 · Orientation: relearning math

Mark done

⏱ 25 min📎 No code🎯 Retrieve before you read

This module has two jobs and neither of them is "learn math". First: measure, honestly, what survived from your bachelor degree. Second: fix the way you will study it this time — because the reason it evaporated last time was not your brain, it was the method.

By the end you can

Name the three passes — Remember → Understand → Implement — and say what each one is for.
Show which of the five core mathematical ideas you are weak in, from a cold diagnostic.
Explain why knowledge learned for an exam decays, and what retrieval practice does about it.
Set up the review rhythm (1 / 3 / 7 days) that keeps each module alive.

Why this module exists

Most engineers who "forgot the math" did not forget it — they never encoded it. Studying for a test with three days left is a pattern-matching activity: recognise the problem type, retrieve the recipe, execute, score. The recipe gets stored without its reason, and reasons are what survive. That is why you can still write a socket server from your computer-systems course (you used the idea for years) but cannot say what an eigenvector is for (you used it once, on paper, in June).

So the fix is not "read a better textbook". The fix is to attach every idea to three things it never had before: a picture (what it looks like geometrically), a purpose (what problem humans had before it existed), and a place in ML (the exact line of code or the exact algorithm where it does work). Those three attachments are the entire design of this notebook.

Pass 1

Remember — retrieve before you read

Do not scroll to the answers until you have written yours

The method: three passes, then space it

Every module in this notebook is built the same way, so the shape becomes automatic:

1 · Remembercold retrieval → 2 · Understandpicture + purpose → 3 · Implementrun it, break it → CheckpointMCQ + open questions → Space it1 / 3 / 7 days

Remember (retrieve, don't re-read). Each module opens with questions you try from memory, in writing before seeing any content. Wrong answers here are the most valuable thing in the notebook — they mark exactly where to pay attention.
Understand (attach a picture and a purpose). For each idea: what problem existed before it, what it looks like, what breaks without it. If you cannot state an idea in one plain sentence, you have recognition, not understanding — keep going.
Implement (make it move). Type the code — do not copy it. Then change one number, predict the effect, run it. The moment an abstraction produces an output you can see, it stops being a formula and becomes a tool.
Checkpoint (prove it). Instant-feedback multiple choice plus open questions you answer aloud. Only then tick Mark done — and only if you can explain it without notes.
Space it (make it stay). Re-open the module in 1 day, then 3, then 7. Ten minutes of recall on day three beats an hour of re-reading on day one.

The forgetting curve, and why cold questions hurt

Memory decays fast unless it is retrieved at the moment it is fading. That is why this notebook opens every module with discomfort: the struggle to half-remember is not a sign you are failing, it is the act that strengthens the trace. Studies of students consistently find that testing yourself outperforms re-reading by a wide margin, even when re-reading feels far more productive. Feeling fluent while reading is the illusion; feeling stuck while recalling is the work.

Cold diagnostic: answer all 12 from memory

Ten minutes. Write short answers — a word, a formula, a sketch. Do not look anything up, do not open the modules. Answers are in the recall cards below, but only after you have written yours; grading yourself against an answer you did not attempt is worthless.

In words: what does $f(x) = x^2$ mean? Then compute $f(3)$ and $f(-2)$.
Expand $(a+b)^2$. How about $(a+b)^3$?
What is a vector? Give a real 3-dimensional example from a machine learning dataset.
What does the dot product of two vectors measure?
What is a matrix? Geometrically, what does multiplying a matrix by a vector do?
What is a derivative, in one sentence?
What is the gradient of $g(x,y) = x^2 + y^2$ at the point $(1,2)$? Which direction does it point?
If $P(A) = 0.3$, $P(B) = 0.4$ and $A$, $B$ are independent, what is $P(A \cap B)$?
What does $\mathbb{E}[X]$ mean? What is the expected value of a fair six-sided die?
Correlation vs causation — one sentence each, and one sentence on why ML cares.
Why do we maximise the log-likelihood instead of the likelihood itself?
Name one ML algorithm and list the mathematical ideas you believe hold it together.

What your answers mean

There is no pass mark. Instead, note which group of questions felt hardest — that maps onto the five ideas every classical ML algorithm is assembled from:

Cold questions	Core idea	It becomes, in ML	Go here
1, 2	Functions, parameters, structure	Hypothesis space, the model as $f(x;\theta)$	Module 01
3, 4, 5	Linear algebra	Dataset = matrix, embeddings, PCA, attention	Module 02
6, 7	Calculus	Gradient descent, backpropagation	Module 03
8, 9, 11	Probability	Bayes classifier, MLE, calibrated predictions	Modules 04, 06
10, 12	Statistics & evidence	Evaluation, A/B tests, generalisation claims	Module 05

Machine learning is one equation — minimise a loss over parameters — and every module in this notebook is one of the tools needed to read, differentiate, or defend that equation.

Pass 2

Understand — the purpose of the whole journey

Why this math, and why before deep learning

What machine learning actually needs from mathematics

Strip away the algorithms and every classical ML model is the same statement:

$$\hat{\theta} = \arg\min_{\theta} \; \frac{1}{n}\sum_{i=1}^{n} L\big(f(x_i;\theta),\, y_i\big) + \lambda\,\Omega(\theta)$$ find the parameters θ that make the model's predictions least wrong, plus a penalty that keeps them sane

That single line is why the syllabus looks the way it does. To read it you need the notation of Module 01. To build $f$ and understand why a matrix of weights can represent a dataset you need linear algebra (Module 02). To move $\theta$ downhill you need derivatives, gradients and the chain rule (Module 03). To know that minimising sum-of-squares is secretly assuming Gaussian noise, and that cross-entropy comes from coding theory, you need probability (Module 04) and information theory (Module 06). To decide whether your 0.91 beats someone else's 0.93 you need statistics (Module 05). To actually run the minimisation — and to understand what $\lambda$ is trading off — you need optimization (Module 07). Module 08 then reassembles the classic algorithms from these parts, and Module 09 shows the same parts reappearing, at larger scale, inside deep learning and agents.

Why before deep learning

Deep learning frameworks compute derivatives for you, so it is tempting to skip calculus. That works until something breaks — and it always breaks. When a loss explodes, when a layer saturates, when a transformer attends to nothing, when a fine-tune forgets what it knew: the diagnosis is always the same three suspects — a matrix behaving badly (linear algebra), a gradient vanishing or exploding (calculus + numerics), or a probability assumption that no longer holds. People who own the fundamentals debug in hours what others debug in weeks. This notebook is that ownership.

How to read a formula from now on

From Module 01 onward, treat every formula as a sentence with a grammar. When you meet one, ask the same five questions, in this order, every time:

What are the objects? Scalars, vectors, matrices, functions, sets. Which letters are data, which are parameters, which are outputs?
What is the shape? If $X$ is $n \times d$ and $w$ is $d$, then $Xw$ is length $n$ — one number per sample. Shape errors are 80% of implementation errors.
What is being summed or averaged? Most ML formulas are an average over data points. A $\sum_i$ over samples is almost always "average the error".
What is the direction of "good"? Minimise or maximise? Which term fights which? What does $\lambda$ trade off?
Where does it live in code? Find the equivalent line in NumPy or scikit-learn. If you cannot find it, you have not understood it yet.

Pass 3

Implement — set up the workspace

Ten minutes, once, and never again

The environment check

From Module 02 onwards every module has runnable code. Right now, confirm the three libraries everything depends on are installed and healthy:

import numpy as np
import matplotlib.pyplot as plt
import sklearn

print("numpy     ", np.__version__)
print("sklearn   ", sklearn.__version__)

# A 1-line sanity check you will reuse constantly:
# dot product of a vector with itself = squared length
v = np.array([3.0, 4.0])
print("||v||^2 =", v @ v)          # expect 25.0
print("||v||   =", np.linalg.norm(v))  # expect 5.0

If those three lines run, you are ready. If not, fix it now — a broken environment two modules from now is the single most common reason people stop halfway.

Exercise · 10 minutes

Write your diagnostic answers down (paper or a notes file). Put today's date on them. You will re-answer the same 12 questions after Module 09 — that comparison is your real score, not any quiz here.
Create one notes file per module, and inside it, only two sections: "The one-sentence version" and "Where I still get stuck". No copying of formulas — a formula you transcribe you will not keep; a formula you re-derive you will.
Book the rhythm: 25–40 focused minutes most days beats a 5-hour Sunday. Block it in your calendar now, including the 1 / 3 / 7-day reviews.

Checkpoint

Prove it, then start Module 01

Answer first, then open the cards

Multiple choice — instant feedback

1. Your goal is to still understand gradient descent in six months. What does the learning-science evidence say works best?

Re-read the module carefully once a week
Try to explain it from memory, get stuck, then check
Highlight the formulas and rewrite them neatly
Watch several video explanations back to back

— Retrieval beats re-exposure. Re-reading, highlighting and re-watching create familiarity, which feels like knowledge but is really just recognition of text you have seen. Actively pulling the idea out of your own head — especially when it is effortful and partly wrong — is what strengthens the memory. Use videos and re-reading only after a failed retrieval attempt, as repair.

2. In this notebook, "Understand" a concept means…

Being able to write the formula correctly from memory
Being able to solve a standard exam problem with it in under five minutes
Being able to say why it exists, what it looks like, and where it is used in ML
Having read a formal proof of it once

— Purpose + picture + place. (a) is recall without meaning, (b) is the exam skill that got you here and evaporated, (d) is optional and usually not the bottleneck. Understanding that survives is the kind you can rebuild: why the idea was invented, what it looks like geometrically, and which ML algorithm uses it. If a friend asks "why would anyone do that?" and you have no answer, you have procedure, not understanding.

3. Which mathematical idea is the backbone of principal component analysis (PCA)?

Bayes' rule and conditional probability
The chain rule of differentiation
Entropy and information gain
Eigenvectors and eigenvalues of a covariance matrix

— Linear algebra (Module 02). PCA finds the directions in which the data varies most, and "the directions that a symmetric matrix leaves unchanged, only scaled" is exactly the definition of an eigenvector. Knowing this map in advance is the point: when you meet an algorithm later, you should be able to name the load-bearing math before you read a line of it. (a) powers Naive Bayes and Bayesian optimisation, (b) powers backpropagation, (c) powers decision trees.

4. You just finished a module and everything made sense while reading it. What should you do next for that knowledge to survive?

Close it, and re-answer the module's questions from memory tomorrow
Immediately start the next module while it is fresh
Re-read it once more now to lock it in
Copy the formulas into a neat cheat sheet

— Feelings of fluency are the trap. Sense-making while reading is necessary but not sufficient — that is encoding, and it decays within days. (b) and (c) both pile new material onto a trace that is already fading; (d) is transcription, which is not retrieval. The fix is spaced testing: revisit with no notes, ideally the next day, then at 3 and 7 days. This is also why the notebook stores your quiz answers — so you can see decay before it becomes loss.

5. Which sequence does this notebook ask you to work in — and why that order?

Read everything once, then attempt the questions at the end of the week
Retrieve cold → understand why → implement → checkpoint → space the review
Implement first, read the theory afterwards once you have code to hang it on
Highlight and re-copy the formulas, then move on as soon as they look familiar

— Retrieval first, always. Attempting a question before reading the answer tells your brain what to pay attention to — the "generation effect". Understanding then attaches a picture and a purpose to the trace you just created; implementing makes it a tool instead of a fact; the checkpoint proves it; spacing keeps it. (a) and (d) are passive exposure, which is the habit that produced the forgetting you are here to fix. (c) can work later in your career, but with weak foundations it produces code that runs for the wrong reasons.

6. A dataset lives in a matrix $X$ with shape $(5000, 12)$. What does one row represent?

One feature measured across all 5000 samples
The 12 model parameters, averaged over 5000 training runs
One sample, described by its 12 features
The covariance between pairs of features

— Rows are samples, columns are features — the $(n, d)$ convention. $n = 5000$ rows, $d = 12$ columns; a row is one customer/product/image as a point in $\mathbb{R}^{12}$. (a) describes a column (or the transpose $X^\top$), (d) describes $X^\top X$, which is $12\times12$. This convention is assumed by every formula on this site, and writing shapes next to symbols before computing anything is the cheapest error-prevention habit available to you — Module 01 makes it a ritual.

Recall cards — the diagnostic answers

Only open these once your own answers are written down.

1–2. What is $f(x)=x^2$? And $(a+b)^2$, $(a+b)^3$?

$f$ is a rule that turns an input into an output: square whatever you give it. $f(3)=9$, $f(-2)=4$ — note $f$ cannot tell $3$ from $-2$, a property that later matters a lot (many different inputs, same output ⇒ the inverse is not a function). $(a+b)^2 = a^2 + 2ab + b^2$ (not $a^2+b^2$ — the $2ab$ term is the single most common expansion error in ML derivations, e.g. when expanding squared errors in least squares). $(a+b)^3 = a^3 + 3a^2b + 3ab^2 + b^3$; coefficients come from Pascal's triangle, and the general case is the binomial theorem.

3–4. What is a vector? What does the dot product measure?

A vector is a list of numbers that belong together as one object — in ML, one data point. A customer might be $(\text{age}=34,\ \text{income}=52000,\ \text{clicks}=7)$: three coordinates, one point in 3-D space. The dot product $u \cdot v = \sum_i u_i v_i$ measures alignment: it is large and positive when the two vectors point the same way, zero when they are perpendicular (orthogonal ⇒ "no relationship"), negative when opposed. Geometrically $u\cdot v = \|u\|\|v\|\cos\theta$. This is why embeddings compare items by dot product — and why in Module 02 we will also normalise to remove length from the comparison.

5. What is a matrix? What does matrix × vector do?

A matrix is a grid of numbers with two roles: a system of equations (each row one equation, each column one variable) and, better, a transformation. Multiplying a matrix $A$ by a vector $x$ produces a new vector — the image of $x$ under the transformation defined by $A$. It can rotate, scale, shear, project to fewer dimensions, or flip. The matrix $A$ with shape $d\times d$ maps $\mathbb{R}^d \to \mathbb{R}^d$; a $k\times d$ matrix maps $\mathbb{R}^d \to \mathbb{R}^k$, which is how a neural layer or PCA compresses 784 pixels into 50 features. In ML: your whole dataset is one matrix $X$ of shape $n\times d$.

6–7. What is a derivative? Gradient of $x^2+y^2$ at $(1,2)$?

A derivative is sensitivity: how much the output changes when you nudge the input. $f'(x)$ is the slope of the tangent line at $x$, and the best local linear approximation of $f$ near $x$. Machine learning uses it to answer "in which direction should I change the parameters to make the error smaller?" For $g(x,y)=x^2+y^2$: $\nabla g = (2x,\ 2y)$, so at $(1,2)$ the gradient is $(2,4)$. It points in the direction of steepest increase; gradient descent therefore steps in $-\nabla g = (-2,-4)$, and with learning rate $\eta=0.1$ the next point is $(1-0.2,\ 2-0.4) = (0.8,\ 1.6)$, closer to the minimum at the origin.

8–9. $P(A\cap B)$ for independent events? $\mathbb{E}[X]$ and a fair die?

Independence means one event tells you nothing about the other, so $P(A\cap B) = P(A)\,P(B) = 0.3 \times 0.4 = 0.12$. $\mathbb{E}[X]$ is the long-run average — the probability-weighted sum $\sum_x x\,P(X=x)$ (an integral for continuous variables). A fair die: $\tfrac{1}{6}(1+2+3+4+5+6) = 3.5$, which is not a value the die can take — expectation is a property of the process, not of any single outcome. In ML, $\mathbb{E}[L]$ is the population risk you actually care about, while your training loss is only one sample of it (that gap is Module 05's whole subject).

10. Correlation vs causation?

Correlation: two measured quantities tend to move together — a statement about joint data. Causation: intervening on one changes the other — a statement about what happens when you act. They diverge whenever a third variable drives both (ice-cream sales and drowning both rise in summer), or when the causal arrow is reversed. ML cares because models learn whatever pattern is in the data, including spurious ones. A credit model that "learns" that applicants from a certain postcode default more has found a correlation, and shipping it may be both unfair and fragile — the moment the world shifts, the pattern breaks. Prediction ≠ explanation; that distinction is why Module 05 exists.

11. Why log-likelihood instead of likelihood?

Three reasons, all practical: (1) products become sums — the likelihood of $n$ independent samples is a product $\prod_i p(y_i\mid x_i)$, which is numerically catastrophic (a million tiny numbers underflow to zero); the log turns it into $\sum_i \log p(y_i\mid x_i)$. (2) logs are monotone, so the argmax is unchanged — maximising the log finds the same best parameters. (3) derivatives get simpler — $\log$ kills products and turns powers into multiplications, which is why Gaussian MLE becomes "minimise squared error" and Bernoulli MLE becomes cross-entropy (Modules 04, 06 and 08 derive both properly).

12. One algorithm and its mathematical skeleton

Linear regression, as an example: a function family $f(x;w) = x^\top w$ (Module 01); the data as a matrix $X$ and the normal equations via matrix inversion / projections (Module 02); training by differentiating the squared error and stepping down the gradient (Module 03, 07); the squared-error loss being the maximum-likelihood estimate under Gaussian noise (Module 04); and its evaluation by confidence intervals and residual analysis (Module 05). Five modules, one algorithm — that overlap is not padding, it is the structure of the field. Module 08 does this decomposition for eight algorithms.

Checkpoint — open questions

Answer aloud, without notes. Short and honest is fine.

Explain, to an imaginary colleague, why knowledge learned for an exam decays — and what retrieval practice changes.
What are the three passes, and what is the specific purpose of each?
Write the generic ML objective from memory. Which symbol is data? Which is parameters? Which is the regulariser?
Name the five core ideas and one ML algorithm each one powers.
When you next meet a formula you have never seen, what five questions will you ask it?
What are your 1 / 3 / 7-day review dates for this module?

Continue in your ML notebook

The practical counterpart to this module is ML module 00 · Orientation & diagnostic, which does the same hole-finding for the modelling side. From here, the two notebooks interleave: this one explains why the math works, that one shows what the model does with it.