Mathematical thinking & notation

You have spent years moving symbols around a page without ever reading them. That skill got you through exams and left you with nothing you could reuse: a formula you can manipulate but cannot read aloud as a sentence is a string, not knowledge. This module replaces the manipulation habit with a reading method — five questions, asked in the same order, of every formula you meet from here on.

Cold warm-up: answer these before scrolling

You learned all of this before, and you half-remember some of it. That is the point. Write a short answer to each — a word, a shape, a number. If you blank on one, mark it and move on; the answers are in the recall cards underneath, but only after your own attempt is on paper.

Why notation exists at all

Notation is a technology, and like every technology it exists because something was hard before it. Before symbols, algebra was written out in words — the rhetoric of quantities — and every calculation had to be held in your head while you read the sentence describing it. Writing $\sum_{i=1}^{n}$ instead of "add up the terms from the first to the $n$-th" moves the bookkeeping out of your working memory and onto the page. That is the entire job: notation buys you working memory.

Which gives you a clean test for whether you actually know a symbol. Can you read it aloud as a sentence? $\frac{1}{n}\sum_{i=1}^{n} L(f(x_i;\theta), y_i)$ says "the average, over all $n$ training examples, of how wrong the model's prediction is for each one." If that sentence comes out easily, the symbol is yours. If you can only say "it's the loss function thing", you have the shape but not the meaning — you can manipulate it, and you cannot read it. The first is exam skill; the second is the thing that survives.

The symbols were designed by people who thought about this hard. Leibniz's $\frac{d}{dx}$ is the famous case: it literally spells out the operation (divide the differential of the top by the differential of the bottom), so you can see how the chain rule must behave before you prove it: the chain rule $\frac{dy}{dx} = \frac{dy}{du}\frac{du}{dx}$ cancels like a fraction. Newton's notation for the same derivative could not do that. A century later, matrix notation let someone write an entire regression model on one line. Every symbol you meet was chosen to make one specific thought cheaper.

The cast of characters: what the letters mean

Four kinds of object, and then a set of letters that mean the same thing nearly everywhere in ML. Learn the letters once and an unfamiliar paper becomes mostly familiar: you will not know what the paper invented, but you will know what it did not have to explain.

The single most useful habit in this whole module is shape discipline: never write a symbol without writing where it lives. Not "$X$ times $w$" but "$X \in \mathbb{R}^{n \times d}$ times $w \in \mathbb{R}^{d}$, giving $Xw \in \mathbb{R}^{n}$".

Sets, spaces and quantifiers

A small alphabet of logic that shows up in every paper, and that exists so claims can be short instead of hedged. This is a reading tool, not a set theory course — you need to recognise the symbols, not prove anything with them.

Put together, the standard claim about a data set reads like this — every sample is a $d$-vector, and stacked they give the matrix:

The point worth internalising: $x \in \mathbb{R}^{d}$ is a complete claim about a data point. From it you know how many numbers are in $x$, that they are finite reals, that $x$ is a point in a $d$-dimensional space (so "distance", "direction" and "angle" between data points are even meaningful), and that the only legal partners for it are other $d$-vectors — $x + y$ only makes sense if $y \in \mathbb{R}^{d}$ too. Nothing more is claimed: not that the coordinates are normalised, not that they are independent, not that they are in any particular order. Papers state exactly this much, and reading it as exactly this much is the skill.

Sums, products and the shape of an average

Almost every ML formula is a sum wearing different clothes. Learn to expand one and the rest decode themselves. Take the variance numerator and read it as a sentence first: "add up, for every data point, how far that point sits from the mean, squared."

Three things to notice about that expansion. The index $i$ is bound — it only exists inside the sum and vanishes in the answer, exactly like a loop variable. The upper limit $n$ tells you how many terms, which is a shape statement ($n$ terms from a dataset of $n$ samples). And the $\frac{1}{n}$ that usually sits in front is what turns "total" into "average" — the most common single symbol in ML objectives.

The workhorse pattern, which you will see in every single model: an ML objective is an average over samples. The loss $L$ judges one pair of prediction and truth; the sum collects all the judgements; the $\frac{1}{n}$ makes the result independent of how much data you happened to train on.

$\prod_{i=1}^{n}$ is the same idea with multiplication instead of addition — the probability of all $n$ independent outcomes happening, $\prod_i p(y_i)$. Products are why we take logarithms: a log turns that product into the sum $\sum_i \log p(y_i)$, which is nicer to differentiate and far nicer in floating point (a million small probabilities multiplied together underflow to zero; their logs merely add up).

Double sums $\sum_i \sum_j$ appear whenever two indices are needed — pairwise quantities over samples or features. The covariance between feature $j$ and feature $k$ is the archetype: for each sample $i$, take how far feature $j$ sits from its mean, multiply by how far feature $k$ sits from its mean, then add that up across all samples and average.

Two more symbols you will meet constantly. $\max$ and $\min$ with an index ask which element, not how big — that is $\arg\max$ again, and it is how a classifier turns probabilities into a predicted class. And the indicator $\mathbb{1}[\cdot]$ is an on/off switch that returns 1 when the condition inside is true and 0 when it is false; multiplying by it means "only count the cases where…".

Functions, parameters and families

A function with a declared domain and codomain tells you its whole contract: what it eats, what it returns.

Now the idea that makes machine learning possible at all. If $f$ is fixed, there is nothing to learn — you would just be calling a function. So we do not study one function; we study a parameterised family: $f(x;\theta)$ where the input $x$ is the data and $\theta$ is a separate slot that selects one member of the family. For linear regression the family is "all straight lines", and $\theta = (w, b)$ picks which one. Training is the search over $\theta$; the model you ship is the winning member.

Functions combine. The composition $(f \circ g)(x) = f\big(g(x)\big)$ means "run $g$, then feed its output into $f$". A neural network is nothing but a long composition — one layer's output is the next layer's input — and writing it as $\circ$ is what lets you talk about a 100-layer model without writing 100 formulas.

Linearity: the test, not the appearance

A function is linear exactly when it satisfies the following, for all scalars $\alpha, \beta$ and all inputs $x, z$:

Test both candidates from the warm-up. Take $f(x) = x^2$ with $\alpha = \beta = 1$: the left side is $(x+z)^2 = x^2 + 2xz + z^2$, the right side is $x^2 + z^2$. They differ by $2xz$, so $x^2$ is not linear — the cross term is exactly what breaks it. Now take $f(x) = wx + b$ with $\alpha = \beta = 1$: the left side is $w(x+z) + b$, the right side is $(wx+b) + (zx+b) = wx + zx + 2b$. They differ by $b$, so $wx+b$ is not linear either — unless $b = 0$. In ML it is common to call $wx+b$ "linear" (it is affine: linear in the parameters, straight line on paper) and to reserve the word for the rule above.

That distinction explains the architecture of every model you will meet. Lines and planes are linear and well-behaved but cannot represent a curve, a threshold or an XOR. So models are built as linear parts + non-linear glue: linear layers do the mixing, a non-linearity ($\mathrm{ReLU}$, $\sigma$, $\tanh$) between them provides the bends, and the composition of many linear-plus-non-linear pieces can approximate anything. Drop the glue and a deep network collapses into one flat line — which is a theorem you will meet again in Modules 08 and 10.

Put it together and you get the equation that every model in this notebook is an instance of. Read it aloud before you look at the table: "the best parameters are the ones that make the average training loss smallest, plus a penalty term for complicated parameters."

$\lambda$ is a dial, not a mystery: $\lambda = 0$ means "fit the data, whatever it takes" and large $\lambda$ means "prefer simple parameters even if the fit gets worse". $\Omega$ is the definition of "simple" — squared weights for ridge and most neural nets, tree count for forests, sparsity for lasso. Every regularisation method is choosing an $\Omega$.

Rates of change, previewed

One section, no theory — just enough to show that "how sensitive is the output to a nudge in the input" is a number you can compute today, with no calculus. Shrink the step $h$, divide the change in output by the change in input.

The exact derivative of $x^2$ at $3$ is $6$, and the finite difference gives $6.01$ — the gap is the price of using a step of $0.01$ instead of $0$. Halve $h$ and you will land around $6.005$. That convergence is the derivative, and it is also how you will check your hand-written gradients against automatic differentiation in Module 03: compute the finite difference, compute the analytic gradient, and confirm they agree to a few decimals. Sensitivity is not mysterious — it is one division away from something you already know how to write.

Reading rituals: five questions for any formula

The orientation module introduced these; here they become muscle memory, run twice on real formulas. Same order every time — the order is what makes it fast.

1. What are the objects? Which letters are data, which are parameters, which are outputs, which are indices? A $y$ that appears with a subscript is a sample; a $\theta$ is a knob.
2. What is the shape? Write $n \times d$, $d$, $n$, scalar — next to every symbol. If the shapes on the two sides of an equals sign disagree, you have already found a bug.
3. What is summed or averaged? Over samples, over features, over classes? And is there a $\frac{1}{n}$ turning a total into an average? Nearly always: yes.
4. What is the direction of good? $\min$ or $\max$? Which terms fight each other? What does $\lambda$ trade off? If you cannot say which way is better, you do not know what the formula optimises.
5. Where does it live in code? Point at the NumPy line. If you cannot find one, the understanding is not finished.

Run 1 — mean squared error. First the formula, then the questions.

Objects: $\hat{y}_i$ is the prediction for sample $i$, $y_i$ is its target, both scalars; $n$ counts samples. Shapes: $\hat{y}, y \in \mathbb{R}^{n}$, so the difference is a vector and the sum collapses it to a scalar. Summed over: samples. Direction: smaller is better, zero only when every prediction is exact. In code: np.mean((yhat - y) ** 2). Thirty seconds, all five.

Run 2 — the Gaussian log-likelihood. Same ritual, on a formula that looks far more intimidating.

Objects: $\mathcal{N}$ is the Gaussian (bell curve) density, $\mu$ is where it is centred, $\sigma^2$ is how wide it is, $y_i$ is an observed value. Shapes: everything inside is scalar; the sum produces one scalar. Summed over: samples again. Direction: larger is better — this is a log-probability, and you want the model to find the observed data likely. In code: a sum of -0.5*np.log(2*np.pi*sigma**2) - (y - mu)**2/(2*sigma**2).

Now the surprise. That scary-looking log-likelihood and the friendly MSE are the same objective, once $\sigma$ is fixed. Five steps:

1. Write down the density. $$\mathcal{N}(y_i \mid \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}\,\exp\!\left(-\frac{(y_i-\mu)^2}{2\sigma^2}\right)$$ It says: a value far from $\mu$ (in units of $\sigma$) is exponentially unlikely.
2. Take the log of one term. The log kills the exponential and turns the prefactor into an additive constant: $$\log \mathcal{N}(y_i \mid \mu, \sigma^2) = -\tfrac{1}{2}\log(2\pi\sigma^2) - \frac{(y_i-\mu)^2}{2\sigma^2}$$
3. Sum over all samples. The constant no longer depends on $i$, so it becomes $\frac{n}{2}\log(2\pi\sigma^2)$, and the second term collects all the squared errors: $$\sum_{i=1}^{n} \log \mathcal{N}(y_i \mid \mu, \sigma^2) = -\frac{n}{2}\log(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^{n}(y_i-\mu)^2$$
4. Drop what does not depend on $\mu$. When you optimise over $\mu$, the first term is a constant — adding a constant shifts the answer but never moves where the peak is: $$\arg\max_{\mu} \sum_{i=1}^{n} \log \mathcal{N}(y_i \mid \mu, \sigma^2) = \arg\max_{\mu} \left(-\sum_{i=1}^{n}(y_i-\mu)^2\right)$$
5. Read the sign and the constant. Maximising $-\sum(\cdot)^2$ is minimising $\sum(\cdot)^2$, and $\frac{1}{2\sigma^{2}}$ is a positive constant that cannot change which $\mu$ wins: $$\hat{\mu} = \arg\max_{\mu} \sum_{i=1}^{n} \log \mathcal{N}(y_i \mid \mu, \sigma^2) \;=\; \arg\min_{\mu} \underbrace{\frac{1}{n}\sum_{i=1}^{n} (\mu - y_i)^2}_{\text{MSE}}$$

That is not a trick specific to Gaussians. It is the reason squared error shows up everywhere: it is what you get when you say "the noise around my predictions is Gaussian and I want the most probable parameters". Change the assumed noise distribution and you get a different loss — Laplace noise gives absolute error, a Bernoulli gives cross-entropy. Module 04 derives that pipeline properly; Module 06 does it for log-loss.

Notation traps that waste hours

Every one of these has cost somebody a debugging session. Most are collisions — the same glyph doing double duty — plus a couple of genuine semantic traps.

The one equation every model is

Close the pass by proving the claim from the orientation module: classical ML is one statement, and the algorithms differ only in which three pieces they plug in. Linear regression, logistic regression, a gradient-boosted ensemble of decision stumps and a deep network are all

Read down the columns instead of across the rows. The $f$ column is where the inductive bias lives — what the model is able to express. The $L$ column is where the probabilistic assumption lives (Gaussian $\Rightarrow$ squared error, Bernoulli $\Rightarrow$ cross-entropy). The $\Omega$ column is where the fight against overfitting lives. And $\arg\min$ is shared, which is why a single optimiser — gradient descent — serves all four. Swap a row and you have a different algorithm; swap nothing and you have a different paper.

Three blocks: shapes, sums, and the MSE / likelihood bridge

NumPy only. The point of each block is not the library call — it is that the array on screen has the shape the formula promised.

import numpy as np
rng = np.random.default_rng(0)

# 5 customers, 3 features: the whole dataset is ONE matrix, rows = samples.
n, d = 5, 3
X = rng.normal(size=(n, d))
w = rng.normal(size=d)          # parameters: one number PER FEATURE, not per sample

print("X    :", X.shape)        # (5, 3)   n x d
print("w    :", w.shape)        # (3,)     d
print("X @ w:", (X @ w).shape)  # (5,)     n  -- one score per sample, exactly n numbers

# Two rows are two customers; their dot product is one number measuring how
# aligned their feature vectors are (positive = similar, near 0 = unrelated).
print("row0 . row1 =", round(float(X[0] @ X[1]), 4))   # ~0.32 with this seed

# Row i of X dotted with w is EXACTLY entry i of X @ w -- the score for sample i.
print("X[2] @ w == (X @ w)[2]:", np.isclose(X[2] @ w, (X @ w)[2]))   # True

The last line is the whole point of shape discipline: $(n \times d)$ times $(d)$ gives $(n)$, and matrix–vector multiplication is just $n$ dot products, one per row, stacked. A model's "score every sample at once" is not a special operation — it is a loop you never had to write.

import numpy as np
rng = np.random.default_rng(1)

n    = 8
y    = rng.normal(loc=3.0, scale=1.0, size=n)   # targets
yhat = y + rng.normal(scale=0.5, size=n)        # predictions: close, not perfect

# NumPy way: the whole average in one expression (this is what you ship)
mse_vec = np.mean((yhat - y) ** 2)

# Formula way: the SAME object, written as (1/n) * sum_i (...)^2
mse_sum = 0.0
for i in range(n):          # this loop IS the sum over i = 1..n
    mse_sum += (yhat[i] - y[i]) ** 2
mse_sum /= n                # the 1/n in front

print(f"{mse_vec:.10f}")    # 0.0524715613 with this seed
print(f"{mse_sum:.10f}")    # 0.0524715613 -- identical
print("identical:", np.isclose(mse_vec, mse_sum))   # True

That is the translation you should be able to do in both directions on sight: symbol → loop, loop → vectorised call. When a paper formula will not run, writing the loop version is usually how you find which index you had wrong.

import numpy as np
rng = np.random.default_rng(2)

n, sigma = 200, 1.0
y = rng.normal(0.0, 1.0, n)

# Three candidate models for the SAME data; only the residual scale differs.
models = {
    "A (good)": y + rng.normal(0, 0.4, n),
    "B (meh)" : y + rng.normal(0, 1.0, n),
    "C (bad)" : y + rng.normal(0, 2.0, n),
}

rows = []
for name, yhat in models.items():
    sse = np.sum((yhat - y) ** 2)
    mse = sse / n
    # log N(y | yhat, sigma^2) summed over i, with sigma FIXED and shared
    ll  = np.sum(-0.5*np.log(2*np.pi*sigma**2) - (yhat - y)**2 / (2*sigma**2))
    # the same number rewritten as a straight line in MSE -- they must agree
    ll2 = -n/2*np.log(2*np.pi*sigma**2) - n*mse/(2*sigma**2)
    assert np.isclose(ll, ll2)
    rows.append((name, mse, ll))

for name, mse, ll in sorted(rows, key=lambda r:  r[1]):
    print(f"{name:9s} MSE    = {mse:8.4f}")      # A 0.1771, B 0.9570, C 4.2481
print()
for name, mse, ll in sorted(rows, key=lambda r: -r[2]):
    print(f"{name:9s} loglik = {ll:10.3f}")      # A -201.502, B -279.486, C -608.597
# Same order both times: with sigma shared, loglik = const - (n / (2 sigma^2)) * MSE,
# a strictly decreasing line in MSE, so the ranking CANNOT differ.

Multiple choice — instant feedback

Open questions — answer aloud, without notes

Short and honest. If an answer will not come, mark the module for a re-read rather than guessing.

Re-answer these in 1, 3 and 7 days. Open the recall cards only after your own answer is written down.