Reference · Notation & formula cheat sheet

Section 1

Notation dictionary

📖 Module 01 · Thinking & notation📖 Module 02 · Linear algebra

Symbol	Read as	What it means
$x,\ y$	ex, why	One data point and its target. Scalars lower-case: $x=3.4$.
$n,\ d$	en, dee	$n$ = number of samples (rows), $d$ = number of features (columns).
$i,\ j$	eye, jay	$i$ counts samples, $j$ counts features. $X_{ij}$ = feature $j$ of sample $i$.
$w,\ \theta$	double-you, theta	The learned parameters. $w$ is a weight vector $\in\mathbb{R}^d$; $\theta$ means "all parameters of any model".
$b$	bee	Bias / intercept: the constant added so the model need not pass through the origin.
$f(\cdot)$	eff of dot	A function. The $\cdot$ marks where the argument sits: $f(x)=x^2$.
$L(\cdot)$	loss	Penalty for one prediction: $L(\hat y, y)$. Larger = worse.
$\mathcal{L}$	script L	The whole objective (average loss + regulariser), or a likelihood in probabilistic texts.
$\eta$	eta	Learning rate — how far each gradient step moves the parameters.
$\lambda$	lambda	Regularisation strength (or a Lagrange multiplier). Careful: in Poisson / Exponential it is the rate parameter instead.
$\alpha$	alpha	Significance level in tests; also a Beta-distribution parameter and the first-moment rate in Adam-style optimisers.
$\sigma$	sigma	Standard deviation (lower-case); also the sigmoid function $\sigma(x)=\frac{1}{1+e^{-x}}$.
$\mu$	mu	A mean — of a distribution (population) or of data ($\bar x$).
$\Sigma$	capital sigma	Covariance matrix (also: sum sign $\sum_i$). $\Sigma_{jk}$ = covariance of features $j$ and $k$.
$\nabla$	nabla / del	Gradient operator. $\nabla_w L$ = vector of all $\partial L/\partial w_j$, pointing uphill.
$\partial$	partial	Partial derivative: $\frac{\partial L}{\partial w_j}$ = sensitivity to one parameter, holding the rest fixed.
$\mathbb{E}$	expected value	Probability-weighted average: $\mathbb{E}[X]=\sum_x x\,p(x)$ (integral if continuous).
$\mathbb{V}$	variance	Spread around the mean: $\mathbb{V}[X]=\mathbb{E}[(X-\mu)^2]=\mathbb{E}[X^2]-\mu^2$.
$\arg\min$	argmin — the argument that minimises	The input that makes $f$ smallest, not the smallest value: $\arg\min_x (x-3)^2 = 3$.
$\arg\max$	argmax	Same idea, other direction. $\hat w_{\mathrm{MLE}}=\arg\max_w p(D\mid w)$.
$X\sim p$	X distributed as p	$X\sim\mathcal{N}(0,1)$: $X$ is drawn from that distribution.
$A\perp B$	independent of	Knowing $B$ tells you nothing about $A$: $P(A\cap B)=P(A)P(B)$.
$x\in\mathbb{R}^d$	in / element of	$x$ is a vector of $d$ real numbers.
$p\propto q$	proportional to	Equal up to a (usually dropped) constant — the standard way to write unnormalised posteriors.
$\hat{\cdot}$	hat — an estimate	$\hat w$ your fitted weights vs. the unknowable true $w$; $\hat y$ the model's prediction; $\hat\mu,\hat\sigma^2$ sample estimates.
$\bar{\cdot}$	bar — sample mean	$\bar x=\frac1n\sum_i x_i$, the plain average of the data you have.
$\\|x\\|_p$	pee-norm	Size of a vector: $\\|x\\|_1=\sum_i\|x_i\|$, $\\|x\\|_2=\sqrt{\sum_i x_i^2}$, $\\|x\\|_\infty=\max_i\|x_i\|$.
$O(\cdot)$	big-O	Growth rate with constants ignored: $O(n^2)$ scales quadratically. Used for runtime and convergence claims.
$\mathbb{1}[\cdot]$	indicator	1 if the condition holds, else 0: $\mathbb{1}[x\gt 0]$.
$A^\top$	A transpose	Rows and columns swapped. $x^\top y$ is the dot product of two column vectors.

Shape conventions

Most implementation bugs are shape bugs. These conventions hold everywhere in this notebook.

Object	Shape	In NumPy
$X\in\mathbb{R}^{n\times d}$	design matrix — $n$ samples × $d$ features	`X.shape == (n, d)`
$y\in\mathbb{R}^{n}$	one target per sample (regression)	`y.shape == (n,)`
$y\in\{1,\dots,K\}^{n}$	class ids; one-hot form is $n\times K$	`y.shape == (n,)` → `np.eye(K)[y]`
$w\in\mathbb{R}^{d},\ b\in\mathbb{R}$	parameters — one weight per feature, plus a scalar	`w.shape == (d,)`
$Xw\in\mathbb{R}^{n}$	one prediction per sample: $(Xw)_i = x_i^\top w$	`X @ w` → shape `(n,)`
$X^\top X\in\mathbb{R}^{d\times d}$	feature × feature Gram matrix — where the normal equations live	`X.T @ X`
$X^\top y\in\mathbb{R}^{d}$	right-hand side of the normal equations	`X.T @ y`
row vs column vector	maths writes vectors as columns; a row is written $x^\top$	NumPy stores both as 1-D `(d,)` — the ambiguity starts here

Reading the generic objective

Every classical model in this notebook is this equation with a different $f$, $L$ and $\Omega$:

$$\hat\theta=\arg\min_\theta\ \mathcal{J}(\theta)=\frac{1}{n}\sum_{i=1}^{n}L\big(f(x_i;\theta),\,y_i\big)+\lambda\,\Omega(\theta)$$ find the parameters that make the average prediction error smallest, plus a penalty that keeps them from over-fitting

Piece	Role	Choose it by changing…
$f(x_i;\theta)$	the model's prediction for sample $i$	the algorithm: $x^\top w$ (linear), $\sigma(x^\top w)$ (logistic), a tree, a network
$L(\hat y,y)$	how wrong one prediction is	the noise / error assumption: squared error ⇔ Gaussian, cross-entropy ⇔ Bernoulli
$\frac1n\sum_i$	average over the data, so $\lambda$ means the same thing at any $n$	rarely — but do not forget it, it is the factor of $n$ that breaks ridge closed forms
$\lambda\,\Omega(\theta)$	penalty on the size of the parameters	$\Omega=\\|w\\|_2^2$ → ridge; $\Omega=\\|w\\|_1$ → lasso; $\lambda=0$ → plain fit
$\arg\min$	the output of training is the parameter values, not a number	the optimiser: closed form, GD, SGD, Adam

Section 2

Linear algebra quick card

📖 Module 02 · derivations & geometry🎯 shapes first, then the identity you need

Identity	Read as / caveat
$u^\top v=\sum_i u_iv_i$	dot product: one number measuring alignment. $=0$ means orthogonal.
$(Ab)_i=\sum_j A_{ij}b_j$	matrix × vector: element $i$ is the dot product of row $i$ with $b$. Needs $A\in\mathbb{R}^{m\times k}$, $b\in\mathbb{R}^{k}$ → result $\in\mathbb{R}^{m}$.
$(AB)^\top=B^\top A^\top$	transpose reverses order. Never $A^\top B^\top$.
$(AB)^{-1}=B^{-1}A^{-1}$	inverse reverses order too — and only exists if both are square and invertible. Otherwise use the pseudo-inverse.
$\det\begin{pmatrix}a&b\\c&d\end{pmatrix}=ad-bc$	signed area scale factor. $0$ ⇒ the matrix squashes space ⇒ not invertible.
$\begin{pmatrix}a&b\\c&d\end{pmatrix}^{-1}=\frac{1}{ad-bc}\begin{pmatrix}d&-b\\-c&a\end{pmatrix}$	the $2\times2$ inverse — swap the diagonal, negate the off-diagonal, divide by the determinant. Needs $ad-bc\neq 0$.
$\mathrm{rank}(A)$	number of linearly independent rows = of columns. $\leq\min(m,n)$; full rank ⇒ invertible (square case).
$Av=\lambda v$	eigenvalue equation: $v$ is a direction the matrix only scales, by $\lambda$. Needs $A$ square; complex $\lambda$ possible for non-symmetric $A$.
$\Sigma=PDP^{-1}$	eigendecomposition of a symmetric $\Sigma$: columns of $P$ are orthonormal eigenvectors, $D$ the eigenvalues. Then $P^{-1}=P^\top$.
$A=U\Sigma V^\top$	SVD, works for any $A\in\mathbb{R}^{m\times n}$. $U\in\mathbb{R}^{m\times m}$, $V\in\mathbb{R}^{n\times n}$ orthogonal, $\Sigma$ diagonal with singular values $\geq 0$. Low rank: $A\approx U_k\Sigma_kV_k^\top$.
$\\|x\\|_1=\sum_i\|x_i\|$, $\ \\|x\\|_2=\sqrt{x^\top x}$, $\ \\|x\\|_\infty=\max_i\|x_i\|$	L1 is sparse-friendly, L2 is the smooth default, L∞ is the worst single coordinate. $\\|x\\|_2^2$'s gradient is $2x$ — that is why L2 is easy to optimise.
$\cos\theta=\frac{u^\top v}{\\|u\\|\,\\|v\\|}$	cosine similarity: dot product with length removed, $[-1,1]$. The default way to compare embeddings.
$A\succ 0 \iff x^\top Ax\gt 0\ \ \forall x\neq 0$	positive definite ⇔ all eigenvalues $\gt 0$ ⇔ unique minimum of $x^\top Ax$. Covariance matrices are positive semidefinite ($\geq 0$ allowed).
$\Sigma=\frac{1}{n-1}X_c^\top X_c$	sample covariance from column-centred $X_c=X-\bar x$. $d\times d$, symmetric, PSD. (Use $1/n$ for the biased MLE version.)
$Z=X_cW_k$	PCA projection: $W_k\in\mathbb{R}^{d\times k}$ = top-$k$ eigenvectors of $\Sigma$ = right singular vectors of $X_c$. $Z\in\mathbb{R}^{n\times k}$.

The two inverses you should actually use

Never code np.linalg.inv(A) @ b. Use np.linalg.solve(A, b) — faster and numerically safer. When $A$ is singular or fat (more features than samples), use np.linalg.lstsq(X, y) or the SVD, which all fall back to the pseudo-inverse. For symmetric matrices use eigh, not eig.

Section 3

Calculus quick card

📖 Module 03 · derivatives, gradients, chain rule🎯 everything gradient descent touches

Rule	Formula	Where it bites in ML
Power	$\frac{d}{dx}x^n=nx^{n-1}$	squared-error loss, L2 penalty
Constant multiple	$(cf)'=cf'$	pulling $\frac1n$ and $\lambda$ out of a sum
Sum	$(u+v)'=u'+v'$	why $\nabla$ of a sum of losses is the sum of gradients — minibatches
Product	$(uv)'=u'v+uv'$	product-of-experts, log-likelihoods of independent factors
Quotient	$\left(\frac{u}{v}\right)'=\frac{u'v-uv'}{v^2}$	normalising constants, softmax's denominator
Chain rule	$\frac{d}{dx}g(u(x))=g'(u)\,u'(x)$	backpropagation, nothing less
Log	$\frac{d}{dx}\ln x=\frac1x$, so $\frac{d}{dx}\ln u=\frac{u'}{u}$	turns products into sums, then differentiates: every MLE derivation
Exponential	$\frac{d}{dx}e^x=e^x$, so $\frac{d}{dx}e^{u}=u'e^{u}$	Gaussian, softmax, exponential decay
Sigmoid	$\sigma(x)=\frac{1}{1+e^{-x}}$, $\quad\sigma'(x)=\sigma(x)\,(1-\sigma(x))$	the derivative is the output times its complement — cheap, and it vanishes at the edges
Softplus	$\frac{d}{dx}\log(1+e^x)=\sigma(x)$	smooth ReLU; its derivative is the sigmoid

Object	Definition	Shape
Derivative $f'(x)$	$\lim_{h\to 0}\frac{f(x+h)-f(x)}{h}$ — sensitivity, the slope of the tangent	scalar → scalar
Gradient $\nabla_w L$	column of all partials $\left[\frac{\partial L}{\partial w_1},\dots,\frac{\partial L}{\partial w_d}\right]^\top$; points uphill	$d$-vector for scalar $L$
Jacobian $J$	$J_{ij}=\frac{\partial f_i}{\partial x_j}$ for a vector-valued $f$	out-dims × in-dims
Hessian $H$	$H_{ij}=\frac{\partial^2 L}{\partial w_i\partial w_j}$ — curvature; equals its transpose if smooth	$d\times d$
Multivariable chain rule	$\frac{\partial L}{\partial x_j}=\sum_i \frac{\partial L}{\partial f_i}\frac{\partial f_i}{\partial x_j}$ — backprop is this sum, run backwards	matrix form: $J_x L = J_f x^\top\, \nabla_f L$

The three gradients you will re-derive most often (shapes: $X$ is $n\times d$, $w,y$ are $d$ and $n$):

$$\nabla_w\|Xw-y\|^2=2X^\top(Xw-y)$$ least squares: the residual vector, pushed back through $X^\top$, gives a $d$-vector

$$\nabla_w\,\mathrm{BCE}=\frac{1}{n}X^\top\big(\sigma(Xw)-y\big)$$ logistic regression: prediction minus target, pushed back through $X^\top$ — same shape as the MSE one

$$\nabla_w\,\lambda\|w\|^2=2\lambda w$$ L2 penalty: a gradient that simply drags every weight a little towards zero

Taylor, integrals and the rest

Item	Formula / statement	Why you need it
Taylor, 1st order	$f(x_0+\Delta)\approx f(x_0)+f'(x_0)\,\Delta$	"the function near here looks linear" — the justification for gradient steps
Taylor, 2nd order	$f(x_0+\Delta)\approx f(x_0)+f'(x_0)\Delta+\tfrac12 f''(x_0)\Delta^2$	adds curvature; Newton's method uses it
Multivariate 2nd order	$f(\theta+\Delta)\approx f(\theta)+\nabla f^\top\Delta+\tfrac12\Delta^\top H\Delta$	the $\Delta^\top H\Delta$ term is where conditioning and second-order optimisers live
Integral	$\int_a^b f(x)\,dx$ = signed area under the curve; $\frac{d}{dx}\int_a^x f = f(x)$	probability densities integrate to 1; likelihoods are products that become sums of log-integrals in the log domain
Gaussian integral	$\int_{-\infty}^{\infty} e^{-x^2}\,dx=\sqrt{\pi}$, hence $\int e^{-x^2/2}dx=\sqrt{2\pi}$	the normalising constant of $\mathcal{N}(\mu,\sigma^2)$ comes from here
Integration by parts	$\int u\,dv=uv-\int v\,du$	entropy and KL derivations, exponential families

Section 4

Probability quick card

📖 Module 04 · Bayes, distributions, MLE🎯 model uncertainty instead of pretending

Rule	Formula	Caveat
Axioms	$0\le P(A)\le 1$, $\quad P(\Omega)=1$, $\quad P(A\cup B)=P(A)+P(B)$ if $A\cap B=\emptyset$	probability is a measure whose total is 1
Conditional	$P(A\mid B)=\frac{P(A\cap B)}{P(B)}$, needs $P(B)\gt 0$	read "given that $B$ happened"
Bayes	$P(A\mid B)=\frac{P(B\mid A)\,P(A)}{P(B)}$	$P(B)=\sum_i P(B\mid A_i)P(A_i)$ — the usually-ignored normaliser
Total probability	$P(B)=\sum_i P(B\mid A_i)P(A_i)$	sum over any partition $\{A_i\}$ of the space
Independence	$A\perp B\iff P(A\cap B)=P(A)P(B)\iff P(A\mid B)=P(A)$	distinct from uncorrelated — uncorrelated $\not\Rightarrow$ independent
Expectation	$\mathbb{E}[X]=\sum_x x\,p(x)$ (or $\int x\,p(x)dx$)	a property of the process, not of any outcome
Linearity	$\mathbb{E}[X+Y]=\mathbb{E}[X]+\mathbb{E}[Y]$ — always, independence not needed	the workhorse of almost every ML derivation
Variance	$\mathrm{Var}(X)=\mathbb{E}[X^2]-\mu^2$, $\ \mathrm{Var}(aX+b)=a^2\mathrm{Var}(X)$	variances add only for independent (or uncorrelated) variables
LLN	$\bar x_n\xrightarrow{\ n\to\infty\ }\mu$ (almost surely)	the average converges — one run of a model
CLT	$\frac{\bar x_n-\mu}{\sigma/\sqrt n}\xrightarrow{d}\mathcal{N}(0,1)$	the sampling distribution of the mean becomes Gaussian — the basis of every test and CI
Standard error	$\mathrm{SE}(\bar x)=\frac{\sigma}{\sqrt n}$	spread of the estimator, not of the data; $\sqrt{n}$, not $n$
Gaussian MLE	$\hat\mu=\bar x$, $\quad\hat\sigma^2_{\mathrm{MLE}}=\frac1n\sum_{i=1}^n(x_i-\bar x)^2$	$1/n$ is the biased MLE; divide by $n-1$ for an unbiased estimate
MAP	$\hat\theta_{\mathrm{MAP}}=\arg\max_\theta\ \log p(D\mid\theta)+\log p(\theta)$	MLE + a prior; uniform prior ⇒ the two coincide
MLE vs log	$\hat\theta_{\mathrm{MLE}}=\arg\max_\theta\ \sum_i \log p(x_i\mid\theta)$	log is monotone, so the argmax is unchanged and products became sums

The nine distributions to know by name

Distribution	PMF / PDF	Mean	Variance	ML use
Bernoulli($p$)	$p^k(1-p)^{1-k}$, $k\in\{0,1\}$	$p$	$p(1-p)$	one yes/no outcome; logistic regression's likelihood
Categorical($p_1..p_K$)	$P(X=k)=p_k$	— (one-hot)	—	$K$-way outcome; softmax classifier + cross-entropy
Binomial($n,p$)	$\binom{n}{k}p^k(1-p)^{n-k}$	$np$	$np(1-p)$	successes out of $n$ trials: clicks, conversions
Poisson($\lambda$)	$\frac{e^{-\lambda}\lambda^k}{k!}$	$\lambda$	$\lambda$	counts of rare events per interval; count regression
Uniform($a,b$)	$\frac{1}{b-a}$ on $[a,b]$	$\frac{a+b}{2}$	$\frac{(b-a)^2}{12}$	maximum-ignorance prior; random initialisation
Gaussian($\mu,\sigma^2$)	$\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}$	$\mu$	$\sigma^2$	the default noise model; why MSE is the default loss
Exponential($\lambda$)	$\lambda e^{-\lambda x}$, $x\ge 0$	$\frac1\lambda$	$\frac{1}{\lambda^2}$	time until an event; survival / time-to-event models
Beta($\alpha,\beta$)	$\frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)}$ on $[0,1]$	$\frac{\alpha}{\alpha+\beta}$	$\frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)}$	distribution over probabilities; conjugate prior for Bernoulli, A/B testing
Log-normal	$\frac{1}{x\sigma\sqrt{2\pi}}e^{-\frac{(\ln x-\mu)^2}{2\sigma^2}}$	$e^{\mu+\sigma^2/2}$	$(e^{\sigma^2}-1)e^{2\mu+\sigma^2}$	heavy right tail: prices, file sizes, durations

One-line link to losses

Pick the distribution that matches your error assumption, take its log-likelihood, negate it: that is your loss. Gaussian noise + neg-log-likelihood = MSE. Bernoulli noise = binary cross-entropy. Categorical = softmax cross-entropy. Derivation: Module 04.

Section 5

Statistics quick card

📖 Module 05 · estimators, tests, evaluation🎯 defend a number with evidence

Item	Formula / definition	Read it as
Unbiased	$\mathbb{E}[\hat\theta]=\theta$	right on average over repeated samples — no guarantee about any single one
Consistent	$\hat\theta\xrightarrow{\ n\to\infty\ }\theta$	gets right with more data; MLE is (under mild conditions) both
Efficient	smallest $\mathbb{V}[\hat\theta]$ among unbiased estimators	the Cramér–Rao bound says you cannot beat it with less data
Bias–variance	$\mathbb{E}\big[(y-\hat f(x))^2\big]=\mathrm{bias}^2(\hat f)+\mathrm{var}(\hat f)+\sigma^2$	error = systematic wrongness + instability + irreducible noise
Confidence interval	$\bar x\pm z_{\alpha/2}\frac{\sigma}{\sqrt n}$ (known $\sigma$; use $t_{n-1}$ and $s$ otherwise)	"95% of intervals built this way contain $\mu$" — not "95% chance $\mu$ is in this one"
z-test statistic	$z=\frac{\bar x-\mu_0}{\sigma/\sqrt n}\ \sim\ \mathcal{N}(0,1)$ under $H_0$	how many standard errors the data sits from the null value
p-value	$P(\text{data at least as extreme as observed}\mid H_0\ \text{true})$	how surprising the data is if $H_0$ were true. Never $P(H_0\mid\text{data})$
Power	$1-\beta=P(\text{reject } H_0\mid H_0\ \text{false})$	chance of detecting a real effect; grows with $n$ and with effect size
Why $n-1$	$s^2=\frac{1}{n-1}\sum_i(x_i-\bar x)^2$	the residuals sum to zero, so one degree of freedom is used up estimating $\bar x$ (Bessel's correction)
k-fold CV	split into $k$ folds; train on $k-1$, score on 1; average the $k$ scores	uses all data for both train and validation; $k=5$ or $10$ is the usual choice
Repeated holdout / test set	one final untouched split, or repeated $k$-fold	the test set is touched once, at the very end

Leakage checklist

Fit scalers, imputers, encoders, PCA and feature selection on the training fold only — inside the CV loop, not before it.
Split before any step that saw the target: selection, hyperparameter tuning, early stopping.
Respect time order: train on the past, validate on the future. Random shuffling leaks the future into the past.
Group by user / session / hospital: all rows from one entity go in the same fold.
Keep the final test set sealed — no threshold-picking, no "just one more feature" checks against it.

Section 6

Information theory quick card

📖 Module 06 · entropy, cross-entropy, KL🎯 the real reason the loss takes a log

Concept	Formula	Read as
Self-information	$I(x)=-\log p(x)$	surprise of one outcome; rare events carry more bits
Entropy	$H(p)=-\sum_x p(x)\log p(x)=\mathbb{E}_p[I(x)]$	average surprise; also the optimal code length in bits (log base 2)
Cross-entropy	$H(p,q)=-\sum_x p(x)\log q(x)$	average surprise of using code $q$ for truth $p$; $=H(p)$ only if $q=p$
KL divergence	$\mathrm{KL}(p\\|q)=\sum_x p(x)\log\frac{p(x)}{q(x)}=H(p,q)-H(p)$	how many extra bits you waste coding $p$ with $q$. $\ge 0$, asymmetric: $\mathrm{KL}(p\\|q)\ne\mathrm{KL}(q\\|p)$
Decomposition	$H(p,q)=H(p)+\mathrm{KL}(p\\|q)$	cross-entropy = irreducible noise + your model's error — the term that gradient descent can shrink
Jensen's inequality	$\phi\big(\mathbb{E}[X]\big)\le\mathbb{E}[\phi(X)]$ for convex $\phi$ ($\ge$ for concave, e.g. $\log$)	used to prove $\mathrm{KL}\ge 0$ and to build the ELBO in VAEs
Mutual information	$I(X;Y)=\sum_{x,y}p(x,y)\log\frac{p(x,y)}{p(x)p(y)}=H(X)-H(X\mid Y)$	how much knowing $Y$ tells you about $X$; $=0$ iff independent
Information gain	$\Delta H=H(\text{parent})-\sum_c\frac{\|S_c\|}{\|S\|}H(\text{child}_c)$	entropy reduction from a split — the criterion behind ID3/C4.5 trees
Gini impurity	$G=1-\sum_{k=1}^K p_k^2$	CART's default split criterion; same spirit as entropy, cheaper
Perplexity	$\mathrm{PP}=2^{H(p)}$ (or $e^{H(p)}$ with natural logs)	effective number of equally likely outcomes; language models report $\exp(\text{mean NLL})$
Temperature	$p_T(x)=\frac{p(x)^{1/T}}{\sum_{x'}p(x')^{1/T}}$	$T\to 0$ → argmax, $T=1$ → the model, $T\to\infty$ → uniform. Softens logits without retraining

Section 7

Optimization quick card

📖 Module 07 · GD, momentum, Adam, regularisation🎯 choose and tune an optimizer for a reason

Method	Update (one step)	Read as
Batch GD	$w\leftarrow w-\eta\,\nabla_w L(w)$	the whole dataset's gradient; deterministic, expensive per step
SGD (minibatch)	$w\leftarrow w-\eta\,\nabla_w L_B(w)$ over a batch $B$	noisy steps that escape shallow traps and amortise the cost; shuffle every epoch
Momentum	$v_t=\gamma v_{t-1}+\eta g_t$, $\quad w\leftarrow w-v_t$	an exponentially weighted average of past gradients ($\gamma\approx 0.9$): dampens oscillation, accelerates along the valley
Adam	$m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t$, $\ v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2$	momentum on the gradient (mean) and on its square (scale)
Adam, bias-corrected step	$\hat m_t=\frac{m_t}{1-\beta_1^t}$, $\ \hat v_t=\frac{v_t}{1-\beta_2^t}$, $\ w\leftarrow w-\eta\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}$	the actual update — divide by $\sqrt{\hat v_t}$ so each coordinate is normalised by its own typical gradient size. Defaults $\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=10^{-8}$

Convexity, KKT and Lagrange

Item	Formula / statement	Why it matters
Convexity	$f\big(\theta x+(1-\theta)y\big)\le\theta f(x)+(1-\theta)f(y)$ for all $\theta\in[0,1]$	any local minimum is global — the guarantee deep learning gives up
Lagrangian	$\mathcal{L}(x,\lambda)=f(x)+\sum_i\lambda_i g_i(x)+\sum_j \nu_j h_j(x)$	turns constrained problems into unconstrained ones: minimise over $x$, maximise over the multipliers
KKT — stationarity	$\nabla f(x^)+\sum_i\lambda_i\nabla g_i(x^)=0$	first-order condition at a constrained optimum
KKT — primal feasibility	$g_i(x^)\le 0$, $\ h_j(x^)=0$	the point must actually be feasible
KKT — complementary slackness	$\lambda_i g_i(x^*)=0$	a constraint either binds ($g_i=0$) or is ignored ($\lambda_i=0$) — why only some weights end up at the L1 corner
KKT — dual feasibility	$\lambda_i\ge 0$	multipliers are non-negative for inequality constraints
Conditioning number	$\kappa(A)=\left\|\frac{\lambda_{\max}}{\lambda_{\min}}\right\|$ (equivalently $\frac{\sigma_{\max}}{\sigma_{\min}}$)	$\kappa\approx 1$: round bowl, big safe steps. $\kappa$ large: narrow valley, gradient descent zig-zags — hence feature scaling
Gradient clipping	$g\leftarrow\frac{g}{\max(1,\ \\|g\\|/c)}$	global-norm clipping caps exploding gradients (RNNs, transformers) at threshold $c$
Warmup	$\eta_t=\eta\cdot\frac{t}{T_{\mathrm{warm}}}$ for $t\lt T_{\mathrm{warm}}$, then constant or decay	start tiny: early gradients are computed at bad parameters, and Adam's $\sqrt{\hat v_t}$ estimate is unreliable for the first few steps

	L1 (lasso): $\lambda\\|w\\|_1$	L2 (ridge): $\lambda\\|w\\|_2^2$
Effect on weights	shrinks some weights exactly to zero	shrinks all weights smoothly towards zero, never exactly zero
Geometry	diamond constraint — corners on the axes, minima land there	ball constraint — smooth, minimum rarely on an axis
Gradient	$\lambda\,\mathrm{sign}(w)$ — undefined at 0, use a subgradient or proximal step	$2\lambda w$ — smooth everywhere, trivial to optimise
Closed form	no clean global one (orthogonal case ⇒ soft-threshold $S_\lambda(x)$)	yes: $\hat w=(X^\top X+n\lambda I)^{-1}X^\top y$ for the $\frac1n\\|Xw-y\\|^2+\lambda\\|w\\|^2$ convention
Use when	you believe many features are irrelevant — implicit feature selection, sparsity	many correlated useful features; the default stabiliser (ridge, weight decay)
Failure mode	with $d\gt n$ picks at most $n$ features, arbitrarily among correlated ones	never removes features, so keep them all in the model

Learning-rate rules of thumb

Adam: start at $10^{-3}$. SGD + momentum on well-scaled data: $10^{-2}$ to $10^{-1}$. Loss explodes or turns NaN → divide by 10. Loss crawls → multiply by ~3. Loss plateaus → decay by $\times 0.1$ or use cosine decay to zero. Same curve shape, wildly different outcomes: the learning rate is the single most consequential hyperparameter you own.

Section 8

Loss functions & objectives

📖 Module 08 · algorithms as math assemblies🎯 gradients are written w.r.t. the prediction $\hat y$ or the logit $z$, not the weights

$$\mathcal{J}(\theta)=\frac{1}{n}\sum_{i=1}^{n}L\big(f(x_i;\theta),\,y_i\big)+\lambda\,\Omega(\theta)$$ the generic regularised objective — every row below is one choice of $L$

Loss	Formula	Used by	Gradient (w.r.t. $\hat y$, or $z$ where noted)	Notes
MSE (squared error)	$(\hat y-y)^2$	linear / ridge regression, regression heads	$2(\hat y-y)$	neg-log-likelihood under Gaussian noise; punishes outliers quadratically
MAE (absolute error)	$\|\hat y-y\|$	robust regression	$\mathrm{sign}(\hat y-y)$, subgradient $0$ at 0	targets the median, not the mean; robust to outliers
Huber	$\tfrac12 e^2$ if $\|e\|\le\delta$, else $\delta\|e\|-\tfrac12\delta^2$, with $e=\hat y-y$	sklearn `HuberRegressor`, robust regression	$e$ if $\|e\|\le\delta$, else $\delta\,\mathrm{sign}(e)$	quadratic near zero, linear in the tails — MAE's robustness without MAE's flat gradient
Binary cross-entropy	$-\big[y\log p+(1-y)\log(1-p)\big]$	logistic regression, binary output heads	wrt logit $z$: $p-y$, with $p=\sigma(z)$	that clean gradient is why sigmoid + BCE is the default pairing
Categorical cross-entropy	$-\sum_{k=1}^{K}y_k\log p_k$	softmax classifiers, language models	wrt logits $z$: $p-y$ (vector form)	softmax + CE share the same cancellation trick
Hinge	$\max(0,\,1-y\,f(x))$, with $y\in\{-1,+1\}$	linear SVM	wrt $f(x)$: $-y$ if $y f(x)\lt 1$, else $0$	optimises the margin, not calibrated probabilities; only support vectors get gradient
KL divergence	$\mathrm{KL}(p\\|q)=\sum_x p(x)\log\frac{p(x)}{q(x)}$	distillation, VAE regulariser, information criteria	wrt $q_k$: $-\frac{p_k}{q_k}$	asymmetric: $\mathrm{KL}(p\\|q)$ (mass-covering) $\ne\mathrm{KL}(q\\|p)$ (mode-seeking)
KL-regularised objective	$\mathbb{E}_{q}[-\log p(y\mid x)]+\beta\,\mathrm{KL}(q(z\mid x)\\|p(z))$	VAE's ELBO, RLHF reward regularisation	— (via reparameterisation)	fit the data and stay close to a prior; $\beta$ trades the two off
Generic regularised	$\frac1n\sum_i L(f(x_i;\theta),y_i)+\lambda\,\Omega(\theta)$	everything above, with $\Omega=\\|w\\|_2^2$ (ridge/weight decay) or $\\|w\\|_1$ (lasso)	$\nabla_\theta\frac1n\sum_i L + \lambda\nabla\Omega$	the $\frac1n$ is why $\lambda$ is comparable across dataset sizes — but libraries differ on whether they average or sum, so read the docs

Two places a bare log explodes

$\log 0=-\infty$. In code, clamp probabilities into a small safe range (e.g. np.clip(p, 1e-12, 1 - 1e-12)) or, better, compute cross-entropy from the logits with a fused, numerically stable routine (scipy.special.expit / np.logaddexp / your framework's BCEWithLogitsLoss). The second explosion is the reverse: predicting $p=0.999999$ confidently on one mislabelled point can push the loss to infinity.

Section 9

Algorithm ↔ math map

📖 Module 08 · Module 09 · where this math goes next🎯 name the load-bearing math before reading the code

Problem	Algorithm	Backbone concept	Module
Regression	Linear regression (OLS)	least squares, normal equations $(X^\top X)^{-1}X^\top y$, gradient descent	02 · 08
Regression, stable	Ridge (L2)	closed form $(X^\top X+n\lambda I)^{-1}X^\top y$, ill-conditioning fixed by $\lambda I$	07 · 08
Regression, sparse	Lasso (L1)	subgradients, soft thresholding, sparsity via the diamond's corners	07 · 08
Binary classification	Logistic regression	sigmoid + Bernoulli MLE + cross-entropy + GD; gradient $\frac1nX^\top(p-y)$	04 · 08
Classification with probabilities	Naive Bayes	Bayes' rule + conditional independence (features multiplied as if independent)	04 · 08
Clustering	k-means	distances, $\min_c\sum_{i\in S_c}\\|x_i-\mu_c\\|^2$, alternation of assignment and update	02 · 08
Dimensionality reduction	PCA / SVD	eigenvectors of the covariance matrix, projections $Z=X_cW_k$, truncation $A\approx U_k\Sigma_kV_k^\top$	02 · 08
Nonlinear classification / regression	Decision tree	recursive binary splits maximising information gain or minimising Gini	06 · 08
Ensemble, low variance	Random forest	bagging + feature subsampling — averaging independent errors shrinks variance	05 · 08
Ensemble, low bias	Gradient boosting	gradient descent in function space: each new tree fits the current negative gradient	03 · 08
Max-margin classification	SVM	convex optimisation, Lagrangian/KKT duals, kernels = implicit feature map	07 · 08
Retrieval / smoothing	k-NN	norms and similarity; the curse of dimensionality makes distance meaningless in high $d$	02 · 08
Next chapter — the same math, scaled up (Module 09)
Universal function fitting	MLP	matrix products + the chain rule: backpropagation is the multivariable chain rule, run backwards	09
Sequences, context	Transformer	matrix products + softmax attention; scaled dot products are cosine-like similarity	09
Generative, latent variables	VAE	the ELBO: Jensen's inequality + KL divergence between approximate and true posterior	09
Generative, images	Diffusion models	score matching (a gradient field), noise schedules, chain rule through many steps	09

If you can name the backbone column before reading an algorithm's code, you already know where it will break.

Section 10

NumPy idioms for all of the above

🎯 $n\times d$ arrays, no Python loops⚠ every pitfall here cost someone an hour

Task	Code	Shape / gotcha
Matrix × vector	`X @ w`	`(n,d) @ (d,)` → `(n,)`. Use `@`, never `*` (that is element-wise).
Matrix × matrix	`X @ W`	`(n,d) @ (d,k)` → `(n,k)`: a layer / projection.
Dot with itself	`v @ v`, `np.linalg.norm(v)`	norm defaults to L2; `ord=1` / `np.inf` for the others.
Solve $Aw=b$	`np.linalg.solve(A, b)`	never `inv(A) @ b`; `lstsq` when $A$ is singular.
Determinant / inverse	`np.linalg.det(A)`, `np.linalg.inv(A)`	both raise on singular input; det near 0 ⇒ ill-conditioned.
Eigen (symmetric)	`vals, vecs = np.linalg.eigh(Sigma)`	`eigh` for symmetric/Hermitian, values ascending — take the last $k$ for PCA. `eig` gives unsorted, possibly complex results.
SVD	`U, s, Vt = np.linalg.svd(X, full_matrices=False)`	economy shapes: `U (n,k)`, `s (k,)`, `Vt (k,d)`. PCA: `W = Vt[:k].T`.
Centre & scale	`Xc = (X - X.mean(0)) / X.std(0)`	add `keepdims=True` if you build it manually; standardise before PCA or distance-based methods.
Covariance	`Xc.T @ Xc / (n - 1)`	`np.cov(X, rowvar=False)` gives the same $d\times d$ matrix.
Row-wise means	`X - X.mean(axis=0, keepdims=True)`	`keepdims` is the whole trick — without it the broadcast goes along the wrong axis.
One-hot	`np.eye(K)[y]`	`y` must be integers in `[0, K)`; result is `(n, K)`.
Gradient clipping	`g *= min(1.0, c / (np.linalg.norm(g) + 1e-12))`	global-norm clip at threshold `c`.

Shape-safe least squares, with the gradient checked against finite differences — the pattern to copy when you write your own objective:

import numpy as np
rng = np.random.default_rng(0)

n, d = 8, 3
X = rng.normal(size=(n, d))   # (n, d) — n samples, d features
y = rng.normal(size=n)        # (n,)   — one target per sample
w = rng.normal(size=d)        # (d,)   — one weight per feature

res  = X @ w - y              # (n,)   — residual per sample
loss = np.mean(res ** 2)      # scalar — ||Xw - y||^2 / n
grad = 2 * X.T @ res / n      # (d,)   — analytic gradient wrt w

# numeric gradient must agree; this is the check that catches shape/axis bugs
eps = 1e-6
num = np.empty(d)
for j in range(d):
    w2 = w.copy(); w2[j] += eps
    num[j] = (np.mean((X @ w2 - y) ** 2) - loss) / eps

print(loss, np.round(grad, 3), np.allclose(grad, num, atol=1e-6))  # expect …, (3,), True

Broadcasting rules that bite, and the two idioms that fix them:

import numpy as np
X = np.arange(6).reshape(3, 2)          # (n, d) = (3, 2)

# (n,1) + (1,m) -> (n,m): the one broadcast you want, for outer sums
grid = np.arange(2)[None, :] + np.zeros((3, 1))   # (3, 2)
print(grid.shape)                                  # (3, 2)

# (n,d) + (n,1) -> (n,d): often a mistake — shifts each ROW by its own offset
rowwise = X + X.mean(axis=1, keepdims=True)        # (3, 2), not what you meant
print(rowwise.shape)                               # (3, 2)

# correct: column means, kept as a (1, d) row so they broadcast down every column
Xc = X - X.mean(axis=0, keepdims=True)             # (3, 2), column-centred
print(np.round(Xc.mean(axis=0), 12))               # [0. 0.]

Sigma = Xc.T @ Xc / (X.shape[0] - 1)               # covariance, (2, 2)
print(Sigma.shape, np.allclose(np.diag(Sigma), Xc.var(axis=0, ddof=1)))  # (2, 2) True

y = np.array([0, 1, 0])                            # class ids
print(np.eye(3)[y].shape)                          # (3, 3) one-hot

Where this reference points next

This page is the lookup surface; the practice lives in the modules linked from each table, and the modelling side in the companion notebook Math for ML Relearn. When a formula here stops feeling mechanical, that is the signal to re-open its module and redo the implement pass — not to re-read the cheat sheet.