Calculus: change, direction, chain

Every training loop is the same sentence: look at how wrong you are, work out which way is less wrong, step that way. The middle part is calculus. This module builds the chain — derivative, gradient, curvature, Jacobian, chain rule — and spends its best section on the payoff: a full backpropagation worked with real numbers, so "autograd" becomes arithmetic you can do on paper.

Cold questions — answer from memory, in writing

Model answers

Only open these after your own answers are written down.

1 · Why change, and why now

A training step says: the parameters moved in a direction that made the error smaller. Three words do the work — "moved" (change), "direction" (a vector, not a number), "smaller" (a scalar to compare). Calculus was invented to talk about exactly those three words. ML did not borrow a tool from physics; it rediscovered the same problem with different nouns.

Newton and Leibniz built calculus in the 1600s to describe motion — position changes, velocity changes — because physics had become a question of rates. Two and a half centuries later the same machinery runs the same play: the loss is a function of the parameters, the gradient is its derivative, and the optimiser is a body rolling downhill under a force (the gradient) instead of gravity.

2 · The derivative, from the limit to the intuition

Start with what you can compute: a secant through two points on a curve, slope = rise/run. Then push the points together. The secant pivots toward the curve and becomes the tangent — the straight line that best touches the curve at that one point. That limit is the derivative.

Worked example, $f(x)=x^2$ at $x=3$. The quotient simplifies before the limit — the $h$ in the denominator cancels, which is why the limit exists:

One number, three readings. Keep all three — they get used in different places:

• Slope of the tangent. The geometric picture: draw the curve, draw the line grazing it at $x$, read its steepness. The sign is a statement about direction of increase — negative means downhill as $x$ grows.
• Sensitivity. Output per input, $\Delta\text{out}/\Delta\text{in}$. The engineer's reading: "how much does the loss react to a nudge in this parameter?" A parameter with derivative near $0$ is one the loss does not care about.
• Best local linear approximation. Near $x$ the function is almost the line $f(x_0) + f'(x_0)(x-x_0)$. Everything downstream — gradients, step sizes, convergence arguments — runs on this approximation, not on $f$ itself.

3 · Rules, each with a reason

You memorised these once for an exam and lost them. This time attach each rule to the reason it exists.

Product rule — where the two terms come from. Nudge both factors and expand, exactly like $(a+b)^2$: each factor gets a turn to change while the other sits still.

Quotient rule (losses and normalisations divide), and the chain rule for composition — the two you will use most:

The table is limited to the functions ML actually differentiates:

The $\ln x$ row deserves one sentence: because $\frac{d}{dx}\ln x = \frac{1}{x}$, differentiating a log-likelihood replaces nasty products with sums and simple ratios. That is why we maximise log-likelihoods — the maths gets cheap exactly where it matters.

The sigmoid derivative, in four steps. Reproduce this without notes; it appears in every derivation of logistic regression:

1. Write $\sigma(x) = (1+e^{-x})^{-1}$ — a power of something, so the chain rule applies.
2. Differentiate: $\sigma'(x) = -(1+e^{-x})^{-2}\cdot(-e^{-x}) = \frac{e^{-x}}{(1+e^{-x})^2}$ (the minus signs cancel).
3. Split the numerator with $e^{-x} = (1+e^{-x})-1$, giving $\frac{1}{1+e^{-x}} - \frac{1}{(1+e^{-x})^2}$.
4. Both pieces are powers of $\sigma$: $\sigma'(x) = \sigma(x) - \sigma(x)^2 = \sigma(x)(1-\sigma(x))$.

4 · Partial derivatives and the gradient

Models have millions of parameters, so $f:\mathbb{R}^n \to \mathbb{R}$. A partial derivative freezes all but one coordinate and asks the single-variable question about that one:

The gradient collects all of them into one vector — a direction with one entry per parameter, the same shape as the weights, which is what makes subtraction legal:

Three geometric facts, each sketchable in ten seconds:

• Steepest increase. The gradient points where $f$ grows fastest at that point; descent goes against it.
• Perpendicular to level sets. Draw the contour where $f$ is constant — the gradient sticks straight out of it, because moving along a contour does not change $f$.
• Everything else is a projection. The rate of change along a unit direction $\mathbf{u}$ is $\nabla f\cdot\mathbf{u} = \|\nabla f\|\cos\theta$, largest when $\theta=0$ — that proof of "steepest", not a claim.

Worked example. $g(x,y)=x^2+y^2$ at $(1,2)$: differentiating $x^2$ with $y$ held fixed gives $2x$, and vice versa.

One gradient descent step subtracts a scaled version. $\eta$ (eta) is the learning rate: it converts a gradient (slope units) into a step (parameter units).

5 · Second derivatives: curvature, minima and saddles

The first derivative says which way is downhill right now; the second says whether downhill is getting steeper or flatter — the curvature, and with it the local shape:

With many variables there is a second derivative per pair of coordinates, so they organise into a symmetric matrix — the Hessian:

At a critical point ($\nabla f = 0$) the test generalises: $H$ positive definite ⇒ every direction curves upward ⇒ local minimum; negative definite ⇒ maximum; mixed signs ⇒ a saddle point, which looks like a minimum from one angle and a maximum from another. In high dimensions saddles are abundant and local minima are rare — one sentence now, developed in Module 07.

In plain English: the first term is where you are, the second is the tilt under your feet (zero at a critical point — which is why first-order methods stall there), the third is how the ground bends. Newton's method uses all three to jump to the bottom of a quadratic bowl; gradient descent uses only the second and inches downhill. And a large Hessian means the gradient changes fast over short distances — a narrow valley demanding small steps — while a flat plain tolerates big ones. That is the whole intuition behind Adam: rescale each coordinate's step by an estimate of its curvature.

6 · Jacobian: gradients of vector-valued functions

Gradients are for scalar outputs; layers return vectors. For $f:\mathbb{R}^d \to \mathbb{R}^k$, stack the $k$ output gradients as rows and you get the Jacobian, a $k \times d$ matrix:

Shapes, because shapes are 80% of the bugs: outputs along rows, inputs down columns. When $k=1$ the Jacobian collapses to one row — the gradient written as a $1\times d$ matrix. A gradient is a degenerate Jacobian, not a different species. And for a single linear layer $f(\mathbf{x}) = W\mathbf{x}$ with $W \in \mathbb{R}^{k\times d}$, the Jacobian is the weight matrix itself: $J = W$.

Why you should care: a whole layer's local behaviour is a matrix, so the vector chain rule is a product of these matrices (Section 7). But you never want to build them — a million-by-million layer has $10^{12}$ entries. Frameworks instead compute only $J$ times a vector (a Jacobian–vector product), in time proportional to the function itself. That trick is reverse-mode automatic differentiation, named in Section 10.

7 · The chain rule is backpropagation

Scalar case first — the formula from the warm-up, stated again on purpose:

Now the vector version, which is what backprop runs. If scalar $L$ is computed from an intermediate vector $\mathbf{y} = \mathbf{y}(\mathbf{x})$, the gradient w.r.t. $\mathbf{x}$ is the incoming gradient multiplied by the transposed Jacobian of the local step:

A full backpropagation, with numbers. The smallest network that still has a hidden non-linearity: one input $x$, one sigmoid, one output, squared loss.

Walk the chain backwards, symbols first, one factor per edge. For $w_2$: it reaches $L$ only through $z_2$, and $z_2 = w_2 a_1$ so $\partial z_2/\partial w_2 = a_1$:

For $w_1$ the path is longer — through $z_2$, then $a_1$, then $z_1$ — so four factors multiply:

Numbers: $x=1$, $w_1=0.5$, $w_2=1.2$, $y=1$. Forward first — these are the values the backward pass reuses:

Both gradients are negative: increasing either weight lowers $L$ here, which is exactly what a descent step will do. Pass 3 checks both numbers against finite differences.

8 · Gradients of the losses you will actually differentiate

These four are the payoff of the module. Learn to re-derive them, not just recite them — each is five lines of Section 3's rules plus a chain rule.

Derivation, shapes checked. Expand: $L = \frac{1}{n}\sum_{i=1}^{n}(x_i^\top w - y_i)^2$. Differentiating w.r.t. $w_j$, the chain rule brings down $2(x_i^\top w - y_i)$ and $\partial(x_i^\top w)/\partial w_j = x_{ij}$ (only $x_i$'s $j$-th coordinate touches $w_j$). So $\frac{\partial L}{\partial w_j} = \frac{2}{n}\sum_i (Xw-y)_i x_{ij}$ — column $j$ of $X$ times the residual vector, i.e. row $j$ of $X^\top(Xw-y)$. Shapes: $X$ is $n\times d$, $w$ is $d$, $y$ is $n$; so $Xw-y$ is $n$ and $X^\top$ ($d\times n$) times it is $d$ — a gradient the size of $w$, as it must be. In code: (2/n) * X.T @ (X @ w - y).

Sigmoid + binary cross-entropy. Differentiate with respect to $z$ (we stop at the pre-activation and let backprop carry it on), using $\frac{d}{dz}\ln\sigma = \frac{\sigma'}{\sigma}$ and $\frac{d}{dz}\ln(1-\sigma) = \frac{-\sigma'}{1-\sigma}$:

Those denominators are the cancellation to watch: $\sigma'/\sigma = 1-\sigma$ and $\sigma'/(1-\sigma) = \sigma$. Multiply out and two terms remain:

The beautiful cancellation: an activation whose derivative is built from its own output, paired with a loss containing that output inside logarithms, cancels to the simplest possible gradient. Not luck — maximum likelihood wearing a nice outfit — and the reason sigmoid and cross-entropy are never casually swapped for other pairings.

Sketch. The softmax Jacobian is $\frac{\partial p_c}{\partial z_j} = p_c(\mathbb{1}[c=j] - p_j)$. Substitute into $\frac{\partial L}{\partial z_j} = -\sum_c \frac{y_c}{p_c}\frac{\partial p_c}{\partial z_j}$: the $p_c$ in the denominator cancels the one in the Jacobian, leaving $-y_j + p_j\sum_c y_c = p_j - y_j$, since $\sum_c y_c = 1$. Same cancellation, one dimension up.

One line: $\|w\|^2 = \sum_j w_j^2$ and each coordinate gives $2w_j$. Practical consequence: the weight-decay step is $w \leftarrow w - \eta\,2\lambda w = (1-2\eta\lambda)w$ — a multiplicative shrink every iteration, which is what "decay" should mean.

9 · Integrals: the other half, used more rarely but needed

Differentiation splits a change into rates; integration rebuilds a whole from rates — addition in the limit. Geometrically $\int_a^b f(x)\,dx$ is the signed area under the curve, defined as the limit of summing ever-thinner rectangles: $\int_a^b f(x)\,dx = \lim_{k\to\infty}\sum_{i=1}^{k} f(x_i^*)\,\Delta x$.

No proof — this notebook does proofs only when they pay rent — but the paragraph worth remembering: if $F$ is an antiderivative of $f$, the accumulated area from $a$ to $x$ has derivative $f(x)$, because it grows exactly as fast as the curve is tall. Two functions with the same derivative differ by a constant; fixing it at one endpoint gives $F(b)-F(a)$. That is the whole theorem, and it is why "find the antiderivative" solves "find the area".

The most common integral in ML is the expectation — the continuous version of the probability-weighted average $\sum_x x\,P(X=x)$, with sums replaced by areas:

Where else ML meets integrals, in increasing order of frequency:

• Expectation of a function. $\mathbb{E}[g(X)] = \int g(x)\,p(x)\,dx$ — variance, risk, and every "average over the data distribution" statement you will meet.
• Probability is volume. $P(a \le X \le b) = \int_a^b p(x)\,dx$: a probability is the area under the density, so densities must integrate to $1$ — the constraint that produces every normalising constant in every distribution.
• The Gaussian integral, once. $\int_{-\infty}^{\infty} e^{-x^2}\,dx = \sqrt{\pi}$ — a beautiful fact about a boring-looking integrand, and the origin of the Gaussian's $\sqrt{2\pi\sigma^2}$.

So the "mysterious" $\sqrt{2\pi\sigma^2}$ is not decoration — it is exactly the number required to make the integral equal $1$. Module 04 completes the picture: expectation, variance, and where least squares comes from.

10 · Symbolic vs numeric vs automatic differentiation

You need derivatives in code. There are three ways to get them; knowing all three lets you choose, and lets you recognise which one your framework is running.

Autodiff is what ML uses, and it is neither of the two things people assume. It is not numerical — no approximation, no cancellation, exact to floating-point precision. And it is not symbolic — it never builds a giant expression. It stores the operations you ran (the graph), then walks that graph backwards applying the chain rule one node at a time, exactly as you did in Section 7, at a small constant factor over the forward pass whatever the dimension. requires_grad=True starts the recording; loss.backward() runs the walk. The word you will see is autograd — now you know it means "the chain rule on a computation graph".

Block 1 · The gradient check every engineer runs

Analytic gradient vs finite differences. If they disagree by more than about $10^{-8}$, one of them is wrong — in practice, your code.

import numpy as np
rng = np.random.default_rng(0)

# central difference: symmetric error is O(h^2), twice as good as the naive version
def fd_grad(fn, p, h=1e-5):
    g = np.zeros_like(p)
    for i in range(p.size):
        e = np.zeros_like(p); e[i] = h
        g[i] = (fn(p + e) - fn(p - e)) / (2 * h)
    return g

# 1) a toy function you can differentiate by eye
f      = lambda p: p[0]**2 + p[1]**2
grad_f = lambda p: np.array([2*p[0], 2*p[1]])
p = np.array([1.5, -2.0])
print("f  analytic:", grad_f(p), " numeric:", fd_grad(f, p),
      " err:", np.abs(grad_f(p) - fd_grad(f, p)).max())   # expect < 1e-8

# 2) the real thing: MSE on synthetic data
n, d = 20, 4
X = rng.normal(size=(n, d))
y = X @ rng.normal(size=d) + 0.1 * rng.normal(size=n)
w = rng.normal(size=d)
mse      = lambda w: np.mean((X @ w - y)**2)
grad_mse = lambda w: (2/n) * X.T @ (X @ w - y)
print("MSE analytic:", np.round(grad_mse(w), 6))
print("MSE numeric :", np.round(fd_grad(mse, w), 6))
print("max error   :", np.abs(grad_mse(w) - fd_grad(mse, w)).max())   # expect < 1e-8

Both errors land near $10^{-11}$. If you ever see $10^{-3}$, you have found a real bug — a missing factor of $2$, a forgotten $1/n$, a transposed matrix. This is a gradient check, and it is how any new custom layer gets validated before anyone trusts a training run to it.

Block 2 · Gradient descent from scratch, three learning rates

import numpy as np
import matplotlib.pyplot as plt

grad = lambda p: 2 * p                      # nabla f for f(x,y) = x^2 + y^2

def descend(eta, n_steps=30, start=(2.5, -2.0)):
    p = np.array(start, float); path = [p.copy()]
    for _ in range(n_steps):
        p = p - eta * grad(p)               # the update rule, nothing else
        path.append(p.copy())
    return np.array(path)

# contour map of the loss, with each trajectory drawn on top
xs = np.linspace(-3, 3, 200); X, Y = np.meshgrid(xs, xs)
plt.contour(X, Y, X**2 + Y**2, levels=12)

for eta, style in [(0.1, '-'), (0.001, '--'), (1.1, ':')]:
    path = descend(eta)
    plt.plot(path[:, 0], path[:, 1], style, marker='o', ms=3, label=f"eta={eta}")
    print(f"eta={eta:<6} after 30 steps: {np.round(path[-1], 4)}")

# eta=0.1   -> [ 0.0031 -0.0025]  converged: factor per step = 1-2*eta = 0.8
# eta=0.001 -> [ 2.3543 -1.8834]  crawling: factor 0.998, barely moves
# eta=1.1   -> [ 593.44 -474.75]  diverging: factor -1.2, |factor| > 1
plt.legend(); plt.axis('equal'); plt.show()

On this function each step multiplies the position by $(1-2\eta)$, so all three outcomes read off one number: $0.8$ converges, $0.998$ crawls, $-1.2$ diverges. Real losses are not quadratics, but the local picture is the same — and this plot is what to sketch when someone asks why their loss exploded after they raised the learning rate.

Block 3 · Backprop by hand, then verified

import numpy as np
sig = lambda z: 1 / (1 + np.exp(-z))

def loss_and_grad(w1, w2, x, y):
    z1 = w1 * x
    a1 = sig(z1)                     # save it: sigma'(z1) is built from a1
    z2 = w2 * a1
    L  = (z2 - y)**2
    # backward pass — one factor per edge, in reverse
    dL_dz2 = 2 * (z2 - y)
    dL_dw2 = dL_dz2 * a1
    dL_dz1 = dL_dz2 * w2 * a1 * (1 - a1)   # chain through the sigmoid
    dL_dw1 = dL_dz1 * x
    return L, np.array([dL_dw1, dL_dw2])

x, y, w = 1.0, 1.0, np.array([0.5, 1.2])
L, g = loss_and_grad(*w, x, y)
print(f"L = {L:.6f}   grad = {np.round(g, 6)}")
# L = 0.064034   grad = [-0.142722 -0.315025]  — matches the paper derivation

# now force the machine to agree
loss_only = lambda w1, w2: loss_and_grad(w1, w2, x, y)[0]
h = 1e-6
fd = np.array([(loss_only(*(w + h*np.eye(2)[i])) - loss_only(*(w - h*np.eye(2)[i]))) / (2*h)
               for i in range(2)])
print("finite diff:", np.round(fd, 6), " max error:", np.abs(fd - g).max())  # ~1e-11

The two agree to twelve decimal places. That is the whole trust story of deep learning: the framework's backward() performs the same multiplications you just did, and you can audit it on any function you write.

Multiple choice — instant feedback

Open questions — answer aloud, without notes

Re-answer these in 1 / 3 / 7 days — cold, no notes. The derivatives you cannot re-derive are the ones you have not learned yet.