Information theory: why log-loss

You already call log_loss(y, proba) and move on. It looks arbitrary — someone chose -log, it is smooth, everybody kept it. It is not arbitrary. The number has units, it has a derivation, and it means something precise: how many bits, on average, your model wastes when it describes the truth using its own beliefs. This module builds that from scratch.

Warm-up: answer from memory

No notes. These five are the load-bearing answers of the module; if two come out shaky, that tells you exactly where to slow down in Pass 2.

1 · Why "information" is a measurable quantity

In 1948 Claude Shannon, an engineer at Bell Labs, published A Mathematical Theory of Communication. The question was a telecoms question: a channel carries symbols with some probability distribution and some noise — how many bits can you actually push through it and still reconstruct the message at the other end? His answer, the noisy-channel coding theorem, was that there is a sharp number — the channel capacity — below which reliable transmission is possible and above which it is impossible. Not "harder". Impossible.

To get there he had to do something more radical than measure channels: he had to give "information" a definition with units. The reframing that makes everything else follow is one sentence:

That inversion is the whole trick. We instinctively say a headline is "informative" because it matters; Shannon says it is informative because it was unlikely. The two usually correlate, but only the second one is arithmetic. And it is anchored to something physical: coding. If an outcome happens with probability $p$, the shortest sensible message that picks it out of the alternatives is about $-\log_2 p$ bits long — a fair coin needs 1 bit, one of eight equally-likely symbols needs 3 bits, a certain event needs 0. Short message ↔ likely event, and the length is the information.

2 · Surprise: $I(x) = -\log_2 p(x)$

The surprise of an outcome — its self-information — is the negative log of its probability:

This function is not picked for convenience. Three requirements pin it down, and only one of them is negotiable:

1. More likely means less informative. If $p_1 \gt p_2$ then $I(p_1) \lt I(p_2)$. A boring event should not shout.
2. Certainty is worth nothing. $I(1) = 0$: if the event was guaranteed, it taught you nothing. This is why the formula needs a minus sign — $\log 1 = 0$ already, and $p \lt 1$ makes the log negative.
3. Independent surprises add. Hearing $x$ and $y$ (independent) should be worth the sum of hearing each: $I(xy) = I(x) + I(y)$.

Requirement 3 is the one that forces the log. Independence means $p(xy) = p(x)\,p(y)$, so the function must obey

Take $b = a$ to get $f(a^2) = 2f(a)$, then $f(a^n) = nf(a)$ by induction, and the same argument for rational exponents gives $f(a^{m/n}) = \tfrac{m}{n}f(a)$. The only monotone function with that property is a multiple of the logarithm: $f(p) = -c\log p$. Requirements 1 and 2 pick the sign and pin $f(1) = 0$; the constant $c$ just picks your unit. And there is no fourth option hiding somewhere: any other shape (like $1/p$) either adds wrong, or fails to vanish at $p = 1$.

Bits and nats. Base 2 gives bits — the unit Shannon wanted, and the one that makes the coding story literal. Base $e$ gives nats. Machine learning uses the natural log throughout, for one boring and decisive reason: $\frac{d}{dx}\ln x = 1/x$ instead of $\frac{1}{x\ln 2}$, so every gradient you write later is one constant cleaner. The choice of base never changes an argmax — changing base multiplies every loss by a constant — but it does change the number you print.

3 · Entropy: the expected surprise

Surprise is a property of one outcome. To get a property of the distribution — something you can say before the draw — take the expectation of the surprise under that distribution. That is entropy:

It is maximised by the uniform distribution — "the least you can say about the outcome" — and for two outcomes you can see the maximum directly. Write $H(p) = -p\log_2 p - (1-p)\log_2(1-p)$, differentiate:

Shape in words: an inverted-U, zero at both ends (a certain outcome is uninformative either way), symmetric about 0.5, and flat-ish at the top — moving from 0.5 to 0.6 costs only 0.03 bits, while moving from 0.9 to 0.95 costs 0.18. A few verified values:

Read the last column: a distribution that is 99% sure has almost no entropy left. That is exactly what you want a confident model to look like — and exactly what a miscalibrated model (0.99 stated, 0.70 real) is lying about.

Three readings of the same number, all true, all used — keep all three, because different parts of ML reach for different ones:

• (a) Uncertainty of the distribution. How much you do not know before the draw. This is the version an RL agent's entropy bonus maximises (keep the policy undecided) and a Bayesian posterior's entropy minimises (get sure).
• (b) Average code length of the optimal code. The compression reading — you cannot encode draws from $p$ using fewer than $H(p)$ bits per symbol on average. This is the version language models are judged by.
• (c) How diffuse the distribution is. A scale-free shape measure: $H = 0$ is a spike, $H = \log_2 K$ is perfectly flat. This is the version feature selection and clustering use ("does knowing this feature sharpen anything?").

Two conventions that stop bites: $0\log 0 := 0$, taken as a limit (since $p\log p \to 0$ as $p \to 0^+$) — an impossible outcome contributes nothing, not NaN. And the range is $0 \le H(p) \le \log_2 K$ for $K$ outcomes, with the upper bound only at the uniform.

4 · Cross-entropy: expected surprise under the wrong beliefs

Now the move that produces your loss function. Instead of averaging the surprise of $p$ under its own probabilities, average it under a different distribution $q$ — the model's beliefs — while the data still comes from $p$:

Two derivations, and you should own both:

1. The coding reading. You want to transmit data generated by $p$. Being stubborn (or ignorant), you build your code table from $q$ instead. Then an outcome $x$ costs you $-\log_2 q(x)$ bits, and the average cost over the real data is $\sum_x p(x)\cdot(-\log_2 q(x)) = H(p,q)$. It is the average message length when you code the truth with the wrong table.
2. The belief reading. Each outcome surprises the model by $-\log q(x)$ (that is its self-information under $q$), and you weight that surprise by how often the outcome actually happens, $p(x)$. So $H(p,q)$ is the expected surprise of the true process, evaluated with the model's beliefs.

Both readings say the same thing: cross-entropy is entropy plus the penalty for being wrong. The decomposition is three lines — add and subtract $\sum_x p(x)\log p(x)$:

That is the punchline the whole module exists for. When you call CrossEntropyLoss or log_loss, you are not "choosing a convenient objective": you are minimising $D_{\mathrm{KL}}(p\Vert q)$ in disguise, which is the unique quantity that is zero exactly when your beliefs match the world.

The ML tie: one-hot labels collapse it to log-loss

For classification the label distribution is a one-hot vector — all the probability sits on the true class $k^\star$. Substituting $p_k = 1$ for $k = k^\star$ and $0$ otherwise kills every term except one:

That is the log-loss. For two classes with predicted probability $\hat y$ of the positive class it becomes binary cross-entropy:

Three numbers, all natural log, all verified — this is what your loss actually returns:

5 · KL divergence: how wrong is wrong?

Cross-entropy contains two things: the truth's own entropy (out of your control) and the part you are responsible for. That part has a name:

Non-negative, with equality iff $p = q$. The reason is Jensen's inequality on the concave logarithm (section 6 proves it properly). Three lines:

Asymmetric. Take a true distribution $p = (0.9, 0.1)$ and a model's $q = (0.6, 0.4)$ over the same two outcomes. Both directions are ordinary sums — and they disagree:

Same two distributions, 37% apart. The reason is which distribution does the weighting: $D(p\Vert q)$ averages over $p$'s outcomes, so it punishes $q$ for assigning too little mass where $p$ has mass ("you failed to cover me"); $D(q\Vert p)$ averages over $q$'s outcomes, so it punishes $q$ for putting mass where $p$ has none ("you invented things I never do"). The second direction is the harsher one here, and it is the one that forces $q$ to be zero wherever $p$ is zero.

Where ML runs KL — the list is longer than it looks, and it is not a coincidence:

• Priors and MAP. Maximising the log-posterior is MLE plus a log-prior term — a penalty that pulls the estimate back towards where the prior puts mass. Module 04 derives ridge and lasso exactly that way (Module 04 · Probability, section 9), and the pattern is this module's: charge an estimate for sitting far from a reference distribution, in log units. With a Gaussian prior the charge is a squared distance, which is ridge's $\lambda\lVert w\rVert^2$.
• Variational inference minimises $\mathrm{KL}(q\Vert p)$ between an approximate posterior and the true posterior — reverse KL, mode-seeking, because a narrow $q$ that sits inside one mode is affordable while a $q$ smeared over modes the posterior does not have is not.
• Knowledge distillation trains the student to match the teacher's full output distribution — minimising $D_{\mathrm{KL}}(\text{teacher} \Vert \text{student})$ — which is why the soft probabilities carry more information than the hard labels.
• VAEs: the evidence lower bound is exactly $\mathbb{E}[\log p(x\mid z)] - \mathrm{KL}(q(z\mid x)\Vert p(z))$, with the KL as the regulariser that keeps the latent space organised.
• Distribution shift measurement: compare train and serving feature distributions with KL (or its siblings) to detect that the world moved.
• Keeping a fine-tuned model near its base model: a KL penalty on outputs (or a divergence between logits) is the standard way to buy task performance without paying for it in the base model's general behaviour — the same term RLHF pipelines report as their KL reward penalty.

6 · Jensen's inequality, and why the log is doing real work

One inequality underwrites half of this module and most of unsupervised learning. For a concave function (and $\log$ is concave everywhere):

Picture it: draw the concave curve, mark two points on it and the chord between them. The chord sits below the curve, so the value at the average position (the curve, evaluated at $\mathbb{E}[X]$) is above the average of the values (the chord, evaluated at $\mathbb{E}[X]$). A concrete pair: $X \in \{1, 9\}$ with equal probability gives $\mathbb{E}[\log_2 X] = \tfrac12(0 + 3.17) = 1.585$ while $\log_2\mathbb{E}[X] = \log_2 5 = 2.322$. The gap is exactly the "Jensen gap" — and it is zero only when $X$ never varies, which is where the equality condition comes from.

Two consequences that matter in ML, and you have already met the first one:

1. Logs make products safe. $\log\prod_i p_i = \sum_i \log p_i$, and the log is monotone, so the argmax does not move — that is Module 04's "maximise the log-likelihood", now with its general principle attached: any monotone transform preserves optimisers, and only the log converts products into sums. Numerically it is not optional: a thousand factors of 0.9 multiply out to $1.7\times 10^{-46}$ — below float32's smallest normal ($1.2\times 10^{-38}$), and even below the smallest subnormal ($1.4\times 10^{-45}$), so it rounds to exactly zero — while the sum of logs, $-105.4$, is an ordinary number. Differentiating is the same gift: $\nabla\log\prod = \sum\nabla\log$, so a million-sample loss has a gradient that is a clean average of per-sample gradients.
2. The bound is the engine of EM and variational inference. When $\log p(x) = \log\sum_z p(x,z)$ has a sum inside the log, it is intractable. Jensen moves the log out: $$\log p(x) = \log\,\mathbb{E}_{z \sim q}\!\left[\frac{p(x,z)}{q(z)}\right]$$ $$\ge \mathbb{E}_{z \sim q}\!\left[\log\frac{p(x,z)}{q(z)}\right]$$ The right-hand side is the ELBO: computable, differentiable, and a guaranteed lower bound on the quantity you actually want. EM alternates between optimising it and tightening it (equality when $q$ is the true posterior); variational inference optimises it directly and accepts a gap. Neither algorithm is "approximating the hard thing" — they are optimising a bound, deliberately, with the equality condition of Jensen telling them how tight it can get.

7 · Mutual information: information shared between variables

Entropy measures uncertainty about one variable. Mutual information measures how much one variable reduces uncertainty about another — and it is not a new invention, it is the KL divergence you already have, applied to the question "does the joint differ from independence?":

The first identity says: compare the real joint against the independence product (the joint you would get if the variables ignored each other); if they are identical, KL is zero and there is nothing shared. The second says: shared information is what is left of $H(X)$ once $Y$ is known. Both readings give the same number — the chain rule $H(X,Y) = H(Y) + H(X\mid Y)$ turns one into the other in one line.

Worked by hand on a $2\times 2$ joint (rows $X$, columns $Y$):

• $H(X) = -0.5\log_2 0.5 - 0.5\log_2 0.5 = 1.0000$ bits (the row marginal is a coin).
• $H(Y) = -0.4\log_2 0.4 - 0.6\log_2 0.6 = 0.5288 + 0.4422 = 0.9710$ bits.
• $H(X,Y) = -(0.3\log_2 0.3 + 0.2\log_2 0.2 + 0.1\log_2 0.1 + 0.4\log_2 0.4) = 1.8464$ bits.
• $I(X;Y) = H(X) + H(Y) - H(X,Y) = \mathbf{0.1245}$ bits — from the unrounded values; the four-decimal terms above are each rounded.
• Cross-check: $H(X\mid Y) = H(X,Y) - H(Y) = 0.8755$ bits, which equals $H(X) - I(X;Y)$ ✓, and the KL of the joint against the product table $p(x)p(y)$ returns the same 0.1245.

Reading: knowing $Y$ removes only 0.12 of the 1.00 bits of uncertainty about $X$ — the variables are related, but weakly; 88% of $X$'s uncertainty survives. If the joint were the product table $(0.2, 0.3, 0.2, 0.3)$ every number above would give $I = 0$.

8 · Entropy in trees: information gain and Gini

A decision tree chooses the split that reduces uncertainty the most. The score is the parent's entropy minus the size-weighted entropy of the children:

Worked on a node with 9 positives and 5 negatives ($n = 14$): $H(\text{parent}) = -\frac{9}{14}\log_2\frac{9}{14} - \frac{5}{14}\log_2\frac{5}{14} = 0.9403$ bits. Two candidate splits, both computed the same way:

Wind wins, and not by a little: it cuts the weighted impurity from 0.9403 to 0.7475, while humidity barely stirs it (0.9253 — one child is almost as mixed as the parent). Note what the size-weighting does: the small 4-row child cannot dominate the score just because it is extreme. (The displayed sums are rounded to four decimals; every gain above is computed from unrounded values, which is why Block 3 prints them to six.)

Gini impurity and the two-thirds rule

CART's default criterion skips the logarithm entirely:

For the same parent, $G = 1 - (9/14)^2 - (5/14)^2 = 0.4592$; wind's children give $0.320$ and $0.375$, size-weighted $0.3357$, so the Gini gain is $0.1235$; humidity gives $0.480$ and $0.375$, weighted $0.4500$, gain $0.0092$. Both criteria rank wind first, and that is the point of the rule of thumb:

That ratio is measured, not asserted: over 1610 random candidate splits of a 9/5 parent, the ratio (entropy gain ÷ Gini gain) ranged from 1.52 to 1.66 with median 1.58, and in head-to-head trials where several candidates were ranked by both criteria, the same winner came out 94% of the time (98% for a balanced 50/50 parent; the ratio drifts upward to 2–4 on very lopsided nodes like 12/2, which is why the rule is quoted for near-balanced ones). Underneath, at the impurity level, the two are simply proportional-ish: over $p \in [0.1, 0.9]$ the ratio $H/G$ stays between 2.00 and 2.61 in bits.

So when to prefer which: Gini is the default because it is marginally cheaper (no log), and it tends to isolate the minority class a little more eagerly as it carves nodes. Entropy has units you can say out loud — "this split bought 0.19 bits of information" is a sentence with meaning, which Gini's 0.12 is not. Neither choice matters much: because they rank candidates almost identically, the trees they produce are usually the same tree, and any difference they do cause is smaller than the difference your max_depth and min_samples_leaf settings make.

9 · Perplexity and temperature: information theory in the LLM era

A language model is judged by the cross-entropy of its next-token predictions, quoted in the exponentiated form — because bits are not intuitive but "number of alternatives" is:

Read that as a sentence: a perplexity of 20 means the model is, on average, as unsure about the next token as if it were picking uniformly among 20 words; a perplexity of 8 means three bits per token. Because exponentiation is monotone, minimising cross-entropy and minimising perplexity are the same optimisation — perplexity is just the loss wearing a friendlier unit. Two hygiene rules: perplexities are only comparable across models that share a tokenizer (different vocabularies change the bits-per-token scale), and it is always computed on held-out text.

Second idea, the one you actually touch when calling an API: temperature. Given the model's distribution $p$, sampling at temperature $T$ means re-weighting and renormalising:

Three facts, in order of how often they are misunderstood. $T \to 0$ collapses the distribution onto its argmax: no entropy, no randomness, pure exploitation. $T \gt 1$ flattens: more entropy, more diversity in what gets sampled. And the argmax never moves — $p_i \mapsto p_i^{1/T}$ is monotone, so the ranking of tokens is identical at every temperature; what changes is the gap between them. Temperature re-weights a distribution; it does not change what the model believes is most likely.

That last fact is what makes temperature a clean control knob in an agentic system. When an agent samples its next action (a tool call, a plan step) from the model's distribution, $T \approx 0$ means "always take the model's best guess" — fast, deterministic, and brittle when the first choice is a locally good trap. Raising $T$ above 1 spreads probability across the runner-up actions, so the agent actually draws them and finds out whether they work: temperature is literally the exploitation vs exploration dial, implemented as an entropy dial. It is a sampling-time knob (weights, beliefs and argmax untouched), which is why you can sweep it at inference without retraining — and it is distinct from exploration driven by the agent's own uncertainty or by an explicit reward for novelty. Module 09's agentic section builds on exactly this.

10 · The map

Every quantity in this module, one line each. If you can fill this table from memory, you own the module:

Block 1 · Entropy, cross-entropy and KL from scratch

Twenty-odd lines of NumPy, no library, with the $0\log 0$ convention handled by masking. Then two independent checks: sklearn.metrics.log_loss should equal the cross-entropy exactly, and the decomposition $H(p,q) = H(p) + D_{\mathrm{KL}}$ should have zero residual.

import numpy as np
from sklearn.metrics import log_loss

# 0 log 0 := 0 -- mask the zero entries rather than letting log() return nan
def entropy(p):
    p = np.asarray(p, float); m = p > 0
    return float(-(p[m] * np.log(p[m])).sum())              # nats

def cross_entropy(p, q):     # expected surprise of p, scored with q's beliefs
    p = np.asarray(p, float); q = np.asarray(q, float); m = p > 0
    return float(-(p[m] * np.log(q[m])).sum())

def kl(p, q):                # H(p,q) - H(p): nothing but the gap between the two
    return cross_entropy(p, q) - entropy(p)

p = np.array([0.6, 0.3, 0.1]); q = np.array([0.5, 0.3, 0.2])
print(f"H(p)={entropy(p):.6f}  H(p,q)={cross_entropy(p, q):.6f}  KL={kl(p, q):.6f}")
print("residual of H(p,q) = H(p) + KL :", cross_entropy(p, q) - entropy(p) - kl(p, q))   # 0.0

# sklearn's log_loss IS H(p,q): 10 labels with frequencies p, model always says q
y10 = np.repeat([0, 1, 2], [6, 3, 1])
print(round(log_loss(y10, np.tile(q, (10, 1))), 6), round(cross_entropy(p, q), 6))        # 0.938024 0.938024
print(round(entropy(np.array([0.0, 0.5, 0.5])), 6))     # 0.693147 -- the 0 log 0 convention

Output: H(p)=0.897946 H(p,q)=0.938024 KL=0.040078, residual exactly 0.0, and the library agrees to the last printed digit. Note that KL is 0.040 nats — the model's beliefs cost 4% of a nat per sample on top of the truth's own 0.898 nats of uncertainty.

Block 2 · Cross-entropy is log-loss, on a real classifier

Fit logistic regression, then compute the loss two ways: log_loss(y, proba) and the BCE formula from section 4, on the same probabilities. Then inject one confident-but-wrong prediction into a batch of 100 and watch what the average does.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss

X, y = make_classification(n_samples=800, n_features=4, n_informative=3,
                           n_redundant=0, class_sep=1.5, random_state=0)
clf = LogisticRegression(max_iter=1000).fit(X, y)
phat = clf.predict_proba(X)[:, 1]

def bce(y, phat):                       # the formula you would write on paper
    phat = np.clip(phat, 1e-15, 1 - 1e-15)
    return float(-(y * np.log(phat) + (1 - y) * np.log(1 - phat)).mean())

print(log_loss(y, phat), bce(y, phat))  # 0.5606918096337622 0.5606918096337622

# one confident-wrong label inside a batch of 100 otherwise-perfect predictions
calm = -np.log(0.99)                    # correct, 0.99 confident: 0.01005
bad  = -np.log(0.001)                   # said 0.999, label said 0: 6.9078
print(calm, bad, (99 * calm + bad) / 100)   # 0.01005 6.9078 0.07903 -- 7.9x the mean

Two lessons. The two implementations agree to machine epsilon, because they are the same object — the library is not doing anything extra. And a single wrong label at 0.999 confidence costs 6.908 nats on its own, which lifts the batch mean from 0.010 to 0.079: a 7.9× jump from one sample out of a hundred. Label noise is not a rounding error in log-loss; it is the dominant term.

Block 3 · Information gain by hand, then what the tree picks

The 14-row toy set from section 8, as code: compute both criteria for both candidate features, then let DecisionTreeClassifier(max_depth=1) choose and read the chosen feature index off the fitted tree.

import numpy as np
from sklearn.tree import DecisionTreeClassifier

wind = np.array([0] * 10 + [1] * 4)          # 10 calm rows, 4 gusty rows
hum  = np.array([1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0])
y    = np.array([1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0])   # 9 yes, 5 no
X = np.column_stack([wind, hum])

def impurity(counts):                     # (entropy, gini) of a [pos, neg] pile
    pr = counts / counts.sum()
    h = -(pr[pr > 0] * np.log2(pr[pr > 0])).sum()
    return h, 1 - (pr ** 2).sum()

Hp, Gp = impurity(np.array([9, 5]))       # parent: 0.940286 bits, 0.459184 gini
for name, f in [("wind", wind), ("humidity", hum)]:
    Hw = Gw = 0.0
    for v in [0, 1]:                      # size-weighted impurity of the two children
        c = np.array([(y[f == v] == 1).sum(), (y[f == v] == 0).sum()])
        h, g = impurity(c)
        Hw += c.sum() / len(y) * h
        Gw += c.sum() / len(y) * g
    print(f"{name:9} gain_H={Hp - Hw:.6f}  gain_G={Gp - Gw:.6f}")
# wind      gain_H=0.192829  gain_G=0.123469
# humidity  gain_H=0.014956  gain_G=0.009184

for crit in ["gini", "entropy"]:          # what the library actually picks
    t = DecisionTreeClassifier(max_depth=1, criterion=crit, random_state=0).fit(X, y)
    print(crit, "picks feature", t.tree_.feature[0])   # both print 0 -- wind, as computed

Your hand computation and the library agree: wind wins under entropy (0.193 vs 0.015) and under Gini (0.123 vs 0.009), and tree_.feature[0] == 0 for both criteria — the two-thirds rule deciding a real split in front of you. And the ratio of the two gains, entropy ÷ Gini, is 0.1928/0.1235 = 1.56 — squarely inside the 1.52–1.66 band from section 8.

Block 4 · Perplexity and temperature

A toy next-token model: three positions of a tiny vocabulary, the model's distribution at each, and what actually happened. Cross-entropy first, then perplexity as its exponent — then the temperature sweep from section 9.

import numpy as np
# a 4-token vocab: the model's next-token distribution at 3 positions, and what happened
preds = np.array([[0.50, 0.30, 0.10, 0.10],
                  [0.20, 0.70, 0.05, 0.05],
                  [0.10, 0.10, 0.10, 0.70]])
actual = np.array([0, 1, 3])
bits = -np.log2(preds[np.arange(3), actual])
ce = bits.mean()
print("per-token bits:", np.round(bits, 4))              # [1. 0.5146 0.5146]
print("cross-entropy:", round(float(ce), 4),
      " perplexity:", round(float(2 ** ce), 4))          # 0.6764 1.5981
print("uniform guess over 4 tokens:", -np.log2(0.25), 2 ** 2.0)   # 2.0 bits, ppl 4.0

# temperature: q_i = p_i^(1/T), renormalised -- the sampling knob
p = np.array([0.7, 0.2, 0.07, 0.03])
for T in [0.5, 1.0, 2.0]:
    q = p ** (1 / T); q = q / q.sum()
    Hb = -(q[q > 0] * np.log2(q[q > 0])).sum()
    print(f"T={T}: H={Hb:.4f} bits  argmax={q.argmax()}  top={q.max():.4f}")
# T=0.5: H=0.4748 bits  argmax=0  top=0.9145
# T=1.0: H=1.2449 bits  argmax=0  top=0.7000
# T=2.0: H=1.7597 bits  argmax=0  top=0.4860    -- argmax never moves

The model spends 0.676 bits per token, so it is as confused as a uniform guess over $2^{0.676} = 1.60$ tokens — better than the 4-token vocabulary's uniform baseline (2 bits, perplexity 4). And in the sweep, entropy rises monotonically with $T$ while the argmax stays at index 0 at every temperature: $T$ changes how often the runner-ups get drawn, not which token the model prefers.

Multiple choice — instant feedback

Open questions — answer aloud, without notes

Re-answer these in 1 day, 3 days and 7 days from memory. Question 1 (the three-line decomposition) and question 5 (one-hot labels collapse to $-\log q_{\text{true}}$) are the two this module exists for — if you can produce both without looking, you can explain log-loss to anyone.