Machine Learning,
from first principles to production.

This module has one job: find the holes. Answer every question from memory, in writing before you scroll to the answers. The discomfort of half-remembering is the part that works — it is what tells your brain the material is worth keeping.

How to use this notebook

1. Attempt the diagnostic cold. Write answers first, no peeking. Ten minutes, badly written answers are fine.
2. Read with a pen. Each module has goals, core ideas, code, pitfall boxes, and recall cards.
3. Run the code. Type it, don't copy it. Then break it on purpose: change one number, predict, run.
4. Do the exercise — usually 15–30 minutes. Skipping it is how people finish 20 modules and learn nothing.
5. Tick "Mark done" only when you can explain it aloud without looking. Progress is saved in your browser.

Diagnostic: answer all 12 cold

Before any model: what ML is, what it isn't, and — most importantly — when it is the wrong tool. Most failed ML projects fail here, not in the math.

The definition you should use

"Improve at a task" means there is a target behaviour and a way to score it. "Experience" usually means historical examples (data), not a person being taught. And the measure of success must be computable — if you cannot write the function that scores a prediction, you cannot train anything.

The families

Supervised splits into regression (continuous number) and classification (discrete label, one or many). Nearly all beginner and most first production projects are supervised. 80% of applied ML in industry is tabular supervised learning — which is why Trees/Boosting (Module 07) matters more than people expect.

The loop

Notice it is a loop, not a line. The feedback arrow — real outcomes coming back as new labels — is the difference between a system that improves and one that slowly rots. A notebook only shows you the middle four boxes; production is mostly the outer two.

Is this a good ML problem?

Apply all three tests. Fail any one and pick a simpler answer:

1. Signal exists. There is a plausible statistical relationship between what you observe before the decision and what you want to predict. Past churn predicts future churn; tomorrow's lottery numbers do not.
2. Data exists — and is representative. Enough examples of the thing, from the same distribution you'll meet in production, with labels you can trust.
3. Actionability. Someone will do something differently when they see the score, and you can measure whether that helped.

You do not need calculus exams. You need four things: vectors and matrices (data has shape), derivatives (how to step downhill), probability (what a model output means), and statistics (what your data is lying about). That's this module.

1 · Linear algebra: data is shape

A dataset is a matrix X with shape (n, d): n rows = examples, d columns = features. A label vector y has shape (n,). A model's weights are vectors or matrices. When you debug shapes in NumPy or PyTorch, you are doing linear algebra.

• Dot product as similarity: vectors pointing the same way give a large positive value; opposite directions give negative. This powers recommendation and search.
• Matrix × vector: X @ w computes the weighted sum for every row at once — that's what a "layer" does.
• Transpose: flips rows/columns; you'll meet it in X.T @ X (normal equations) and in attention.
• Broadcasting: NumPy/PyTorch stretch a shape-(d,) vector across an (n, d) matrix — this is why a bias can be added to a whole batch without a loop.

import numpy as np

X = np.array([[1700, 3, 12],    # house: sqft, bedrooms, age
              [1400, 2, 20],
              [1900, 4,  5]])
y = np.array([500, 400, 640])   # price in $k
w = np.array([0.2, 40, -5])     # weights per feature
b = 50                          # bias / intercept

print(X.shape, y.shape, w.shape)   # (3, 3) (3,) (3,)
preds = X @ w + b                  # one line: every row at once
print(preds)                       # [470. 350. 570.]

# "which training example is most like this new one?"
new = np.array([1650, 3, 10])
sims = X @ new                     # dot products = similarity scores
print(sims.argmax())               # 0 → the first house

2 · Calculus: derivatives are slopes, gradients are directions

A derivative answers: if I nudge input a little, how much does the output move? Positive slope → output rises; negative → falls. Training is nothing more than repeatedly nudging each weight downhill on the loss surface.

With many inputs you get a gradient: the vector of all partial derivatives, ∇L = [∂L/∂w₁, ∂L/∂w₂, …]. It points in the direction of steepest ascent, so training subtracts it:

Worked example: one step of gradient descent

Model ŷ = w·x, single example x = 2, y = 10, start w = 1, learning rate η = 0.1, loss L = (ŷ − y)²:

1. Predict: ŷ = 1 × 2 = 2
2. Loss: (2 − 10)² = 64
3. Gradient: ∂L/∂w = 2(ŷ − y) · x = 2(−8)(2) = −32
4. Update: w ← 1 − 0.1(−32) = 4.2
5. Next round: ŷ = 8.4, loss 2.56 — already downhill.

3 · Probability: what a score actually means

• Joint / conditional: P(A,B) = P(A|B)·P(B). P(spam | "free") is what classifiers want; P("free" | spam) is what the training data gives you.
• Bayes' rule:

• Base-rate intuition: a test with 99% accuracy on a disease affecting 1 in 10,000 is mostly flagging healthy people — most positives are false. Precision requires the prior.
• Independence: P(A,B) = P(A)·P(B) when A tells you nothing about B. Naive Bayes (Module 08) pretends features are independent — a lie that often works.
• Expected value: E[X] = Σ x·P(x) — the long-run average; how you compare decisions under uncertainty.
• Entropy / uncertainty: a coin is max-uncertain (1 bit); a rigged coin is predictable. Decision trees split where uncertainty drops most.

4 · Statistics: what your data is lying about

NumPy for math, pandas for tables, scikit-learn for models. This module is muscle memory: if you can't write this from memory, type it three times.

Environment

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install numpy pandas scikit-learn matplotlib jupyter
jupyter notebook                   # or: jupyter lab

One rule: never install into your system Python. One virtual environment per project — that habit alone prevents weeks of "it works on my machine" later in Track D.

NumPy — fast homogeneous arrays

import numpy as np

a = np.array([3.0, 1.0, 4.0, 1.0, 5.0])
print(a.shape, a.dtype, a.ndim)     # (5,) float64 1

print(a.mean(), a.std(), a.min(), a.argmax())
print(a > 3)                        # boolean mask → [F F T F T]
print(a[a > 3])                     # filter with the mask → [4. 5.]

M = np.arange(12).reshape(3, 4)     # build a matrix
print(M.shape)                      # (3, 4)
print(M[:, 0])                      # all rows, first column
print(M.sum(axis=0))                # sum DOWN the rows → shape (4,)
print(M.sum(axis=1))                # sum ACROSS cols → shape (3,)
                                    # axis = the axis you COLLAPSE

# broadcasting: subtract column means without a loop
X = np.random.randn(100, 4)
X_centered = X - X.mean(axis=0)     # (100,4) - (4,) → stretched

pandas — labeled tables

import pandas as pd

df = pd.read_csv("customers.csv")
print(df.shape, df.dtypes)
print(df.head(3))
print(df.describe(include="all"))         # summary stats
print(df.isna().sum())                    # missing values per column
print(df["segment"].value_counts(normalize=True))   # class balance

# selection: [] for columns, .loc/.iloc for rows+columns
sub = df.loc[df["churn"] == 1, ["tenure", "plan", "charges"]]
top = df.nlargest(10, "charges")

# transforms
df["avg_monthly"] = df["total"] / df["tenure"].clip(lower=1)
df["plan_enc"] = df["plan"].map({"basic": 0, "pro": 1, "max": 2})

# aggregation — the bread and butter
report = (df.groupby("plan")
            .agg(customers=("churn", "size"),
                 churn_rate=("churn", "mean"),
                 avg_charge=("charges", "mean"))
            .sort_values("churn_rate", ascending=False))
print(report)

# joins
df = df.merge(plan_meta, on="plan", how="left")
print(df.duplicated().sum())              # check for dupes
df = df.dropna(subset=["churn"])          # drop rows missing the target

scikit-learn — one convention, every model

Every model — linear regression, random forest, k-means — uses the same three verbs. Transformers (scalers, encoders) have fit/transform but no predict. That uniformity is why you can swap models in ten lines.

Capstone of this module: a full baseline

import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix

# 1. data
df = pd.read_csv("telco_churn.csv")
df = df.dropna(subset=["Churn"])
print(df["Churn"].value_counts(normalize=True))     # always check balance!

# 2. features / target  (keep only what you'd have at prediction time)
X = df.drop(columns=["Churn", "customerID"])
y = (df["Churn"] == "Yes").astype(int)
X = pd.get_dummies(X, drop_first=True)              # one-hot categoricals

# 3. split BEFORE touching statistics
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)   # stratify keeps class balance

# 4. fit
scaler = StandardScaler().fit(X_tr)                 # fit on TRAIN only
model = LogisticRegression(max_iter=1000).fit(scaler.transform(X_tr), y_tr)

# 5. honest evaluation on untouched data
pred = model.predict(scaler.transform(X_te))
print(confusion_matrix(y_te, pred))
print(classification_report(y_te, pred, digits=3))

# 6. look at what it learned
imp = pd.Series(np.abs(model.coef_[0]), index=X.columns).nlargest(10)
print(imp)

Before you pick a model: decide what "correct" means, how you'll measure it without cheating, and what dumb baseline you must beat. Get this wrong and a fancier model only makes you more confident in a wrong number.

Step 1 · Frame before you fit

Write four sentences. If you can't, you're not ready to open a notebook:

1. The decision: who will do what differently with this prediction?
2. The input/output: what is X at the moment of prediction, and what is y?
3. The primary metric: the one number that decides ship / no-ship.
4. The baseline: the dumbest thing that could produce that number honestly.

Step 2 · Split the data the way reality splits it

Step 3 · Hunt leakage

Leakage = information about the target that would not be available at prediction time. It is the single most common reason a great dev score means nothing. Four disguises:

• Target leakage: a feature is a consequence of the outcome (days_until_cancel in a churn model; total_refunds in a refund model).
• Temporal leakage: you used future information (averages computed over the whole timeline, including the test period).
• Preprocessing leakage: scaling/imputation/feature selection fit on all data before splitting.
• Group leakage: duplicate or near-duplicate rows (same user, same session) on both sides of the split.

Step 4 · Baselines, in order of dumbness

1. Random / majority class: tells you the floor (and exposes accuracy traps).
2. Global statistic: always predict the train mean, or the last known value (persistence — shockingly strong for time series).
3. Human-level: what does a person get in 10 seconds with the same info?
4. Simple model: logistic regression, shallow tree, one rule.
5. Your model. If it doesn't beat step 4 by a margin wider than your noise, ship step 4 — it's cheaper, faster and explainable.

Evaluation toolkit

from sklearn.metrics import (confusion_matrix, classification_report,
                             precision_recall_curve, roc_auc_score,
                             mean_absolute_error, mean_squared_error, r2_score)

# --- classification ---
print(confusion_matrix(y_true, y_pred))          # [[TN FP],[FN TP]]
print(classification_report(y_true, y_pred, digits=3))
print("roc-auc:", roc_auc_score(y_true, proba))  # needs PROBABILITIES
print("pr-auc :", ... )                          # precision_recall_curve → auc()

# --- regression ---
print("MAE :", mean_absolute_error(y_true, y_pred))       # average |error|, same units as y
print("RMSE:", mean_squared_error(y_true, y_pred) ** 0.5)  # punishes big errors
print("R²  :", r2_score(y_true, y_pred))                   # variance explained vs predicting the mean

# --- slices: where does it fail? ---
report = X_test.assign(y=y_test, p=y_pred).groupby("segment").apply(
    lambda g: (g.y == g.p).mean())
print(report.sort_values())                     # overall 0.91, but segment X: 0.62

Linear regression is the "Hello, world" of ML — and still a top-5 model in production. It's also the clearest place to see loss, gradient descent, overfitting, and regularization all at once.

The model

Training = find the w, b that minimize the mean squared error between predictions and truth. We can't just "see" that answer with many features, but two facts matter: with zero features the optimum is the mean of y; with many features, the solution is a weighted blend where correlated features share credit unpredictably (multicollinearity).

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, r2_score

rng = np.random.default_rng(0)
X = rng.uniform(500, 3500, 300).reshape(-1, 1)      # sqft
y = 50 + 0.15 * X.ravel() + rng.normal(0, 25, 300)  # price = signal + noise

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)
m = LinearRegression().fit(X_tr, y_tr)
pred = m.predict(X_te)

print("weight:", round(m.coef_[0], 4), " bias:", round(m.intercept_, 2))
print("MAE:", round(mean_absolute_error(y_te, pred), 2),
      " R²:", round(r2_score(y_te, pred), 3))
# read it: +0.15 price units per sqft, baseline 50 when sqft = 0

Gradient descent, the three flavours

# gradient descent from scratch — the whole algorithm in 8 lines
w, b, eta, epochs = 0.0, 0.0, 0.01, 1000
for _ in range(epochs):
    pred = X_tr * w + b                       # forward
    err  = pred - y_tr
    grad_w = np.mean(2 * err * X_tr)          # dL/dw
    grad_b = np.mean(2 * err)                 # dL/db
    w -= eta * grad_w                         # step downhill
    b -= eta * grad_b
print("from scratch:", round(w, 4), round(b, 2))
print("sklearn    :", round(m.coef_[0], 4), round(m.intercept_, 2))
# both should land near 0.15 / 50 — same destination, different engines

Underfitting vs overfitting

from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

# underfit: a line on curved data
under = LinearRegression().fit(X_tr, y_tr)

# just right: mild flexibility
ok = make_pipeline(PolynomialFeatures(2), LinearRegression()).fit(X_tr, y_tr)

# overfit: degree 15 dances through every training point
over = make_pipeline(PolynomialFeatures(15), LinearRegression()).fit(X_tr, y_tr)

for name, model in [("line", under), ("deg2", ok), ("deg15", over)]:
    print(name,
          "train R²", round(model.score(X_tr, y_tr), 3),
          "val R²  ", round(model.score(X_te, y_te), 3))
# deg15: train ≈ 0.999, val ≈ 0.5  → the classic overfitting signature

Regularization: shrink to generalize

• Ridge shrinks weights toward zero but keeps them all — default choice when many features contribute a little.
• Lasso drives some weights to exactly zero → automatic feature selection. Use it when you suspect only a few features matter.
• ElasticNet — both, for grouped/correlated features.
• Intuition: regularization is a bet that a simpler explanation that fits the training data slightly worse will fit new data better.

from sklearn.linear_model import Ridge, Lasso

X2 = np.hstack([X_tr, rng.normal(size=(len(X_tr), 20))])   # 20 noise features
for name, model in [("none", LinearRegression()),
                    ("ridge", Ridge(alpha=10.0)),
                    ("lasso", Lasso(alpha=0.1, max_iter=10000))]:
    model.fit(X2, y_tr)
    kept = np.count_nonzero(model.coef_) if name == "lasso" else len(model.coef_)
    print(f"{name:6s} R²={model.score(X2, y_tr):.3f}  non-zero coefs={kept}")
# lasso throws the noise features away — that's feature selection for free

The name lies: it's a classifier. It's also the workhorse of production ML — a model that outputs calibrated probabilities, trains fast, and explains itself. This module is where classification metrics finally click.

From score to probability

Linear regression can output 14.7 or −3 — useless as a probability. Logistic regression takes the linear score and squashes it into (0, 1) with the sigmoid:

Training maximizes likelihood — "make the probability of the observed labels as large as possible". That's equivalent to minimizing the log loss (cross-entropy):

import numpy as np, pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (confusion_matrix, classification_report,
                             precision_recall_curve, roc_auc_score, roc_curve)

X, y = make_classification(n_samples=4000, n_features=12, n_informative=6,
                           weights=[0.9, 0.1], random_state=42)  # 10% positives
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)

clf = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
proba = clf.predict_proba(X_te)[:, 1]        # probabilities, not classes
pred  = clf.predict(X_te)                    # threshold 0.5 applied for you

print(confusion_matrix(y_te, pred))
#                predicted
#              neg      pos
# actual neg  [TN       FP]
# actual pos  [FN       TP]

print(classification_report(y_te, pred, digits=3))
print("ROC-AUC:", round(roc_auc_score(y_te, proba), 3))   # ranking quality

The confusion matrix is the source of truth

ROC vs PR: which curve to trust

• ROC-AUC plots TPR vs FPR at every threshold. It measures ranking: the probability a random positive ranks above a random negative. Optimistic on heavy imbalance because FPR is diluted by the huge number of negatives.
• PR-AUC plots precision vs recall. Focuses on the positive class — the honest choice when positives are rare (fraud, disease, churn). A random model scores ≈ the positive rate here, so always compare against that.
• Both are threshold-free. The threshold is a product decision you make afterwards.

# --- choosing a threshold from real costs, not vibes ---
cost_fp, cost_fn = 10, 100          # false alarm costs $10, a miss costs $100
thresholds = np.arange(0.05, 0.95, 0.01)
best = min(thresholds, key=lambda t:
           cost_fp * ((proba >= t) & (y_te == 0)).sum() +
           cost_fn * ((proba <  t) & (y_te == 1)).sum())
print("cost-optimal threshold:", round(best, 2))

for t in [0.1, 0.3, 0.5, 0.7]:
    p = (proba >= t).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_te, p).ravel()
    print(f"t={t:.1f}  precision={tp/(tp+fp):.2f}  recall={tp/(tp+fn):.2f}")

# --- is the model overconfident? (needed if you act on the numbers) ---
from sklearn.calibration import calibration_curve
frac, avg = calibration_curve(y_te, proba, n_bins=10)
print(np.round(avg, 2))   # if 0.8-bin samples are right 55% of the time → miscalibrated

Multi-class

• Softmax regression (multinomial): all classes compete, probabilities sum to 1 — the natural model for "which of these 5 topics is this ticket?"
• One-vs-rest: train one binary model per class, take the highest score. This is also how many non-linear models (SVM) handle multi-class.
• Multi-label ≠ multi-class: an article can be both sports and opinion → treat as independent binary problems per label.

If your data is a table, gradient boosting is usually the strongest model you can train — and it's what wins most Kaggle competitions and runs inside a lot of bank/retail/insurance systems. Learn trees properly and you understand ensembles, bias-variance, and feature importance in one stroke.

How one tree decides

• Greedy: each split optimizes locally; no backtracking. Hence "decision tree", not "optimal tree".
• Depth is the main dial: depth 1 (a stump) underfits; unlimited depth memorizes (every leaf has 1 row → zero training error, garbage generalization).
• Strengths: no scaling needed, handles mixed types and missing values, captures non-linearity and interactions for free, gives interpretable rules.
• Weaknesses: high variance (small data change → very different tree), axis-aligned splits struggle with diagonal boundaries, extrapolation is impossible (a tree can never predict outside the range of its training targets — fatal for price curves).

Bagging → Random Forest

Bagging = aggregate bags of models: train many trees on bootstrap samples (sample with replacement, ~63% unique rows each), then average their predictions. Averaging uncorrelated errors shrinks variance — like asking 100 opinions instead of one. Random forest adds one twist: at each split, only consider a random subset of features. That decorrelates the trees, making the average even better.

Boosting → XGBoost / LightGBM / CatBoost

Instead of averaging independent models, boosting builds them sequentially: each new tree fits the residuals (or the gradient of the loss) left by the previous ones — a small, corrective step. Many weak learners × small learning rate = one strong, low-bias model.

1. Predict the baseline (mean / prior).
2. Compute errors (residuals or loss gradients).
3. Fit a small tree (depth 2–6) to those errors.
4. Add it: pred += lr × tree(x). Repeat 100–3000 times.
5. Stop early when validation loss stops improving (early stopping — the key defense against overfitting).

import numpy as np, pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.inspection import permutation_importance

X, y = make_classification(n_samples=6000, n_features=15, n_informative=6,
                           random_state=42)
X = pd.DataFrame(X, columns=[f"f{i}" for i in range(X.shape[1])])
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          stratify=y, random_state=42)

# --- random forest ---
rf = RandomForestClassifier(n_estimators=400, min_samples_leaf=3,
                            n_jobs=-1, random_state=42).fit(X_tr, y_tr)
print("RF   ROC-AUC:", round(roc_auc_score(y_te, rf.predict_proba(X_te)[:,1]), 3))

# --- gradient boosting with early stopping ---
from sklearn.ensemble import HistGradientBoostingClassifier
gb = HistGradientBoostingClassifier(max_iter=1000, learning_rate=0.08,
                                    early_stopping=True,
                                    validation_fraction=0.15,
                                    random_state=42).fit(X_tr, y_tr)
print("GBDT ROC-AUC:", round(roc_auc_score(y_te, gb.predict_proba(X_te)[:,1]), 3))

# --- importance, two ways ---
perm = permutation_importance(gb, X_te, y_te, n_repeats=10, random_state=42)
imp = pd.Series(perm.importances_mean, index=X.columns).sort_values(ascending=False)
print(imp.head(8))

Three "classical" models worth keeping in your pocket: KNN (learning by analogy), SVM (learning the widest street between classes) and Naive Bayes (learning probabilities, honestly about its own lies). Each has a niche where it still beats everything else.

K-nearest neighbours — no training at all

• k small → wiggly boundary, high variance (memorizes noise). k large → smooth, high bias (averages away real structure). k=1 gives 0 training error and often poor test error.
• Scaling is not optional: with one feature in cm (0–180) and one in income (0–120000), distance is decided entirely by income. Standardize first — always.
• Curse of dimensionality: as features grow, all points become roughly equidistant, "nearest" stops meaning anything. Works great up to ~20 meaningful features; collapses at 1000.
• Cost: zero training time, expensive prediction (must scan the dataset — use KD-trees / approximate nearest neighbours for scale).
• Good niches: recommender baselines, duplicate detection, "similar cases" retrieval, low-data problems with a sensible distance metric.

Support vector machines — the widest street

SVM finds the decision boundary that maximizes the margin — the distance to the closest points on each side (support vectors, the only points that matter). Wider margin → more tolerant of new data. Points inside the margin or misclassified are penalized by C.

The kernel trick: instead of transforming data into infinite dimensions by hand, the kernel computes the similarity in that space directly. Intuition: project to a higher space where the problem becomes linearly separable, without ever visiting it. RBF is the "bend the space to fit" default.

SVMs are still excellent for medium-sized, high-dimensional problems (text classification, gene data). On large tabular datasets they're beaten by boosting and scale poorly (training cost grows faster than linearly with n).

Naive Bayes — fast, honest about its assumption

• It multiplies probabilities — so it uses log-space to avoid underflow (never multiply 10,000 small numbers).
• Gaussian NB: continuous features, assumes each is Gaussian per class. Multinomial/Bernoulli NB: word counts / binary presence — the classic spam filter.
• Why it works anyway: for ranking (which class is most likely) the exact probabilities don't need to be right — only the ordering.
• Strengths: trains in seconds, needs very little data, great text baseline, handles thousands of features.
• Add Laplace smoothing (+1 counts) so an unseen word doesn't zero out the whole product.

The classical model map

from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import cross_val_score

docs = ["free prize click now", "meeting tomorrow agenda",
        "win a lottery jackpot", "quarterly report attached",
        "claim your free money", "lunch at noon?"]
labels = [1, 0, 1, 0, 1, 0]          # 1 = spam

# text → tf-idf vectors → model. The classic two-step.
X = TfidfVectorizer().fit_transform(docs)
print(cross_val_score(MultinomialNB(), X, labels, cv=3).mean())

# KNN on tabular data: scaling is MANDATORY
knn = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))
print(cross_val_score(knn, X.toarray(), labels, cv=3).mean())

No labels, no answer key. Unsupervised methods find structure you can't see directly — and their real production value is often indirect: better features, segmentation for the product team, or a sanity check on your data.

k-means — the simplest clustering

1. Pick k initial centroids (randomly — or use k-means++ for smarter starts).
2. Assign every point to its nearest centroid.
3. Recompute each centroid as the mean of its members.
4. Repeat 2–3 until assignments stop changing.
5. Report: within-cluster sum of squares (inertia) — how tight the groups are.

PCA — shrink dimensions, keep variance

Finds the directions (principal components) along which the data varies most, and projects onto the top few. Component 1 = the axis carrying the most information. It's linear, unsupervised, and deterministic — no iterations.

• Uses: 2D/3D visualization of high-d data, removing correlated features, speeding training, denoising.
• Rule 1: scale first. PCA maximizes variance, so an unscaled feature in dollars will own every component.
• Rule 2: components are mixtures. PC1 = 0.4·age + 0.7·income − 0.5·debt — they're hard to interpret but easy to plot.
• Choose k by cumulative explained variance (e.g. keep 95%).
• Not for non-linear structure (UMAP/t-SNE exist for visualization of that — but t-SNE distances between clusters are not meaningful, don't over-read it).

Others worth knowing

import numpy as np, pandas as pd
from sklearn.datasets import make_blobs
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.metrics import silhouette_score

X, y_true = make_blobs(n_samples=900, centers=4, cluster_std=1.1,
                       random_state=42)
Xs = StandardScaler().fit_transform(X)          # scale BEFORE clustering

# choosing k: inertia elbow + silhouette
for k in range(2, 8):
    km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(Xs)
    print(f"k={k}  inertia={km.inertia_:8.1f}  silhouette="
          f"{silhouette_score(Xs, km.labels_):.3f}")

km = KMeans(n_clusters=4, n_init=10, random_state=42).fit(Xs)

# visualize in 2D and profile the segments
coords = PCA(n_components=2).fit_transform(Xs)
prof = pd.DataFrame(Xs, columns=["f0", "f1"]).assign(cluster=km.labels_)
print(prof.groupby("cluster").mean().round(2))   # "cluster 2 = high f0, low f1"

Deep learning is not magic — it's linear regression stacked with non-linearities, trained with the chain rule at scale. This module builds that picture from a single neuron up to transformers, and is honest about when you need it (rarely, for tabular data).

One neuron

• It's the linear model from Module 05 plus a squashing function.
• Why activations are non-negotiable: stack 50 linear layers without them and the whole thing collapses into… one linear layer. All that depth would compute W·x + b. Non-linearity is what lets networks bend to fit reality.
• ReLU is the default (cheap, no saturation for positive values). ReLU dying (always 0) is a classic failure — symptom: a layer's outputs are all zero.

From neuron to network

A "hidden layer" learns representations: layer 1 finds edges/word patterns, layer 2 combines them into shapes/phrases, layer 3 into faces/topics. That automatic feature learning is the entire reason deep learning wins on unstructured data — classical models make you hand-design those features.

Training: backprop in one paragraph

1. Forward pass: run x through all layers, record intermediate values, get loss.
2. Backward pass: chain rule from loss backwards → gradient of loss w.r.t. every weight (∂L/∂w for millions of weights in one sweep).
3. Update: every weight steps a little downhill: w -= lr × grad.
4. Repeat over mini-batches for many epochs (one epoch = one full pass over the dataset).

Overfitting: the full defense kit

• More data / data augmentation (rotate, crop, back-translate — the highest-leverage fix for vision/NLP).
• Dropout: randomly zero a fraction of activations during training — forces redundant representations. Off at inference.
• Weight decay (L2): penalize large weights — exactly Module 05's idea.
• Early stopping: monitor validation loss, stop when it stops improving, restore the best weights. Always do this.
• Smaller model: fewer layers/width. Capacity is a dial, not a virtue.
• Batch normalization / layer normalization: stabilizes internal scales → faster, more stable training.

The architecture map

# pip install torch
import torch, torch.nn as nn
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X, y = make_moons(n_samples=2000, noise=0.25, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler().fit(X_tr)                 # scale for NNs!
X_tr = torch.tensor(scaler.transform(X_tr), dtype=torch.float32)
X_te = torch.tensor(scaler.transform(X_te), dtype=torch.float32)
y_tr = torch.tensor(y_tr, dtype=torch.float32).unsqueeze(1)
y_te = torch.tensor(y_te, dtype=torch.float32).unsqueeze(1)

model = nn.Sequential(
    nn.Linear(2, 64), nn.ReLU(), nn.Dropout(0.2),   # hidden layer
    nn.Linear(64, 64), nn.ReLU(), nn.Dropout(0.2),  # hidden layer
    nn.Linear(64, 1),                               # logit output
)
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)
loss_fn = nn.BCEWithLogitsLoss()                    # numerically stable sigmoid+log loss

for epoch in range(300):
    model.train()
    logit = model(X_tr)                             # forward
    loss = loss_fn(logit, y_tr)                     # loss
    opt.zero_grad(); loss.backward(); opt.step()    # backward + update

    if epoch % 50 == 0:
        model.eval()
        with torch.no_grad():                       # forward WITHOUT dropout
            acc = ((torch.sigmoid(model(X_te)) > 0.5).float() == y_te).float().mean()
        print(f"epoch {epoch:3d}  loss={loss.item():.4f}  val_acc={acc.item():.3f}")

# In Keras this is ~10 lines: model.compile(optimizer="adam",
# loss="binary_crossentropy", metrics=["accuracy"]) + model.fit(X, y, epochs=30)

Models don't learn from data — they learn from representations of data. Better features beat better algorithms almost every time, and doing preprocessing safely (no leakage) is the difference between a credible score and a embarrassing one.

The feature checklist

Pipelines: the anti-leakage machine

A Pipeline chains transformers and a final estimator so that fit runs training-only statistics through every step, and predict applies the same transformations. It also means the exact same code runs at serving time — no "why is the API preprocessing differently?" bugs.

import numpy as np, pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score

df = pd.read_csv("churn.csv")
y = (df.pop("churn") == "yes").astype(int)
num = ["tenure", "monthly", "total", "support_calls"]
cat = ["plan", "region", "device"]

# each column group gets its own treatment; nothing sees the validation data
prep = ColumnTransformer([
    ("num", Pipeline([("imp", SimpleImputer(strategy="median")),
                      ("sc",  StandardScaler())]), num),
    ("cat", Pipeline([("imp", SimpleImputer(strategy="most_frequent")),
                      ("ohe", OneHotEncoder(handle_unknown="ignore"))]), cat),
])

model = Pipeline([
    ("prep", prep),
    ("sel",  SelectKBest(f_classif, k=12)),
    ("clf",  HistGradientBoostingClassifier(max_iter=300, random_state=42)),
])

X_tr, X_te, y_tr, y_te = train_test_split(df, y, test_size=0.2,
                                          stratify=y, random_state=42)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X_tr, y_tr, cv=cv, scoring="roc_auc")
print(f"CV ROC-AUC: {scores.mean():.3f} ± {scores.std():.3f}")

model.fit(X_tr, y_tr)                                  # refit on all training data
print("test ROC-AUC:", round(roc_auc_score(y_te, model.predict_proba(X_te)[:,1]), 3))

Imbalanced data — three independent levers

1. The algorithm: class_weight="balanced" (or scale_pos_weight in XGBoost) — penalizes mistakes on the rare class more. Free, always try first.
2. The data: undersample the majority, oversample the minority, or synthesize (SMOTE: interpolate between minority neighbours). Use inside CV folds only — synthetic points must never be generated before the split.
3. The threshold: train normally, then lower the decision threshold until recall meets your cost target (Module 06). Often the best lever of the three.

Feature selection

• Filter (cheap, model-independent): correlation, mutual information, variance threshold, chi-square. Good first pass.
• Wrapper (model-in-the-loop): recursive feature elimination — accurate but slow.
• Embedded (free with the model): lasso's zeroed coefficients, tree/boosting importance, SHAP. Best bang for the buck.
• Stability check: does the selected set survive resampling? If a feature is selected in 3 of 5 folds, its signal is shaky.

The best ML engineers don't have better models — they have better loops: a trustworthy validation scheme, disciplined search, and a habit of reading errors instead of numbers.

Cross-validation schemes

Searching hyperparameters efficiently

• Search logarithmically for scales: learning rates, regularization strength (log-uniform 1e-5 … 1e-1) — linear grids waste budget.
• Tune the influential knobs first: model capacity (depth, layers), learning rate, regularization. Ignore n_jobs-style knobs.
• Fix a budget of folds × configs, report it, and stop. Unlimited tuning is overfitting with extra steps.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import (cross_val_score, learning_curve,
                                     RandomizedSearchCV, StratifiedKFold)
from sklearn.ensemble import HistGradientBoostingClassifier
from scipy.stats import loguniform, randint

X, y = make_classification(n_samples=2500, n_features=20, n_informative=10,
                           weights=[0.85, 0.15], random_state=42)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# --- 1. random search over the knobs that matter ---
space = {"learning_rate": loguniform(1e-3, 0.3),
         "max_depth": randint(2, 10),
         "min_samples_leaf": randint(5, 100),
         "l2_regularization": loguniform(1e-4, 10)}
search = RandomizedSearchCV(HistGradientBoostingClassifier(max_iter=300,
                             random_state=42),
                            space, n_iter=40, scoring="roc_auc",
                            cv=cv, random_state=42, n_jobs=-1).fit(X, y)
print("best params:", search.best_params_)
print("best CV auc:", round(search.best_score_, 3))

# --- 2. learning curves: should you get more data? ---
sizes, tr, val = learning_curve(search.best_estimator_, X, y, cv=cv,
                                train_sizes=np.linspace(0.2, 1.0, 5),
                                scoring="roc_auc", n_jobs=-1)
print("train:", np.round(tr.mean(axis=1), 3))
print("valid:", np.round(val.mean(axis=1), 3))
# both rising & close → MORE DATA will help
# both flat & low  → model too simple / features bad → more capacity or better features
# train high, val low → overfit → regularize / simplify / more data

Error analysis: the step everyone skips

1. Get probabilities, not just labels for the whole validation set.
2. Sort into four buckets: high-confidence correct, high-confidence wrong (the dangerous ones — investigate first), low-confidence correct, low-confidence wrong.
3. Slice by everything you have: segment, language, device, time of day, price band, source system. Compute per-slice precision/recall and find where it collapses.
4. Label a sample of errors by hand. Is the model wrong, or is the label wrong? Noisy labels cap your ceiling more often than model capacity does.
5. Turn each finding into one action: a new feature, a data source fix, a per-slice threshold, a rule for an obviously-wrong case, or "collect 500 more examples of slice X".
6. Re-measure. One action at a time so you know what worked.

# a compact error-analysis table
import pandas as pd
res = pd.DataFrame({"y": y_val, "p": proba, "segment": df_val.segment,
                    "device": df_val.device})
res["pred"] = (res.p >= 0.5).astype(int)
res["correct"] = res.pred == res.y

print(res.groupby("segment").agg(n=("correct","size"),
                                 acc=("correct","mean"),
                                 avg_p=("p","mean")).sort_values("acc"))

worst = res[(res.correct == False) & ((res.p > 0.9) | (res.p < 0.1))]
print("confident-but-wrong:", len(worst))     # these are the ones to read
print(worst.sort_values("p", ascending=False).head(20))

You can't explain a score you don't understand, and you can't defend a decision you can't audit. Interpretability is a requirement, not a luxury — and fairness is interpretability applied to people.

The interpretability toolkit

# pip install shap
from sklearn.inspection import permutation_importance, PartialDependenceDisplay
import shap

# 1. global: what matters?
pi = permutation_importance(model, X_te, y_te, n_repeats=15, random_state=42)
top = sorted(zip(X_te.columns, pi.importances_mean), key=lambda t: -t[1])[:8]

# 2. how does it behave?
PartialDependenceDisplay.from_estimator(model, X_te, ["tenure"])
# upward slope → more tenure, less churn. Flat → irrelevant. Jagged → suspicious.

# 3. why this row?
explainer = shap.TreeExplainer(fitted_gbdt)
sv = explainer(X_te.head(5))
shap.waterfall_plot(sv[0])        # "tenure=2 added +0.14 to the score"

Fairness: where bias comes from

1. Historical bias: the data reflects past discrimination. A hiring model trained on 10 years of promotions learns who got promoted, not who was good.
2. Measurement bias: the feature or label is measured differently across groups (better fraud labels for one region; sentiment data noisier for some dialects).
3. Sampling/aggregation bias: some groups are under-represented → worse performance for them.
4. Evaluation bias: you only measured an aggregate metric — the failure was hidden in the average. And label choice: using arrests instead of crime, repayment instead of ability to repay.

They can't all hold at once when base rates differ between groups (Kleinberg/Ruszak/Sharma-type impossibility results). Choosing which definition to violate is a values decision that belongs to stakeholders and legal — not something you settle by picking a metric in a notebook. Your job: compute them all, show the trade-offs, and never silently pick one.

A model is one component of a product. The moment it ships, your job changes: from "is the score high?" to "does the system deliver value, safely, repeatedly, at acceptable cost?"

The system around the model

Every box can fail independently of model quality. In practice, most production incidents are in the data and serving boxes: an upstream schema changed, a feature job stopped, a default value kicked in, a cache served stale scores. The model itself rarely breaks.

Money math: the cost model

Worked example — churn intervention:

• 100,000 customers, 5% churn. Offer costs $10 each, only given to flagged customers.
• Model at threshold t: precision 0.40, recall 0.60 → flags 750, saves 3,000 × 0.60 = 1,800 customers, each worth $200 → benefit $360,000; cost of offers 750 × $10 = $7,500.
• But: also add offers to people who wouldn't have churned (cannibalization — some accept the discount and were never leaving), model/serving cost, and the cost of annoying customers. Then sweep the threshold and find where net peaks — not where F1 peaks.

Risk tiers decide process

Ship a boring MVP first

1. Heuristic v0: the rule a smart person would write today. Ship it, measure it. This is your baseline and it takes a day.
2. Model v1 in shadow: run the model in production, log predictions, nobody sees them. Compare against outcomes and the heuristic for weeks. Free risk.
3. Model v2 assisted: show scores to humans alongside their own judgement; measure whether they get faster/better.
4. Model v3 automated for the low-risk slice only, with monitoring and rollback.
5. Expand coverage as evidence accumulates.

The four documents

Notebooks are for thinking; pipelines are for running. MLOps means making every step — data, training, evaluation, packaging — versioned, reproducible, automated and testable. This is the module that turns a demo into something you can re-run in six months.

Reproducibility: five things to version

# a training run that is reproducible and self-describing
import json, hashlib, git, mlflow

repo = git.Repo(search_parent_directories=True)
sha  = repo.head.object.hexsha[:12]
data_hash = hashlib.sha256(open("train.csv","rb").read()).hexdigest()[:16]

mlflow.set_experiment("churn-v2")
with mlflow.start_run(run_name="ridge-α10"):
    mlflow.log_params({"model": "ridge", "alpha": 10.0, "seed": 42})
    mlflow.log_metrics({"cv_auc": 0.871, "cv_std": 0.012,
                        "test_auc": 0.864})
    mlflow.set_tags({"git_sha": sha, "data_hash": data_hash,
                     "sklearn": __import__("sklearn").__version__})
    mlflow.sklearn.log_model(model, "model")
    mlflow.log_artifact("error_analysis.md")
print("run:", mlflow.last_active_run.info.run_id)

The pipeline as code

# train.py — callable from cron, Airflow, Dagster, GitHub Actions, or bash
"""Usage: python train.py --config configs/churn.yaml"""
import argparse, pathlib, yaml, joblib
from datetime import datetime, timezone

def run(cfg: dict) -> dict:
    # 1. ingest — immutable, dated, validated
    df = load_snapshot(cfg["data"]["table"], cfg["data"]["as_of"])
    validate_schema(df, cfg["data"]["schema"])       # fail loudly, not silently

    # 2. features (same code path as serving — see Module 16)
    X, y = build_features(df, cfg["features"])

    # 3. train + CV + holdout evaluation against a fixed metric
    scores = cross_validate(X, y, cfg["model"], cfg["cv"])
    gate = scores.mean - scores.std
    if gate < cfg["thresholds"]["min_auc"]:           # quality gate
        raise SystemExit(f"model below threshold: {gate:.3f}")

    # 4. package with metadata
    artifact = {
        "model": joblib.dump(model, "model.joblib")[0],
        "created": datetime.now(timezone.utc).isoformat(),
        "metrics": scores, "config": cfg,
        "git_sha": get_sha(), "data_hash": data_hash,
    }
    register(artifact)            # → registry → staging (never straight to prod)
    return artifact

if __name__ == "__main__":
    p = argparse.ArgumentParser(); p.add_argument("--config", required=True)
    run(yaml.safe_load(open(p.parse_args().config)))

Scheduling: when does training run?

Feature stores: the consistency problem

Training reads features from the warehouse ("last 30-day spend, computed over history"). Serving needs the same feature in 10ms from a low-latency store. If the two implementations differ — timezone handling, default values, a window defined as "last 30 days" differently — you get training/serving skew, the #1 silent killer: the model performs great in validation and mysteriously badly in production.

• Fix: one definition per feature, used by both paths (a shared library, or a feature store like Feast/Tecton/Redis+registry).
• Online store (low latency, latest values) + offline store (full history for training) + a registry with definitions and versions.
• Point-in-time correctness: training must use the feature value as it was at that moment — never join on "current" values or you leak the future.

CI/CD for ML

How a prediction gets to the person who needs it. Choose this before choosing infrastructure — latency, cost and failure modes follow from the pattern, not the other way around.

Four patterns

Packaging a model

• Load once at startup, keep in memory — never load from disk per request.
• Warm up with a sample prediction to avoid a slow first request.
• Big models: serve with a dedicated runtime (Triton, TorchServe, BentoML) or a GPU; A/B the memory/latency trade-off.

Release safely

• Shadow catches data/serving bugs with zero user risk (Module 14).
• Canary: route a small slice to the new version, watch dashboards, expand only if healthy. Requires: automatic rollback, not "someone notices on Monday".
• A/B test: measures business impact on real users. Needs enough sample for significance, a pre-declared metric, and no peeking-until-significant.
• Version your API (/v1/predict) and keep the previous model loadable — rollback = flip a pointer, not a redeploy.

Latency & cost budget

A request's time = queue + feature lookup + preprocessing + inference + postprocessing + network.

• Measure p50 / p95 / p99, not the average. Users feel the tail.
• Typical ballparks: feature store lookup 1–5ms; small tree/linear model <1ms on CPU; small NN 5–20ms CPU; LLM call 300–2000ms.
• Ways to cut cost: batch multiple requests per forward pass; cache predictions (with a TTL matched to how often inputs actually change); cheaper model for easy cases (cascading: fast filter → heavy model only for the ambiguous 10%); quantize (FP32 → INT8, often 2–4× faster, <1% accuracy loss); distill into a smaller net; prune; downgrade GPU class.
• Don't pay for the GPU twice: batching + autoscaling + spot instances for training; CPU for anything small at inference.

Theory ends here. By the end of this module you will have a trained pipeline served behind an HTTP API, containerized, tested, and reachable with curl. That's the deliverable — let's build it.

Project layout

1 · Train and save the whole pipeline

# train.py — save preprocessing AND model as one artifact (Module 16)
import joblib, pandas as pd
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.ensemble import HistGradientBoostingClassifier

FEATURES = ["tenure", "monthly_charges", "total_charges", "support_calls",
            "contract", "payment_method"]
NUM = ["tenure", "monthly_charges", "total_charges", "support_calls"]
CAT = ["contract", "payment_method"]

prep = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                      ("scale", StandardScaler())]), NUM),
    ("cat", Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                      ("ohe", OneHotEncoder(handle_unknown="ignore"))]), CAT),
])
pipe = Pipeline([("prep", prep),
                 ("clf", HistGradientBoostingClassifier(max_iter=300,
                     random_state=42))])

def train(df: pd.DataFrame, y) -> Pipeline:
    pipe.fit(df[FEATURES], y)          # fit learns; the fitted object is the artifact
    return pipe

if __name__ == "__main__":
    df = pd.read_csv("churn.csv")
    y = (df.pop("churn") == "yes").astype(int)
    train(df, y)
    joblib.dump({"pipeline": pipe, "features": FEATURES, "threshold": 0.35},
                "model.joblib")
    print("saved model.joblib")

2 · The API

# app.py
import time, joblib, logging
import pandas as pd
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field

log = logging.getLogger("uvicorn.info")

# ---- load ONCE at startup; never inside a request handler ----
bundle = joblib.load("model.joblib")
MODEL, FEATURES, THRESHOLD = bundle["pipeline"], bundle["features"], bundle["threshold"]
MODEL.predict(pd.DataFrame([{f: 0 for f in FEATURES}]))   # warm-up call

class Customer(BaseModel):
    tenure: int = Field(..., ge=0, description="months as customer")
    monthly_charges: float = Field(..., ge=0)
    total_charges: float = Field(..., ge=0)
    support_calls: int = Field(..., ge=0, le=50)
    contract: str = Field(..., examples=["month-to-month", "one year"])
    payment_method: str = Field(..., examples=["electronic check"])

class Prediction(BaseModel):
    churn_probability: float
    will_churn: bool
    threshold: float
    model_version: str

app = FastAPI(title="Churn Scorer", version="1.0.0",
              description="Scores a customer's churn risk.")
MODEL_VERSION = "churn-2026-10-01"

@app.get("/health")
def health() -> dict:
    """Liveness: is the service up AND is the model loadable?"""
    return {"status": "ok", "model": MODEL_VERSION, "features": len(FEATURES)}

@app.post("/predict", response_model=Prediction)
def predict(c: Customer) -> Prediction:
    t0 = time.perf_counter()
    row = pd.DataFrame([c.model_dump()])[FEATURES]       # enforce feature order!
    try:
        proba = float(MODEL.predict_proba(row)[0, 1])
    except Exception as e:
        log.error("inference failed: %s", e)
        raise HTTPException(status_code=500, detail="scoring failed")
    log.info("scored in %.1f ms", (time.perf_counter() - t0) * 1000)
    return Prediction(churn_probability=round(proba, 4),
                      will_churn=proba >= THRESHOLD,
                      threshold=THRESHOLD,
                      model_version=MODEL_VERSION)

@app.post("/predict/batch", response_model=list[Prediction])
def predict_batch(rows: list[Customer]) -> list[Prediction]:
    """Batch endpoint: amortizes overhead — use it wherever you can."""
    frame = pd.DataFrame([r.model_dump() for r in rows])[FEATURES]
    probs = MODEL.predict_proba(frame)[:, 1]
    return [Prediction(churn_probability=round(float(p), 4),
                       will_churn=p >= THRESHOLD, threshold=THRESHOLD,
                       model_version=MODEL_VERSION) for p in probs]

3 · Contract tests

# tests/test_api.py
import pytest
from fastapi.testclient import TestClient
from app import app

client = TestClient(app)
GOOD = {"tenure": 12, "monthly_charges": 70.5, "total_charges": 846.0,
        "support_calls": 2, "contract": "month-to-month",
        "payment_method": "electronic check"}

def test_health():
    assert client.get("/health").json()["status"] == "ok"

def test_predict_shape_and_bounds():
    r = client.post("/predict", json=GOOD)
    assert r.status_code == 200
    body = r.json()
    assert 0.0 <= body["churn_probability"] <= 1.0
    assert isinstance(body["will_churn"], bool)
    assert body["model_version"]

def test_validation_rejects_bad_input():
    bad = dict(GOOD, tenure=-3)
    assert client.post("/predict", json=bad).status_code == 422   # pydantic catches it

def test_missing_field_is_422():
    bad = dict(GOOD); bad.pop("contract")
    assert client.post("/predict", json=bad).status_code == 422

def test_batch_matches_single():
    a = client.post("/predict", json=GOOD).json()["churn_probability"]
    b = client.post("/predict/batch", json=[GOOD]).json()[0]["churn_probability"]
    assert a == b

4 · Containerize

# requirements.txt  →  fastapi uvicorn[standard] scikit-learn pandas numpy joblib

# Dockerfile
FROM python:3.12-slim
WORKDIR /srv
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py schemas.py model.joblib ./
EXPOSE 8000
# single worker is fine to start; scale with replicas, not workers, first
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

# --- build & run ---
docker build -t churn-scorer:1.0 .
docker run --rm -p 8000:8000 churn-scorer:1.0

# --- poke it ---
curl -s localhost:8000/health
curl -s -X POST localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"tenure":12,"monthly_charges":70.5,"total_charges":846.0,
       "support_calls":2,"contract":"month-to-month",
       "payment_method":"electronic check"}'
# → {"churn_probability":0.7312,"will_churn":true,"threshold":0.35,...}

# docs are free with FastAPI:
open http://localhost:8000/docs

Deploying is the midpoint, not the finish line. Models decay — quietly. Monitoring is how you find out before your customers do, and this module is what separates "we deployed ML" from "we run ML".

The three decays

Data drift often precedes concept drift — that's why you monitor inputs even when you don't have labels yet. And beware sudden vs gradual drift (step change from a release vs slow decay) and recurring drift (weekends, seasons) — alerts must know the difference or you'll drown in false alarms.

What to monitor: five layers

1. Service health (immediately): request rate, error rate, p50/p95/p99 latency, uptime, saturation. These page you on day one.
2. Data quality: null %, missing features, unexpected categories, out-of-range values, row counts, schema changes, default-value spikes.
3. Input distributions: per-feature drift vs a training reference window (PSI / KS-test / KL). Alert on a set of features crossing a threshold, or a sustained shift — not one noisy feature.
4. Model outputs: score distribution, predicted class rate, score quantiles, embedding norms. A shifting score distribution is your earliest "something's wrong" signal even without labels.
5. Performance + business (delayed): per-slice metrics when labels arrive (hours to months later), and the business KPI the model exists to move.

# a minimal drift report: PSI on every feature vs the training reference
import numpy as np, pandas as pd

def psi(expected: np.ndarray, actual: np.ndarray, bins: int = 10) -> float:
    """Population Stability Index. <0.1 stable · 0.1–0.25 investigate · >0.25 retrain."""
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual,   edges)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)   # avoid log(0)
    return float(np.sum((a - e) * np.log(a / e)))

def drift_report(ref: pd.DataFrame, live: pd.DataFrame) -> pd.DataFrame:
    rows = []
    for col in ref.columns:
        if pd.api.types.is_numeric_dtype(ref[col]):
            s = psi(ref[col].dropna().values, live[col].dropna().values)
        else:   # categorical: PSI over category frequencies
            cats = set(ref[col].dropna()) | set(live[col].dropna())
            s = psi(ref[col].value_counts(normalize=True).reindex(cats).values,
                    live[col].value_counts(normalize=True).reindex(cats).values)
        rows.append({"feature": col, "psi": round(s, 4),
                     "status": "RETRAIN" if s > 0.25
                               else "investigate" if s > 0.1 else "ok"})
    out = pd.DataFrame(rows).sort_values("psi", ascending=False)
    return out[out.status != "ok"]

# reference = training features (immutable snapshot)
# live      = last 24h of logged inputs (from your API logs)
print(drift_report(reference, live_inputs))

Alerting without crying wolf

• Baseline vs absolute: alert on deviation from a seasonal baseline (same hour last week), not on a fixed number that breaks every holiday.
• Two-stage: warn on one window, page only if it persists across two consecutive windows. Single-sample alerts train people to ignore alerts.
• Route by owner. Every alert links to a runbook; every runbook ends with a human decision. An alert with no action is noise — delete it.
• Track alert quality itself: % true positives. If it's low, the thresholds are wrong.

Retraining: triggers and strategy

• Always beat the incumbent or keep the incumbent. Automated retraining with a quality gate — never "newest model wins".
• Window choice: train on all history (stable) vs recent only (adapts fast, can forget). Recurring seasonality needs at least a full cycle of data.
• Champion/challenger: new model shadows or takes a traffic slice; the incumbent keeps serving until evidence accumulates.
• Auto-retrain ≠ auto-deploy. Automating training is safe; automation of promotion should depend on risk tier (Module 14).

Incident runbook (the one-page version)

1. Confirm: is it the model or the service? Check error rate and latency first — most "model" incidents are infrastructure.
2. Rollback if users are affected: point traffic at the previous model version. Diagnosis happens after the bleeding stops.
3. Check the data path: upstream job ran? schema changed? null rate spiked? feature values sane? (Fixes 80% of cases.)
4. Compare distributions: today's inputs vs training reference (the PSI report) and vs yesterday.
5. Slice: is the failure everywhere or in one segment/region/time window?
6. Fix, verify, postmortem — one page: timeline, root cause, blast radius, action items, and which monitor should have caught it (add it).

Reading about driving isn't driving. This ladder takes you from "one good notebook" to "a system I own end-to-end" — each rung adding exactly one new kind of pressure. Do them in order; don't skip rung 2 to jump at an LLM.

The four rungs

What a strong project README contains

• One-line problem + why it matters (the decision it serves)
• Data: source, size, time range, known quality issues, license/PII notes
• Baseline and the number it achieved
• Approach: split strategy, metric, models tried, the winner and the margin (with mean ± std)
• Leakage & bias audit: what you checked and what you found
• Errors: worst slice, mislabelled sample, what you'd fix with more time
• Serving: how to run it, endpoint examples, latency/cost measured
• Monitoring: what you'd watch and your retraining rule
• Reproduce: exact commands, environment, seed
• Limitations + intended use (model card, half a page)

The last module is about keeping it. A schedule you can actually follow, a deck to test yourself against, and the two checklists that tell you whether you're ready for real work.

The 4-week schedule (8–10 h/week)

Falling behind is normal. Compress by: skimming 08 and 09 (read goals + pitfalls + recall cards only), and doing Module 10 only if your direction needs deep learning. Protect the two capstones — they're where the learning actually consolidates.

Spaced repetition: the maintenance plan

Put these in a real calendar. The tool doesn't matter (Anki, a paper card box, this notebook's recall boxes) — the schedule matters. Cards you write yourself are 2× better than cards someone hands you.

Final recall deck · 24 questions

Answer out loud first. Click only after you've spoken an answer.

Pre-production checklist · run before you call it live

• Framing: decision, metric, baseline, and cost model written down
• Split: matches reality (time/group aware where needed); test set untouched until the end
• Leakage audit: every feature passes the "exists at prediction time?" test
• Pipeline: one artifact for preprocessing + model; feature-parity test passes
• Metrics: mean ± std across folds; beats baseline by more than the noise
• Slices: per-segment performance known; worst slice is acceptable or documented
• Threshold: chosen from cost, recorded with the model
• Validation: API input validation, clear errors, health check, graceful failure mode
• Tests: data schema, golden row, API contract — running in CI
• Release: shadow → canary → ramp, with automatic rollback
• Monitoring: five layers charted; alerts have owners and runbooks
• Labels: pipeline exists to capture outcomes and join them to predictions
• Retraining: trigger, quality gate and promotion rule written down
• Docs: model card, runbook, decision log, README with reproduce commands
• Ownership: a named human on call, and a max tolerable error rate

After the notebook: where to go deeper