Cheat sheet

Part 5 · Model Validation & Cross-Validation — Cheat Sheet

Six illustrated pages — foundations, classification, regression, probability estimation, clustering, and model validation.

← Read the full post (Part 5)Updated June 2026
Part 5 · Model Validation & Cross-Validation — Cheat Sheet — printable cheat sheet
Download PNG

Or read the searchable version below.

This six-page cheat sheet is shared with Part 4 — Evaluation Metrics. Pages 1–5 (foundations, classification, regression, probability, clustering) belong to Part 4; Page 6 — Model validation is the focus of this post and the reason the cheat sheet lives here too.

Page 1 · Foundations

Page 1 of 6 of the Evaluation Metrics cheat sheet. Eleven numbered concepts covering: (1) model evaluation — scoring model performance; (2) model validation — checking whether performance generalizes to unseen / production data; (3) metric-business-goal alignment — the metric must match the problem; (4) problem families need different metrics — classification, scoring, probability estimation, clustering; (5) classification problems — predicting labels; (6) scoring / regression problems — predicting numbers; (7) no-target problems — finding structure without labels; (8) probability estimation problems — predicting class probabilities, not only labels; (9) model parameters — learned from data; (10) hyperparameters — chosen / tuned around training, not learned directly; (11) train / validation / test sets — three splits for learn, tune, and final unbiased check.
The setup behind every evaluation decision: what you're measuring, what kind of problem you have, and the data splits you'll need.

Page 2 · Classification metrics

Page 2 of 6. Thirteen concepts covering confusion matrix, accuracy, why accuracy fails on imbalanced data, precision, recall, specificity, precision-recall trade-off, F1, MCC, accuracy vs recall vs MCC comparison, Cohen's kappa, what to do with imbalanced data, and choosing the right classification metric by application.
How to judge a classifier, especially when classes are imbalanced. The confusion matrix is the foundation; every metric is just a different ratio of its four cells.

Page 3 · Regression / scoring metrics

Page 3 of 6. Six concepts: residuals, RMSE, R² / coefficient of determination, TSS / RSS decomposition, MAE, and why absolute error is often not a good project objective.
Measuring numeric prediction error. Every regression metric is a different way of squashing the residual vector into one number.

Page 4 · Probability-estimation metrics

Page 4 of 6. Five concepts: probability outputs / confidence scores, decision thresholds, ROC curve, AUC, and double density plots.
Evaluating scores and thresholds. When the model outputs a probability instead of a label, where you put the threshold is its own decision.

Page 5 · Clustering evaluation

Page 5 of 6. Four concepts: choosing k in clustering, compactness / separation, silhouette, and trying different clustering algorithms (k-medians, k-medoids, Gaussian mixtures, density-based).
Judging structure without labels. Clustering metrics measure how compact your clusters are and how distinct they are from one another.

Page 6 · Model validation / generalization — the Part 5 page

Page 6 of 6. Thirteen concepts: (40) model problems — bias, variance, overfit, nonsignificance; (41) bias-variance trade-off — sweet spot with lowest test error; (42) training error vs test error — training error can dramatically underestimate test error; (43) train-test split is not enough — keeps the test set 'used'; (44) hold-out validation — 60% / 20% / 20%; (45) single split vs multiple splits — luck vs reliability; (46) cross-validation resampling — many splits, evaluate on each, average; (47) k-fold cross-validation — each fold becomes validation once; (48) LOOCV / repeated k-fold — exhaustive variants; (49) retraining with best parameters — after CV, retrain on full training data, deploy; (50) bootstrapping — resample with replacement for small samples and confidence intervals; (51) significance testing / p-values — compare model to a naive / null model; (52) distribution drift — training vs live distributions can change over time.
Page 6 is the focus of this post. Checking whether performance holds up on new data — model problems, bias-variance, cross-validation, bootstrap, significance, and drift.