Model Evaluation

How good is your model, really?

Training a model is only half the job. You must evaluate it rigorously — and the right metric depends entirely on the problem.

The confusion matrix

              Predicted
              Pos    Neg
Actual Pos  [ TP  |  FN ]
Actual Neg  [ FP  |  TN ]

• True Positive (TP) — correctly flagged.
• False Positive (FP) — wrongly flagged (a false alarm).
• False Negative (FN) — wrongly missed.
• True Negative (TN) — correctly cleared.

Metrics

• Accuracy = (TP + TN) / total. Misleading when classes are imbalanced.
• Precision = TP / (TP + FP). Of the positives we predicted, how many were real?
• Recall = TP / (TP + FN). Of the actual positives, how many did we catch?
• F1 score — the harmonic mean of precision and recall. Balances the two.

For a spam filter, low precision means real email gets buried — annoying. For a cancer screening, low recall means real cancers get missed — dangerous. Choose the metric that matches the cost of being wrong.

Cross-validation

A single train/test split can be unlucky. k-fold cross-validation splits the data into k chunks, trains on k−1 and tests on 1, rotating until every chunk has been the test set. Average the results for a robust estimate.

The bias-variance tradeoff

• High bias (underfitting) — model is too simple; misses the real pattern.
• High variance (overfitting) — model is too complex; fits noise.

Good modeling balances the two: complex enough to capture real patterns, simple enough to generalize.