The confusion matrix

The same model and the same 99.18% accuracy, broken out by class — and beside it the same asset modelled honestly, which is where the number the brief warns about actually appears.

Accuracy

99.18%

Recall on failures

64.3%

92 of 143 caught

Already the headline has slipped: 64.3% of failure windows caught, from a model advertised at 99.18%. 97.89% of that accuracy is the model correctly saying "normal" — the easy part, on a machine that is running normally almost all of the time.

Accuracy

93.05%

Recall on failures

0.0%

0 of 361 caught

This is the same asset with the leakage removed and the split ordered in time — the version that would be deployed. It still reports 93.05% accuracy. It catches nothing.

Put the two accuracies side by side 99.18% and 93.05%. Nine points apart, both excellent-looking, and one of them describes a model with zero recall on the only class anyone cares about. Accuracy did not distinguish them, and no amount of staring at accuracy ever would.

Why this happens, arithmetically

Failure windows are 2.66% of the data. A model that predicts "normal" for every window scores 97.34% without learning anything at all. Every model in this demonstration is competing against that floor, and accuracy measures the distance from zero rather than the distance from the floor.

A single accuracy figure cannot express production readiness on an imbalanced problem. It is not that the number is wrong; it is that it answers a question nobody needed answering. The metrics that do answer it are on the next screen.

Computed by scripts/build_models.py in 86.4s at 2026-08-09T19:08:15+00:00 · scikit-learn 1.6.1 · seed 20260809 · source: MetroPT-3 (Air Production Unit of a Metro do Porto train), CC BY 4.0, DOI 10.24432/C5VW3R. Nothing on this page is hardcoded.