Measurement leakage, and what happens when it is removed

The first model was not detecting an impending failure. It was detecting a failure already in progress — and the features that let it do so are the obvious, honest features any competent engineer would build first.

Why the naive feature set cannot fail

An air leak under high stress is, by definition, the compressor never unloading. So the sensors that describe "the compressor is working hard right now" are near-deterministic restatements of the label, not predictors of it.

Sensor	Mean, normal	Mean, in failure window	Lift
`TP2`	1.232	8.115	6.6×
`TP3`	8.999	8.288	0.9×
`H1`	7.720	0.038	0.0×
`DV_pressure`	0.020	1.860	94.8×
`Reservoirs`	8.999	8.290	0.9×
`Oil_temperature`	62.383	75.598	1.2×
`Motor_current`	1.980	5.532	2.8×
`COMP`	0.854	0.004	0.0×

This is not a synthetic trap. No feature was planted to make this happen. The leak is native to the obvious framing of the problem, which is exactly why it is worth twenty minutes of a training course.

The score, as each form of leakage is removed

R3 — in-window label

0.995

AUPRC. Publishable.

R4 — same features, time-ordered

0.022

AUPRC. The collapse.

R7 — strictly causal features

0.019

AUPRC, against a prevalence floor of 0.0200

Three changes separate the first bar from the last, and each is a discipline rather than a technique:

The label became a prediction. "Is it failing now" became "does a failure start within the next 24 hours". The first question has no operational value; by the time it is answered the machine has stopped.
The features became causal. Every rolling statistic is computed on a window ending strictly before the prediction time and shifted by one step. Failure windows and the repair-recovery period are excluded from training entirely, and a 30-minute guard band is dropped, because an alert ten minutes ahead is worthless.
The split became time-ordered with a 48-hour embargo between train and test. A random shuffle puts the ten minutes either side of a failure in both halves.

What the leaky model was actually using

Permutation importance on the in-window model. Every top feature is a present-tense state, not a trend.

Feature	Importance
`now_DV_pressure_min`	0.0131
`whole_duty_vs_overall`	0.0069
`now_TP2_max`	0.0046
`now_DV_pressure_max`	0.0027
`now_Oil_temperature_std`	0.0016
`now_DV_pressure_std`	0.0015
`now_TP3_min`	0.0015
`now_Reservoirs_min`	0.0013

The four leakage patterns

Contemporaneous state (L1) — a feature that is a consequence of the label. Isolated as run R5: AUPRC 0.0212.
Whole-series aggregates (L3) — any statistic, scaler or normalisation fitted across the full series leaks the future backwards. Run R6: 0.0172.
Downstream artefacts — maintenance flags and repair codes, recorded because the failure happened.
Group leakage — the same unit or batch either side of the split.

The diagnostic habit worth taking away: a score that looks too good is evidence of leakage, not of skill. Investigate before celebrating.

Computed by scripts/build_models.py in 86.4s at 2026-08-09T19:08:15+00:00 · scikit-learn 1.6.1 · seed 20260809 · source: MetroPT-3 (Air Production Unit of a Metro do Porto train), CC BY 4.0, DOI 10.24432/C5VW3R. Nothing on this page is hardcoded.