Drift detection and the retraining trigger

A model is fitted to a distribution. When the distribution moves, the model is wrong in a way no test-set score will reveal, because the test set came from the old world.

Reference against current

Reference window: 2020-02-01 → 2020-06-01
Current window: 2020-07-01 → 2020-09-02

Features drifting significantly

of 7 monitored, PSI ≥ 0.25

Score distribution PSI

1.243

significant shift

Retraining trigger: FIRED. The rule is a documented threshold, evaluated on a schedule, with an owner — not a quarterly meeting at which somebody wonders whether the model is still any good.

Per-signal drift

Signal	PSI	Verdict	KS D	KS p
`Oil_temperature`	2.3376	significant shift	0.5538	0.00e+00
`DV_eletric`	0.2002	moderate shift	0.1730	2.40e-127
`Motor_current`	0.4024	significant shift	0.2195	1.64e-205
`TP3`	0.3082	significant shift	0.1122	1.25e-53
`TP2`	1.1127	significant shift	0.2594	2.58e-288
`Caudal_impulses`	0.0000	insufficient variation in reference	0.1126	5.22e-54
`H1`	0.1875	moderate shift	0.1391	2.83e-82

The trap in this table

Every KS p-value here is effectively zero. With tens of thousands of samples per window, a KS test will call any change significant, including one of no consequence whatsoever. The D statistic is the effect size, and the PSI is the actionable quantity. Alerting on the p-value produces a drift monitor that fires every day and is switched off within a month.

The conventional PSI bands — below 0.1 stable, 0.1 to 0.25 moderate, above 0.25 significant — are a rule of thumb, not a test. Their value is that they are written down in advance and tied to an action.

Computed by scripts/build_models.py in 86.4s at 2026-08-09T19:08:15+00:00 · scikit-learn 1.6.1 · seed 20260809 · source: MetroPT-3 (Air Production Unit of a Metro do Porto train), CC BY 4.0, DOI 10.24432/C5VW3R. Nothing on this page is hardcoded.