A formal hypothesis test

Regime segmentation is a plausible improvement. Plausibility is not evidence. This is the test, stated before it was run, and its result — which is not the result we wanted.

The hypotheses

Null (H₀). Adding operating-regime segmentation to the causal feature set does not change the proportion of 10-minute windows classified correctly, at a matched budget of one false alert per operating day.

Alternative (H₁). Regime segmentation changes that proportion. Two-sided; we did not assume the direction before running it.

Test selection

McNemar's exact test. The two models are scored on the same out-of-sample windows, so the samples are paired and only the discordant windows carry evidence. An unpaired two-proportion test would be wrong here.

Matched budget. Both models are compared at the same operational cost — 1.0 false alert per operating day — rather than at an arbitrary 0.5 threshold. Comparing two models at a fixed probability threshold compares their calibration, not their discrimination.

Result

p-value

1.63e-22

significant at α=0.05

Effect size

-0.0330

accuracy difference, regime − causal

95% CI on the effect: [-0.0396, -0.0264], n = 13,547 paired windows.

Bootstrap on ΔAUPRC

Paired bootstrap, the appropriate method for a metric with no closed-form variance: -0.0011, 95% CI [-0.0042, 0.0009]. The interval contains zero.

The honest reading. The p-value is overwhelming and the effect is negative. Regime segmentation did not help; on this asset it made the classifier slightly worse. The hypothesis is rejected in the direction we did not expect.

Why the p-value is not the finding

p = 1.63e-22 is a statement that the difference is probably not exactly zero. With 13,547 paired windows, almost nothing is exactly zero. The finding is the effect size and its interval: an accuracy change of -3.30%, which is both real and operationally irrelevant, and in the wrong direction.

Report these together, always

p-value — is the effect distinguishable from zero
Effect size with an interval — is it large enough to matter
The sample size and the paired structure — could the test have detected it

Power

With four failure events no event-level test can reach significance: the exact binomial floor for n=4 is p=0.0625 one-sided even if one method wins on every event. The test above is therefore run at the window level, and the windows inside one event are heavily autocorrelated. Read the window-level p-value as within-event consistency, not as independent-event evidence. The AUPRC interval is bootstrapped in day-sized blocks for the same reason.

Computed by scripts/build_models.py in 86.4s at 2026-08-09T19:08:15+00:00 · scikit-learn 1.6.1 · seed 20260809 · source: MetroPT-3 (Air Production Unit of a Metro do Porto train), CC BY 4.0, DOI 10.24432/C5VW3R. Nothing on this page is hardcoded.