A preregistered CPU audit suite that asks which world-model diagnostics actually predict closed-loop control.
World models are trained and reported on pixel fidelity (MSE, NLL, FVD), but planners consume counterfactual action effects and reward predictions. wm-audit measures each diagnostic against the real closed-loop return and correlates them across seed and capacity, with percentile-bootstrap CIs and Holm-corrected p-values fixed before seeing data.
Perceptual fidelity numbers can look good while a planner still fails to solve the task. Without a preregistered split between "looks right" and "controls well", a model can be selected on the wrong axis.
| Hypothesis | Observed | Verdict |
|---|---|---|
| H1: pixel composite weakly tracks return (|ρ| < 0.4) | ρ = −0.317, |ρ| = 0.32 | Threshold met — weak/tentative; not significant (n=9, CI crosses zero, Holm p=1.0) |
| H2: control composite predicts return (ρ > 0.6) | ρ = 0.259 | Not supported |
| H3: INT8 degrades calibration faster than pixel fidelity (n_pairs=9) | pixel 0.433 > control 0.352 | Not supported |
Strongest single diagnostic: uncertainty_rho ρ = 0.70, Holm-adjusted p = 0.393
(not significant). All Holm-adjusted p-values are non-significant at n=9; these are directional signals,
not confirmed effects.
pixel_lpips and rollout_fvd
are deterministic seeded random-conv filter banks standing in for learned VGG/AlexNet backbones; real
numbers, reproducible, but not publication-grade LPIPS/FVD.