milia.benchmark#
Benchmark harness: run folders, provenance, seeded conditions and significance tests.
The decision discipline follows the experiment repos: store every run with its
settings, versions and input hashes; treat each seed as its own condition; and
decide between methods with Diebold-Mariano tests (Benjamini-Hochberg
corrected) and the Model Confidence Set on per-unit losses, never on a blended
average rank alone. Significance tests come from twiga (pundamilia[bench]).
Modules
Significance tests between conditions on per-unit results (twiga's DM and MCS, and the Wilcoxon signed-rank test). |
|
Seeds as named conditions: |
|
Provenance sidecars: which settings, package versions and input files produced an output. |
|
Run folders: |