milia.benchmark#

Benchmark harness: run folders, provenance, seeded conditions and significance tests.

The decision discipline follows the experiment repos: store every run with its settings, versions and input hashes; treat each seed as its own condition; and decide between methods with Diebold-Mariano tests (Benjamini-Hochberg corrected) and the Model Confidence Set on per-unit losses, never on a blended average rank alone. Significance tests come from twiga (pundamilia[bench]).

Modules

compare

Significance tests between conditions on per-unit results (twiga's DM and MCS, and the Wilcoxon signed-rank test).

conditions

Seeds as named conditions: "lof@s2" is the LOF condition replicated with seed 2.

provenance

Provenance sidecars: which settings, package versions and input files produced an output.

runs

Run folders: <root>/<experiment>/runs/<timestamp>-<hash>-<commit>/ with a latest link.