Compare#
milia.evaluate and milia.benchmark compare methods on per-unit results, record how every run was produced, and
test whether a difference is real.
Per-unit losses#
A comparison needs one loss for each unit, such as each meter, so that the test sees the spread and not only the
average. compare.pairwise runs the Diebold-Mariano test on every pair with a Benjamini-Hochberg adjustment, and
compare.mcs returns the Model Confidence Set: the methods that are statistically tied with the best.
import numpy as np
from milia.benchmark import compare
rng = np.random.default_rng(0)
losses = {
"knn": rng.normal(0.20, 0.05, 40),
"ensemble": rng.normal(0.25, 0.05, 40),
"lof": rng.normal(0.20, 0.05, 40),
}
compare.pairwise(losses).round(3)
/Users/anthonyfaustine/Documents/projects/safina-docs/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
| condition_a | condition_b | p_value | p_adjusted | significant | winner | |
|---|---|---|---|---|---|---|
| 0 | knn | ensemble | 1.000 | 0.000 | True | knn |
| 1 | ensemble | lof | 0.000 | 0.000 | True | lof |
| 2 | knn | lof | 0.639 | 0.723 | False | knn |
compare.mcs(losses)
ModelConfidenceSet(included=['knn', 'lof'], eliminated=['ensemble'], p_values={'ensemble': 0.0, 'knn': 1.0, 'lof': 1.0}, alpha=0.1, best='knn')
Seeds as named conditions#
conditions names a replicate by its seed, so "lof@s2" is the LOF condition with seed 2. A claim then rests on
several seeds, not on one lucky run.
Run folders and provenance#
runs writes each experiment to a folder named by timestamp, hash and commit, with a latest link, and provenance
saves the settings, the package versions and the input files next to the output. A number can then be traced to the code
that produced it.
Harnesses#
milia.evaluate holds the harnesses: the self-baseline benchmark for detectors, which scores each entity against its own
normal days with injected anomalies, and the benchmarks for retrieval and ranking.
The self-baseline protocol: fit on one block of normal days, set the threshold on another, measure the false-alarm rate on a third, and measure recall on days with an injected fault.#
Limits#
A significance test says a difference is reproducible on these units, not that it matters for your decision. Report the size of the difference with the test.