milia.benchmark.compare#
Significance tests between conditions on per-unit results (twiga’s DM and MCS, and the Wilcoxon signed-rank test).
A unit is whatever is independent enough to compare on: an entity (meter) in a detector benchmark, a fold in a retrieval benchmark. Lower loss must be better.
Functions
|
Per-entity |
|
The Model Confidence Set: the conditions statistically tied with the best (twiga). |
|
Every pair of conditions compared by Diebold-Mariano, Benjamini-Hochberg corrected (twiga). |
|
Two-sided p-value of the Wilcoxon signed-rank test of paired results. |
- milia.benchmark.compare.losses(results)#
Per-entity
1 - AUCof each condition, fromselfbaseline()tables.Entities are sorted, so the arrays pair unit by unit across conditions as the significance tests require.
- Parameters:
results (
Mapping[str,DataFrame]) – Condition name to its selfbaseline table.- Return type:
- Returns:
Condition to an array of per-entity losses (lower is better).
Examples
>>> from milia.anomaly.ensemble import members >>> from milia.data import profiles, synthetic >>> from milia.evaluate.anomaly import selfbaseline, split >>> cube = profiles.tensor(synthetic.generate(meters=3, days=60, steps=24).pivot) >>> results = {"ecod": selfbaseline(cube, members(["ecod"])[0][1], sets=split(60, (30, 15, 8)), count=2)} >>> losses(results)["ecod"].shape (3,)
- milia.benchmark.compare.signedrank(first, second)#
Two-sided p-value of the Wilcoxon signed-rank test of paired results.
The test asks whether the paired differences are centred on zero, without assuming they are normal, so it suits shares, AUCs and errors compared unit by unit (a transformer, a household, a split).
- Parameters:
- Return type:
- Returns:
The p-value; NaN when every pair is equal, since the test is then undefined.
Examples
>>> rng = np.random.default_rng(0) >>> base = rng.uniform(0.5, 0.9, 30) >>> signedrank(base + 0.05, base) < 0.001, bool(np.isnan(signedrank(base, base))) (True, True)
- milia.benchmark.compare.pairwise(losses, alpha=0.05)#
Every pair of conditions compared by Diebold-Mariano, Benjamini-Hochberg corrected (twiga).
- Parameters:
- Return type:
- Returns:
One row per pair, sorted by adjusted p-value (see twiga’s
pairwise_dm_table).
Examples
>>> import numpy as np >>> rng = np.random.default_rng(0) >>> per = {"good": rng.normal(0.1, 0.02, 40), "bad": rng.normal(0.3, 0.02, 40)} >>> len(pairwise(per)) 1
- milia.benchmark.compare.mcs(losses, alpha=0.1, boot=1000, seed=0)#
The Model Confidence Set: the conditions statistically tied with the best (twiga).
Report this rather than a single winner: it says honestly which conditions the data cannot separate.
- Parameters:
- Returns:
twiga’s
ModelConfidenceSet.
Examples
>>> import numpy as np >>> rng = np.random.default_rng(0) >>> per = {"good": rng.normal(0.1, 0.02, 40), "bad": rng.normal(0.3, 0.02, 40)} >>> mcs(per, boot=200).included ['good']