milia.benchmark.compare#

Significance tests between conditions on per-unit results (twiga’s DM and MCS, and the Wilcoxon signed-rank test).

A unit is whatever is independent enough to compare on: an entity (meter) in a detector benchmark, a fold in a retrieval benchmark. Lower loss must be better.

Functions

losses(results)

Per-entity 1 - AUC of each condition, from selfbaseline() tables.

mcs(losses[, alpha, boot, seed])

The Model Confidence Set: the conditions statistically tied with the best (twiga).

pairwise(losses[, alpha])

Every pair of conditions compared by Diebold-Mariano, Benjamini-Hochberg corrected (twiga).

signedrank(first, second)

Two-sided p-value of the Wilcoxon signed-rank test of paired results.

milia.benchmark.compare.losses(results)#

Per-entity 1 - AUC of each condition, from selfbaseline() tables.

Entities are sorted, so the arrays pair unit by unit across conditions as the significance tests require.

Parameters:

results (Mapping[str, DataFrame]) – Condition name to its selfbaseline table.

Return type:

dict[str, ndarray]

Returns:

Condition to an array of per-entity losses (lower is better).

Examples

>>> from milia.anomaly.ensemble import members
>>> from milia.data import profiles, synthetic
>>> from milia.evaluate.anomaly import selfbaseline, split
>>> cube = profiles.tensor(synthetic.generate(meters=3, days=60, steps=24).pivot)
>>> results = {"ecod": selfbaseline(cube, members(["ecod"])[0][1], sets=split(60, (30, 15, 8)), count=2)}
>>> losses(results)["ecod"].shape
(3,)
milia.benchmark.compare.signedrank(first, second)#

Two-sided p-value of the Wilcoxon signed-rank test of paired results.

The test asks whether the paired differences are centred on zero, without assuming they are normal, so it suits shares, AUCs and errors compared unit by unit (a transformer, a household, a split).

Parameters:
  • first (ArrayLike) – Results of one method, one per unit.

  • second (ArrayLike) – Results of the other method on the same units, in the same order.

Return type:

float

Returns:

The p-value; NaN when every pair is equal, since the test is then undefined.

Examples

>>> rng = np.random.default_rng(0)
>>> base = rng.uniform(0.5, 0.9, 30)
>>> signedrank(base + 0.05, base) < 0.001, bool(np.isnan(signedrank(base, base)))
(True, True)
milia.benchmark.compare.pairwise(losses, alpha=0.05)#

Every pair of conditions compared by Diebold-Mariano, Benjamini-Hochberg corrected (twiga).

Parameters:
  • losses (Mapping[str, ArrayLike]) – Condition to per-unit losses, all paired unit by unit.

  • alpha (float) – False-discovery rate.

Return type:

DataFrame

Returns:

One row per pair, sorted by adjusted p-value (see twiga’s pairwise_dm_table).

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> per = {"good": rng.normal(0.1, 0.02, 40), "bad": rng.normal(0.3, 0.02, 40)}
>>> len(pairwise(per))
1
milia.benchmark.compare.mcs(losses, alpha=0.1, boot=1000, seed=0)#

The Model Confidence Set: the conditions statistically tied with the best (twiga).

Report this rather than a single winner: it says honestly which conditions the data cannot separate.

Parameters:
  • losses (Mapping[str, ArrayLike]) – Condition to per-unit losses.

  • alpha (float) – Significance level; the set has confidence 1 - alpha.

  • boot (int) – Bootstrap replications.

  • seed (int) – Bootstrap seed.

Returns:

twiga’s ModelConfidenceSet.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> per = {"good": rng.normal(0.1, 0.02, 40), "bad": rng.normal(0.3, 0.02, 40)}
>>> mcs(per, boot=200).included
['good']