milia.anomaly.conformal#

Split-conformal thresholds on anomaly scores, globally or per group.

Calibrating on held-out, known-normal scores turns “pick a percentile” into a checkable guarantee: a new normal point is flagged with probability at most alpha. Scores here are anomaly scores (higher = more anomalous); a fitted detector’s own calibrate() does the same in one step. pvalues() gives the same test as conformal p-values, smoothed for tied scores, and select() flags many queries at once while bounding the share of flagged queries that are normal (the false discovery rate).

Functions

bestf1(calibration, faulty)

The threshold that maximises F1 on labelled scores: normal calibration scores and known faulty scores.

calibrate(scores[, alpha])

Split-conformal threshold from held-out, known-normal anomaly scores.

flag(scores, threshold)

True where a score exceeds the threshold.

gaussian(calibration[, alpha])

Threshold of a Gaussian assumption: the calibration mean plus the normal quantile times their spread.

groupflag(scores, groups, thresholds)

Flag each score against its own group's threshold.

hampel(calibration[, alpha])

Robust threshold: the median plus the normal quantile times the scaled median absolute deviation.

mondrian(scores, groups[, alpha])

One conformal threshold per group (Mondrian conformal prediction).

peaksover(calibration[, alpha])

Extreme-value threshold: a generalised Pareto tail fitted to the excesses over the median, read at alpha.

pvalues(calibration, scores[, smoothed, rng])

Conformal p-values of anomaly scores against held-out normal scores.

select(pvalues[, alpha])

Benjamini-Hochberg selection: flag queries while bounding the false discovery rate.

milia.anomaly.conformal.calibrate(scores, alpha=0.1)#

Split-conformal threshold from held-out, known-normal anomaly scores.

Parameters:
  • scores (ArrayLike) – Anomaly scores (higher = more anomalous) of a calibration set not used in fitting.

  • alpha (float) – Target false-positive rate on new normal points.

Return type:

float

Returns:

The threshold; a new normal score exceeds it with probability at most alpha.

Examples

>>> calibrate(np.arange(1.0, 101.0), alpha=0.1)
91.0
milia.anomaly.conformal.flag(scores, threshold)#

True where a score exceeds the threshold.

Parameters:
Return type:

ndarray

Returns:

Boolean mask.

Examples

>>> flag([0.1, 0.95], 0.9).tolist()
[False, True]
milia.anomaly.conformal.mondrian(scores, groups, alpha=0.1)#

One conformal threshold per group (Mondrian conformal prediction).

Use when groups differ systematically (e.g. PV and non-PV customers), so a single threshold would over-flag one group and under-flag the other.

Parameters:
  • scores (ArrayLike) – Calibration anomaly scores.

  • groups (ArrayLike) – Group label of each score.

  • alpha (float) – Target false-positive rate within each group.

Return type:

dict[Hashable, float]

Returns:

Group label to threshold.

Examples

>>> thresholds = mondrian(np.r_[np.arange(10.0), np.arange(10.0) + 100], ["a"] * 10 + ["b"] * 10, alpha=0.2)
>>> thresholds["a"] < 10 < thresholds["b"]
True
milia.anomaly.conformal.groupflag(scores, groups, thresholds)#

Flag each score against its own group’s threshold.

Parameters:
Return type:

ndarray

Returns:

Boolean mask.

Examples

>>> groupflag([5.0, 5.0], ["a", "b"], {"a": 3.0, "b": 9.0}).tolist()
[True, False]
milia.anomaly.conformal.pvalues(calibration, scores, smoothed=False, rng=None)#

Conformal p-values of anomaly scores against held-out normal scores.

The p-value of a score v is (1 + #{s_i >= v}) / (n + 1), so a new normal score has a p-value of at most alpha with probability at most alpha (Vovk et al., 2005; Bates et al., 2023). With tied scores, which detectors on count features produce, this bound is conservative and the threshold coarse. The smoothed p-value (#{s_i > v} + u (1 + #{s_i = v})) / (n + 1), with u uniform on [0, 1], breaks ties at random and makes the probability exactly alpha.

Parameters:
  • calibration (ArrayLike) – Anomaly scores of known-normal calibration data.

  • scores (ArrayLike) – Anomaly scores of the queries.

  • smoothed (bool) – Break ties at random, giving exact rather than conservative validity.

  • rng (Generator | None) – Generator for the smoothing; a fresh default generator when omitted.

Return type:

ndarray

Returns:

P-values in (0, 1], one per query.

Examples

>>> pvalues(np.arange(1.0, 10.0), [0.5, 5.0, 20.0]).round(2).tolist()
[1.0, 0.6, 0.1]
>>> p = pvalues(np.ones(9), np.ones(1000), smoothed=True, rng=np.random.default_rng(0))
>>> 0.45 < float(p.mean()) < 0.55  # tied scores spread uniformly instead of all at 1.0
True
milia.anomaly.conformal.select(pvalues, alpha=0.1)#

Benjamini-Hochberg selection: flag queries while bounding the false discovery rate.

A per-query threshold bounds the probability that each normal query is flagged; when many meters are tested at once, it does not bound the share of flagged meters that are normal. Applied to conformal p-values, the Benjamini-Hochberg procedure bounds that share, the false discovery rate, at alpha (Bates et al., 2023, who show the p-values are positively dependent).

Parameters:
  • pvalues (ArrayLike) – Conformal p-values of the queries, from pvalues().

  • alpha (float) – Target false discovery rate.

Return type:

ndarray

Returns:

Boolean mask of the selected (flagged) queries.

Examples

>>> select([0.001, 0.008, 0.039, 0.041, 0.6], alpha=0.05).tolist()
[True, True, False, False, False]
milia.anomaly.conformal.gaussian(calibration, alpha=0.1)#

Threshold of a Gaussian assumption: the calibration mean plus the normal quantile times their spread.

It holds its false-alarm rate only when the scores are normal; detector scores are usually skewed, so it is a baseline against which calibrate() is compared.

Parameters:
  • calibration (ArrayLike) – Calibration anomaly scores.

  • alpha (float) – Target false-alarm rate.

Return type:

float

Returns:

The threshold.

Examples

>>> round(gaussian(np.random.default_rng(0).normal(size=10000), 0.1), 1)  # the normal quantile, 1.28
1.3
milia.anomaly.conformal.hampel(calibration, alpha=0.1)#

Robust threshold: the median plus the normal quantile times the scaled median absolute deviation.

The median absolute deviation measures the spread of the centre of the scores, not of their tail, so the rule flags more or fewer than alpha of normal scores depending on the shape of the tail.

Parameters:
  • calibration (ArrayLike) – Calibration anomaly scores.

  • alpha (float) – Target false-alarm rate.

Return type:

float

Returns:

The threshold.

Examples

>>> round(hampel(np.random.default_rng(0).normal(size=10000), 0.1), 1)
1.3
milia.anomaly.conformal.peaksover(calibration, alpha=0.1)#

Extreme-value threshold: a generalised Pareto tail fitted to the excesses over the median, read at alpha.

Parameters:
  • calibration (ArrayLike) – Calibration anomaly scores.

  • alpha (float) – Target false-alarm rate; must be below the share of scores above the median.

Return type:

float

Returns:

The threshold.

Examples

>>> threshold = peaksover(np.random.default_rng(0).exponential(size=5000), 0.05)
>>> bool(2.5 < threshold < 3.5)  # the exponential's 95th percentile is 3.0
True
milia.anomaly.conformal.bestf1(calibration, faulty)#

The threshold that maximises F1 on labelled scores: normal calibration scores and known faulty scores.

Its false-alarm rate follows from the share of faults among the labelled scores rather than from a target, and it needs labelled faults of the kind to be detected.

Parameters:
  • calibration (ArrayLike) – Scores of normal rows.

  • faulty (ArrayLike) – Scores of faulty rows.

Return type:

float

Returns:

The threshold; rows scoring above it are flagged.

Examples

>>> bestf1([1.0, 2.0, 3.0], [5.0, 6.0])
3.0