milia.anomaly.conformal#
Split-conformal thresholds on anomaly scores, globally or per group.
Calibrating on held-out, known-normal scores turns “pick a percentile” into a
checkable guarantee: a new normal point is flagged with probability at most
alpha. Scores here are anomaly scores (higher = more anomalous); a fitted
detector’s own calibrate() does the same
in one step. pvalues() gives the same test as conformal p-values, smoothed
for tied scores, and select() flags many queries at once while bounding the
share of flagged queries that are normal (the false discovery rate).
Functions
|
The threshold that maximises F1 on labelled scores: normal calibration scores and known faulty scores. |
|
Split-conformal threshold from held-out, known-normal anomaly scores. |
|
|
|
Threshold of a Gaussian assumption: the calibration mean plus the normal quantile times their spread. |
|
Flag each score against its own group's threshold. |
|
Robust threshold: the median plus the normal quantile times the scaled median absolute deviation. |
|
One conformal threshold per group (Mondrian conformal prediction). |
|
Extreme-value threshold: a generalised Pareto tail fitted to the excesses over the median, read at |
|
Conformal p-values of anomaly scores against held-out normal scores. |
|
Benjamini-Hochberg selection: flag queries while bounding the false discovery rate. |
- milia.anomaly.conformal.calibrate(scores, alpha=0.1)#
Split-conformal threshold from held-out, known-normal anomaly scores.
- Parameters:
- Return type:
- Returns:
The threshold; a new normal score exceeds it with probability at most
alpha.
Examples
>>> calibrate(np.arange(1.0, 101.0), alpha=0.1) 91.0
- milia.anomaly.conformal.flag(scores, threshold)#
Truewhere a score exceeds the threshold.- Parameters:
scores (
ArrayLike) – Anomaly scores.threshold (
float) – Fromcalibrate().
- Return type:
- Returns:
Boolean mask.
Examples
>>> flag([0.1, 0.95], 0.9).tolist() [False, True]
- milia.anomaly.conformal.mondrian(scores, groups, alpha=0.1)#
One conformal threshold per group (Mondrian conformal prediction).
Use when groups differ systematically (e.g. PV and non-PV customers), so a single threshold would over-flag one group and under-flag the other.
- Parameters:
- Return type:
- Returns:
Group label to threshold.
Examples
>>> thresholds = mondrian(np.r_[np.arange(10.0), np.arange(10.0) + 100], ["a"] * 10 + ["b"] * 10, alpha=0.2) >>> thresholds["a"] < 10 < thresholds["b"] True
- milia.anomaly.conformal.groupflag(scores, groups, thresholds)#
Flag each score against its own group’s threshold.
- Parameters:
- Return type:
- Returns:
Boolean mask.
Examples
>>> groupflag([5.0, 5.0], ["a", "b"], {"a": 3.0, "b": 9.0}).tolist() [True, False]
- milia.anomaly.conformal.pvalues(calibration, scores, smoothed=False, rng=None)#
Conformal p-values of anomaly scores against held-out normal scores.
The p-value of a score
vis(1 + #{s_i >= v}) / (n + 1), so a new normal score has a p-value of at mostalphawith probability at mostalpha(Vovk et al., 2005; Bates et al., 2023). With tied scores, which detectors on count features produce, this bound is conservative and the threshold coarse. The smoothed p-value(#{s_i > v} + u (1 + #{s_i = v})) / (n + 1), withuuniform on[0, 1], breaks ties at random and makes the probability exactlyalpha.- Parameters:
calibration (
ArrayLike) – Anomaly scores of known-normal calibration data.scores (
ArrayLike) – Anomaly scores of the queries.smoothed (
bool) – Break ties at random, giving exact rather than conservative validity.rng (
Generator|None) – Generator for the smoothing; a fresh default generator when omitted.
- Return type:
- Returns:
P-values in
(0, 1], one per query.
Examples
>>> pvalues(np.arange(1.0, 10.0), [0.5, 5.0, 20.0]).round(2).tolist() [1.0, 0.6, 0.1] >>> p = pvalues(np.ones(9), np.ones(1000), smoothed=True, rng=np.random.default_rng(0)) >>> 0.45 < float(p.mean()) < 0.55 # tied scores spread uniformly instead of all at 1.0 True
- milia.anomaly.conformal.select(pvalues, alpha=0.1)#
Benjamini-Hochberg selection: flag queries while bounding the false discovery rate.
A per-query threshold bounds the probability that each normal query is flagged; when many meters are tested at once, it does not bound the share of flagged meters that are normal. Applied to conformal p-values, the Benjamini-Hochberg procedure bounds that share, the false discovery rate, at
alpha(Bates et al., 2023, who show the p-values are positively dependent).- Parameters:
- Return type:
- Returns:
Boolean mask of the selected (flagged) queries.
Examples
>>> select([0.001, 0.008, 0.039, 0.041, 0.6], alpha=0.05).tolist() [True, True, False, False, False]
- milia.anomaly.conformal.gaussian(calibration, alpha=0.1)#
Threshold of a Gaussian assumption: the calibration mean plus the normal quantile times their spread.
It holds its false-alarm rate only when the scores are normal; detector scores are usually skewed, so it is a baseline against which
calibrate()is compared.- Parameters:
- Return type:
- Returns:
The threshold.
Examples
>>> round(gaussian(np.random.default_rng(0).normal(size=10000), 0.1), 1) # the normal quantile, 1.28 1.3
- milia.anomaly.conformal.hampel(calibration, alpha=0.1)#
Robust threshold: the median plus the normal quantile times the scaled median absolute deviation.
The median absolute deviation measures the spread of the centre of the scores, not of their tail, so the rule flags more or fewer than
alphaof normal scores depending on the shape of the tail.- Parameters:
- Return type:
- Returns:
The threshold.
Examples
>>> round(hampel(np.random.default_rng(0).normal(size=10000), 0.1), 1) 1.3
- milia.anomaly.conformal.peaksover(calibration, alpha=0.1)#
Extreme-value threshold: a generalised Pareto tail fitted to the excesses over the median, read at
alpha.- Parameters:
- Return type:
- Returns:
The threshold.
Examples
>>> threshold = peaksover(np.random.default_rng(0).exponential(size=5000), 0.05) >>> bool(2.5 < threshold < 3.5) # the exponential's 95th percentile is 3.0 True
- milia.anomaly.conformal.bestf1(calibration, faulty)#
The threshold that maximises F1 on labelled scores: normal calibration scores and known faulty scores.
Its false-alarm rate follows from the share of faults among the labelled scores rather than from a target, and it needs labelled faults of the kind to be detected.
- Parameters:
- Return type:
- Returns:
The threshold; rows scoring above it are flagged.
Examples
>>> bestf1([1.0, 2.0, 3.0], [5.0, 6.0]) 3.0