milia.core.conformal#

The split-conformal quantile, shared by every conformal method in milia.

Anomaly thresholds, cluster prediction sets and retrieval trust radii all reduce to the same step: take the finite-sample-corrected 1 - alpha quantile of a set of calibration nonconformity scores. That step lives here once.

Functions

level(n, alpha)

The corrected quantile level ceil((n + 1)(1 - alpha)) / n, capped at 1.

pvalue(scores, values)

Conformal p-value of each value against calibration scores: (1 + #{s_i >= v}) / (n + 1).

quantile(scores, alpha)

Split-conformal threshold from held-out calibration scores.

milia.core.conformal.level(n, alpha)#

The corrected quantile level ceil((n + 1)(1 - alpha)) / n, capped at 1.

The plain 1 - alpha quantile of n calibration scores under-covers by about 1 / n; the (n + 1) correction restores the finite-sample guarantee of split-conformal prediction.

Parameters:
  • n (int) – Number of calibration scores.

  • alpha (float) – Target miscoverage rate, in (0, 1).

Return type:

float

Returns:

The quantile level to take of the calibration scores.

Raises:

ValueError – If n < 1 or alpha is outside (0, 1).

Examples

>>> level(100, 0.1)
0.91
>>> level(10, 0.1)
1.0
milia.core.conformal.quantile(scores, alpha)#

Split-conformal threshold from held-out calibration scores.

The threshold is the ceil((n + 1)(1 - alpha))-th smallest of the n scores, an exact order statistic as in twiga’s calculate_conformal_value; interpolating between order statistics would give a threshold slightly wider than the guarantee needs. A new exchangeable score exceeds it with probability at most alpha. When n is too small for that rank to exist, the largest score is returned and the guarantee holds only approximately.

Parameters:
  • scores (ArrayLike) – Nonconformity scores of a held-out calibration set, higher meaning less conforming.

  • alpha (float) – Target miscoverage rate (0.1 gives 90% coverage).

Return type:

float

Returns:

The calibrated threshold.

Examples

>>> scores = np.array([0.1, 0.2, 0.15, 0.9, 0.3, 0.25, 0.18, 0.22, 0.12, 0.4])
>>> round(quantile(scores, alpha=0.1), 2)
0.9
>>> float(quantile(np.arange(1.0, 101.0), alpha=0.1))  # the 91st smallest of 100
91.0
milia.core.conformal.pvalue(scores, values)#

Conformal p-value of each value against calibration scores: (1 + #{s_i >= v}) / (n + 1).

A value lies within the threshold of quantile() at level alpha when its p-value exceeds alpha (Vovk et al., 2005), so a small p-value marks a query that conforms less than all but a few calibration cases. Ported from twiga’s MultivariateConformal.p_value.

Parameters:
  • scores (ArrayLike) – Nonconformity scores of a held-out calibration set.

  • values (ArrayLike) – Scores of the queries.

Return type:

ndarray

Returns:

P-values in (0, 1], with the shape of values.

Examples

>>> pvalue(np.arange(1.0, 10.0), [0.5, 5.0, 20.0]).round(2).tolist()
[1.0, 0.6, 0.1]