milia.recommend.retrieval#

Content-based retrieval: predict an entity’s outcome from its nearest labelled neighbours.

Two split-conformal calibrations on held-out labelled rows say how far to believe a retrieval.

  • The trust radius bounds novelty. A query’s score is the distance to its nearest labelled neighbour, and the radius \(\hat q\) is the conformal quantile of the held-out scores; a query farther than \(\hat q\) has no trustworthy precedent, e.g. a customer from another population or one changed by new DER.

  • The error interval bounds accuracy. With the held-out outcomes, the residual \(|y - \hat y|\) is calibrated into an interval \(\hat y \pm \varepsilon\); in its locally adaptive form the residual is divided by the spread of the neighbours’ outcomes \(\hat\sigma_k\), giving \(\hat y \pm \varepsilon\, \hat\sigma_k\), wider where the neighbours disagree [Romano et al., 2019].

Either calibration can be made per group, such as a network or a regime (Mondrian conformal prediction, Vovk et al., 2005), so that its guarantee holds within each group rather than only on average over them.

Functions

coverage(train, cal, target[, alpha])

Share of a target population trusted under a radius calibrated on the source population.

Classes

KNNRetriever([n_neighbors, metric])

Mean outcome of the k nearest labelled neighbours, with a conformal trust radius.

class milia.recommend.retrieval.KNNRetriever(n_neighbors=5, metric='minkowski')#

Bases: RegressorMixin, BaseEstimator

Mean outcome of the k nearest labelled neighbours, with a conformal trust radius.

Parameters:
  • n_neighbors (int) – Neighbours averaged per prediction.

  • metric (str) – Distance metric.

Variables:
  • tau – Trust radius set by calibrate() (inf until then).

  • scores – Held-out distances to the nearest labelled neighbour, kept by calibrate() for pvalue().

  • epsilon – Error quantile set by calibrate() with outcomes (nan until then).

  • group_tau – Trust radius of each group, when calibrated with groups.

  • group_epsilon – Error quantile of each group, when calibrated with outcomes and groups.

Examples

>>> rng = np.random.default_rng(0)
>>> X = rng.normal(size=(100, 2))
>>> model = KNNRetriever(n_neighbors=5).fit(X[:70], X[:70, 0]).calibrate(X[70:], alpha=0.1)
>>> model.trusted([[0.0, 0.0], [20.0, 20.0]]).tolist()
[True, False]
fit(X, y)#

Index the labelled rows.

Parameters:
Return type:

KNNRetriever

Returns:

The fitted retriever, with an infinite tau_ until calibrate().

predict(X)#

Mean outcome of each row’s nearest labelled neighbours.

Parameters:

X (ArrayLike) – Query features.

Return type:

ndarray

Returns:

One prediction per row.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> X = rng.normal(size=(80, 2))
>>> y = 2 * X[:, 0]
>>> model = KNNRetriever(n_neighbors=3).fit(X[:60], y[:60])
>>> float(np.corrcoef(model.predict(X[60:]), y[60:])[0, 1]) > 0.9
True
distance(X)#

Distance from each row to its nearest labelled neighbour.

Parameters:

X (ArrayLike) – Query features.

Return type:

ndarray

Returns:

One distance per row.

spread(X)#

Standard deviation of the outcomes of each row’s nearest labelled neighbours.

Floored at a millionth of the standard deviation of all labelled outcomes, so that a query whose neighbours agree exactly still has a positive scale.

Parameters:

X (ArrayLike) – Query features.

Return type:

ndarray

Returns:

One spread per row.

Examples

>>> model = KNNRetriever(n_neighbors=2).fit([[0.0], [1.0], [10.0], [11.0]], [0.0, 2.0, 5.0, 5.0])
>>> model.spread([[0.5], [10.5]]).round(3).tolist()
[1.0, 0.0]
calibrate(X, alpha=0.1, y=None, groups=None, adaptive=False)#

Calibrate the trust radius and, given outcomes, the error interval on held-out labelled rows.

The trust radius tau_ is the split-conformal quantile of the held-out rows’ distances to their nearest labelled neighbour, so a new exchangeable row lies within it with probability at least 1 - alpha. With y, the error quantile epsilon_ is the same quantile of the residuals |y - predict(X)|, divided by spread() when adaptive, and interval() then covers a new exchangeable outcome with probability at least 1 - alpha. With groups, both are also calibrated within each group.

Parameters:
  • X (ArrayLike) – Held-out labelled features, not used in fit.

  • alpha (float) – Miscoverage level.

  • y (ArrayLike | None) – Held-out outcomes; the error interval is calibrated only when given.

  • groups (ArrayLike | None) – Group of each held-out row, such as its network; calibrates per group when given.

  • adaptive (bool) – Scale the residuals by the neighbours’ spread.

Return type:

KNNRetriever

Returns:

The retriever, calibrated.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> X = rng.normal(size=(80, 2))
>>> y = 2 * X[:, 0]
>>> model = KNNRetriever().fit(X[:60], y[:60]).calibrate(X[60:])
>>> bool(np.isfinite(model.tau_)), model.trusted([[50.0, 50.0]]).tolist()
(True, [False])
trusted(X, groups=None)#

Whether each row has a labelled neighbour within the trust radius.

Parameters:
  • X (ArrayLike) – Query features.

  • groups (ArrayLike | None) – Group of each row; each is then held to its own group’s radius, and a group never calibrated has no trustworthy precedent.

Return type:

ndarray

Returns:

Boolean mask; False marks a query with no trustworthy precedent.

Examples

>>> model = KNNRetriever(n_neighbors=1).fit([[0.0], [10.0]], [0.0, 0.0])
>>> model = model.calibrate([[0.5], [1.0], [10.2], [10.4]], alpha=0.4, groups=["a", "a", "b", "b"])
>>> model.trusted([[0.8], [10.8], [0.8]], groups=["a", "b", "c"]).tolist()
[True, False, False]
pvalue(X)#

Conformal p-value of each row’s distance to its nearest labelled neighbour.

A row is trusted at level alpha when its p-value exceeds alpha; a small p-value marks a query that lies farther from every labelled row than all but a few held-out rows did.

Parameters:

X (ArrayLike) – Query features.

Return type:

ndarray

Returns:

One p-value per row, in (0, 1].

Raises:

ValueError – If the retriever was not calibrated.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> X = rng.normal(size=(100, 2))
>>> model = KNNRetriever().fit(X[:60], X[:60, 0]).calibrate(X[60:])
>>> model.pvalue([[0.0, 0.0], [20.0, 20.0]]).round(3).tolist()
[0.78, 0.024]
interval(X, groups=None)#

Conformal interval for each row’s outcome, from calibrate() with outcomes.

Parameters:
  • X (ArrayLike) – Query features.

  • groups (ArrayLike | None) – Group of each row, to use its own group’s error quantile.

Return type:

tuple[ndarray, ndarray]

Returns:

Lower and upper bounds.

Raises:

ValueError – If the error interval was not calibrated, or a group was never calibrated.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(1)
>>> X = rng.uniform(size=(400, 1))
>>> y = X[:, 0] + rng.normal(scale=0.1, size=400)
>>> model = KNNRetriever(n_neighbors=10).fit(X[:200], y[:200]).calibrate(X[200:300], y=y[200:300])
>>> lower, upper = model.interval(X[300:])
>>> 0.8 <= float(np.mean((y[300:] >= lower) & (y[300:] <= upper))) <= 1.0
True
set_score_request(*, sample_weight='$UNCHANGED$')#

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters#

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns#

selfobject

The updated object.

Return type:

KNNRetriever

milia.recommend.retrieval.coverage(train, cal, target, alpha=0.1)#

Share of a target population trusted under a radius calibrated on the source population.

About 1 - alpha in distribution; well below it under domain shift, e.g. another utility’s customers or customers changed by new DER.

Parameters:
  • train (ArrayLike) – Labelled source features.

  • cal (ArrayLike) – Held-out source features for calibration.

  • target (ArrayLike) – Features of the population checked.

  • alpha (float) – Miscoverage level.

Return type:

float

Returns:

The trusted share of target.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> X = rng.normal(size=(80, 2))
>>> y = 2 * X[:, 0]
>>> coverage(X[:40], X[40:60], X[60:] + 10.0)
0.0