milia.evaluate.retrieval#

Benchmarking retrieval against baselines, and recommendation by replayed outcome.

Functions

benchmark(X, y, predictors[, folds, ...])

Cross-validated scores of several predictors, ranked by diversity-weighted Borda.

faultycases(cube, cases, outcome, fault, share)

The cost of faulty meter readings to estimates retrieved from a case base, with and without screening.

success(recommender, X, actions, outcome[, ...])

Share of held-out cases the recommended action actually fixes.

milia.evaluate.retrieval.benchmark(X, y, predictors, folds=5, metrics=('mae', 'spearman'), random_state=0)#

Cross-validated scores of several predictors, ranked by diversity-weighted Borda.

Parameters:
  • X (ArrayLike) – Features.

  • y (ArrayLike) – Outcome.

  • predictors (Mapping[str, BaseEstimator]) – Name to unfitted regressor (e.g. KNNRetriever, GroupMean, DummyRegressor).

  • folds (int) – Shuffled k-fold splits.

  • metrics (Sequence[str]) – Names from SCORES.

  • random_state (int) – Fold seed.

Return type:

DataFrame

Returns:

One row per predictor with the mean of each metric and the ranking columns of models(), best first.

Examples

>>> import numpy as np
>>> from sklearn.dummy import DummyRegressor
>>> from milia.recommend.retrieval import KNNRetriever
>>> X = np.random.default_rng(0).normal(size=(60, 2))
>>> benchmark(X, X[:, 0] ** 2, {"knn": KNNRetriever(), "mean": DummyRegressor()}).index[0]
'knn'
milia.evaluate.retrieval.success(recommender, X, actions, outcome, test=0.3, stratify=None, random_state=0)#

Share of held-out cases the recommended action actually fixes.

Parameters:
  • recommender (BaseEstimator) – Unfitted classifier from case features to actions.

  • X (ArrayLike) – Case features.

  • actions (ArrayLike) – The action that fixed each past case.

  • outcome (Callable[[int, object], bool]) – outcome(case, action) replays a case with an action and says whether it is fixed (e.g. a power-flow rerun).

  • test (float) – Held-out share.

  • stratify (ArrayLike | None) – Labels to stratify the split by (e.g. violation type).

  • random_state (int) – Split seed.

Return type:

dict[str, float]

Returns:

cases, fixed and rate.

Examples

>>> import numpy as np
>>> from milia.recommend.action import CaseRecommender
>>> X = np.r_[np.zeros((10, 1)), np.ones((10, 1))]
>>> actions = np.repeat(["a", "b"], 10)
>>> success(CaseRecommender(), X, actions, lambda case, act: act == actions[case])["rate"]
1.0
milia.evaluate.retrieval.faultycases(cube, cases, outcome, fault, share, splits=50, holdout=0.4, neighbours=10, seed=0, featureset='sizes')#

The cost of faulty meter readings to estimates retrieved from a case base, with and without screening.

Each case is a meter’s simulated outcome (for example its contribution to an evening peak), retrieved for a query as the mean outcome of its nearest cases in standardised size features (sizes()). In each split, the cases of a share of the meters kept for retrieval are corrupted (meterfault()), and the estimates for held-out meters are compared from the clean, the corrupted and the screened case base (screen()).

Parameters:
  • cube (ArrayLike) – Power in kW, shape (meters, days, steps).

  • cases (ArrayLike) – The meter of each case; a meter may have several cases.

  • outcome (ArrayLike) – The outcome of each case.

  • fault (str) – One of METERFAULTS.

  • share (float) – Share of the kept meters whose readings are corrupted.

  • splits (int) – Random splits of the meters.

  • holdout (float) – Share of the meters whose cases are queries.

  • neighbours (int) – Number of nearest cases averaged.

  • seed (int) – Seed of the splits and the faults.

  • featureset (str) – "sizes" (sizes()) or "shape" (shapes(), invariant to a constant under-recording).

Return type:

DataFrame

Returns:

One row per split and case base (clean, corrupted, screened) – mae, tau (Kendall’s tau between estimated and true outcomes), removed (the share of kept cases screened out), fault and share.

Examples

>>> rng = np.random.default_rng(0)
>>> cube = rng.uniform(0.2, 3.0, (30, 40, 48)).round(3)
>>> result = faultycases(cube, np.arange(30), cube.mean(axis=(1, 2)), "dropout", 0.3, splits=2, neighbours=3)
>>> sorted(result["casebase"].unique())
['clean', 'corrupted', 'screened']