milia.evaluate.retrieval#
Benchmarking retrieval against baselines, and recommendation by replayed outcome.
Functions
|
Cross-validated scores of several predictors, ranked by diversity-weighted Borda. |
|
The cost of faulty meter readings to estimates retrieved from a case base, with and without screening. |
|
Share of held-out cases the recommended action actually fixes. |
- milia.evaluate.retrieval.benchmark(X, y, predictors, folds=5, metrics=('mae', 'spearman'), random_state=0)#
Cross-validated scores of several predictors, ranked by diversity-weighted Borda.
- Parameters:
- Return type:
- Returns:
One row per predictor with the mean of each metric and the ranking columns of
models(), best first.
Examples
>>> import numpy as np >>> from sklearn.dummy import DummyRegressor >>> from milia.recommend.retrieval import KNNRetriever >>> X = np.random.default_rng(0).normal(size=(60, 2)) >>> benchmark(X, X[:, 0] ** 2, {"knn": KNNRetriever(), "mean": DummyRegressor()}).index[0] 'knn'
- milia.evaluate.retrieval.success(recommender, X, actions, outcome, test=0.3, stratify=None, random_state=0)#
Share of held-out cases the recommended action actually fixes.
- Parameters:
recommender (
BaseEstimator) – Unfitted classifier from case features to actions.X (
ArrayLike) – Case features.actions (
ArrayLike) – The action that fixed each past case.outcome (
Callable[[int,object],bool]) –outcome(case, action)replays a case with an action and says whether it is fixed (e.g. a power-flow rerun).test (
float) – Held-out share.stratify (
ArrayLike|None) – Labels to stratify the split by (e.g. violation type).random_state (
int) – Split seed.
- Return type:
- Returns:
cases,fixedandrate.
Examples
>>> import numpy as np >>> from milia.recommend.action import CaseRecommender >>> X = np.r_[np.zeros((10, 1)), np.ones((10, 1))] >>> actions = np.repeat(["a", "b"], 10) >>> success(CaseRecommender(), X, actions, lambda case, act: act == actions[case])["rate"] 1.0
- milia.evaluate.retrieval.faultycases(cube, cases, outcome, fault, share, splits=50, holdout=0.4, neighbours=10, seed=0, featureset='sizes')#
The cost of faulty meter readings to estimates retrieved from a case base, with and without screening.
Each case is a meter’s simulated outcome (for example its contribution to an evening peak), retrieved for a query as the mean outcome of its nearest cases in standardised size features (
sizes()). In each split, the cases of a share of the meters kept for retrieval are corrupted (meterfault()), and the estimates for held-out meters are compared from the clean, the corrupted and the screened case base (screen()).- Parameters:
cube (
ArrayLike) – Power in kW, shape(meters, days, steps).cases (
ArrayLike) – The meter of each case; a meter may have several cases.outcome (
ArrayLike) – The outcome of each case.fault (
str) – One ofMETERFAULTS.share (
float) – Share of the kept meters whose readings are corrupted.splits (
int) – Random splits of the meters.holdout (
float) – Share of the meters whose cases are queries.neighbours (
int) – Number of nearest cases averaged.seed (
int) – Seed of the splits and the faults.featureset (
str) –"sizes"(sizes()) or"shape"(shapes(), invariant to a constant under-recording).
- Return type:
- Returns:
One row per split and case base (
clean,corrupted,screened) –mae,tau(Kendall’s tau between estimated and true outcomes),removed(the share of kept cases screened out),faultandshare.
Examples
>>> rng = np.random.default_rng(0) >>> cube = rng.uniform(0.2, 3.0, (30, 40, 48)).round(3) >>> result = faultycases(cube, np.arange(30), cube.mean(axis=(1, 2)), "dropout", 0.3, splits=2, neighbours=3) >>> sorted(result["casebase"].unique()) ['clean', 'corrupted', 'screened']