milia.evaluate.anomaly#
Evaluating detectors by injection against each entity’s own normal baseline.
The protocol, for each entity (a meter with many days of readings):
Split its days into training, calibration, held-out normal and an anomaly pool.
Fit the detector on the training days and score every other day.
Inject known anomalies into pool days and score them too.
Calibrate a conformal threshold on the calibration days, then report the false-positive rate on held-out normal days, the recall on injected days, and the threshold-free AUC.
selfbaseline() produces every score once; summarize() turns them into
metrics at any alpha without refitting, so an alpha sweep is cheap.
Functions
|
Share of meters still described by a segmentation, and still trusted, as a growing share adopts PV or EVs. |
|
|
|
ROC AUC of anomaly scores separating injected days (positive) from normal days. |
|
Test the energy balance of a transformer block by block, with one meter tampering in the held-out block. |
|
Compare representations of a day for detecting behaviour faults, each entity against its own past. |
|
Share of scores still within a fixed threshold as some drift parameter grows. |
|
Mean per-entity AUC of normal against injected scores, for each injection separately. |
|
Feature matrix of the chosen days of one entity. |
|
Precision and recall of a flagged set against a ground-truth set. |
|
Detect, week by week, the meters that stop resembling their segment after adopting a technology. |
|
A novelty detector fitted on normal rows, whose |
|
Point adjustment: a true anomalous segment counts as found in full if any one of its readings is flagged. |
|
Profile features of each meter of a cube of daily readings, one row per meter. |
|
Score calibration, held-out normal and injected days for every entity. |
|
Shuffle day indices and split them into training, calibration, normal and anomaly-pool sets. |
|
Detection metrics at one |
Classes
|
The outcome of |
- milia.evaluate.anomaly.split(n, sizes=(280, 40, 20), exclude=(), seed=1)#
Shuffle day indices and split them into training, calibration, normal and anomaly-pool sets.
- Parameters:
- Return type:
- Returns:
{"train", "cal", "normal", "pool"}index arrays.- Raises:
ValueError – If the sizes leave no anomaly pool.
Examples
>>> sets = split(100, sizes=(50, 20, 10)) >>> [len(sets[k]) for k in ("train", "cal", "normal", "pool")] [50, 20, 10, 20]
- milia.evaluate.anomaly.features(days, index, featurize)#
Feature matrix of the chosen days of one entity.
- Parameters:
- Return type:
- Returns:
Array with one row per chosen day.
Examples
>>> features(np.arange(12.0).reshape(3, 4), np.array([0, 2]), lambda day: day[:2]).tolist() [[0.0, 1.0], [8.0, 9.0]]
- milia.evaluate.anomaly.auc(normal, anomalous)#
ROC AUC of anomaly scores separating injected days (positive) from normal days.
- Parameters:
- Return type:
- Returns:
The AUC, 1 for perfect separation and 0.5 for chance.
Examples
>>> auc([0.1, 0.2], [0.9, 0.8]) 1.0
- milia.evaluate.anomaly.selfbaseline(entities, detector, featurize=None, injections=None, sets=None, count=5)#
Score calibration, held-out normal and injected days for every entity.
Each entity gets its own detector, fitted on its own training days, so the baseline is the entity’s normal behaviour rather than the population’s. Injection seeds are derived from the injection, entity, repeat and day, so results reproduce across processes.
- Parameters:
entities (
Mapping[Hashable,ArrayLike] |ArrayLike) – Per-entity(days, steps)arrays, as a mapping or a(entities, days, steps)array.detector (
BaseDetector) – Unfitted detector, cloned per entity.featurize (
Callable[[ndarray],ndarray] |None) – Maps one day’s readings to a feature vector; identity when omitted.injections (
Mapping[str,Callable[...,ndarray]] |None) – Name to injector;INJECTIONSwhen omitted.sets (
dict[str,ndarray] |None) – Day split fromsplit(); a default split when omitted.count (
int) – Injected days per anomaly type per entity.
- Return type:
- Returns:
One row per scored day –
entity,set("cal","normal"or the injection name),dayandscore(higher = more anomalous).
Examples
>>> from milia.anomaly.ensemble import members >>> from milia.data import profiles, synthetic >>> cube = profiles.tensor(synthetic.generate(meters=2, days=60, steps=24).pivot) >>> sets = split(60, sizes=(30, 15, 8)) >>> results = selfbaseline(cube, members(["ecod"])[0][1], sets=sets, count=2) >>> sorted(results["set"].unique()) ['cal', 'drift', 'dropout', 'normal', 'stuck']
- milia.evaluate.anomaly.summarize(results, alpha=0.1)#
Detection metrics at one
alpha: mean per-entity AUC, pooled false-positive rate and recall.Each entity’s threshold is calibrated on its own
"cal"scores, so the false-positive rate should not exceedalphaby more than sampling noise.- Parameters:
results (
DataFrame) – Output ofselfbaseline().alpha (
float) – Target false-positive rate.
- Return type:
- Returns:
auc,fpr,recallandrecall<kind>for each injection.
Examples
>>> from milia.anomaly.ensemble import members >>> from milia.data import profiles, synthetic >>> cube = profiles.tensor(synthetic.generate(meters=2, days=60, steps=24).pivot) >>> sets = split(60, sizes=(30, 15, 8)) >>> results = selfbaseline(cube, members(["ecod"])[0][1], sets=sets, count=2) >>> sorted(summarize(results, alpha=0.1)) ['auc', 'fpr', 'recall', 'recalldrift', 'recalldropout', 'recallstuck']
- milia.evaluate.anomaly.alphas(results, levels=(0.2, 0.1, 0.05))#
summarize()at severalalphalevels: target coverage against realised false-positive rate and recall.- Parameters:
results (
DataFrame) – Output ofselfbaseline().
- Return type:
- Returns:
One row per level with
alpha,coverageand the metrics.
Examples
>>> from milia.anomaly.ensemble import members >>> from milia.data import profiles, synthetic >>> cube = profiles.tensor(synthetic.generate(meters=2, days=60, steps=24).pivot) >>> sets = split(60, sizes=(30, 15, 8)) >>> results = selfbaseline(cube, members(["ecod"])[0][1], sets=sets, count=2) >>> alphas(results)["alpha"].tolist() [0.2, 0.1, 0.05]
- milia.evaluate.anomaly.drift(scores, threshold)#
Share of scores still within a fixed threshold as some drift parameter grows.
For example a retrieval trust radius calibrated before DER adoption, checked at increasing adoption levels: a falling share shows the threshold going stale.
- Parameters:
- Return type:
- Returns:
One row per level with
levelandwithin.
Examples
>>> drift({0: [0.1, 0.2], 50: [0.1, 0.9]}, threshold=0.5)["within"].tolist() [1.0, 0.5]
- milia.evaluate.anomaly.match(flagged, truth)#
Precision and recall of a flagged set against a ground-truth set.
- Parameters:
- Return type:
- Returns:
flagged,truth,hits,precisionandrecall.
Examples
>>> match(["a", "b"], ["b", "c"]) {'flagged': 2, 'truth': 2, 'hits': 1, 'precision': 0.5, 'recall': 0.5}
- milia.evaluate.anomaly.balance(head, meters, imports, generation, demand, blocks, hours, sizes, kind='underreport', models=('head', 'components'), alpha=0.1, steps=48, step=0.5, contaminate=0)#
Test the energy balance of a transformer block by block, with one meter tampering in the held-out block.
For each held-out block (a week, say), the loss model (
lossmodel()) is fitted and the conformal threshold of the daily unexplained energy calibrated on the other blocks; the meter then tampers throughout the held-out block (tamper()). A size of zero measures the false-alarm rate. Holding out a whole block, rather than random days, tests a threshold calibrated on other seasons, as an operator’s would be. Withcontaminate, the meter also tampers, at the same size, in that many calibration blocks, as it would if it had tampered before the operator calibrated.- Parameters:
head (
ArrayLike) – Head power per step, whole days in time order.meters (
ArrayLike) – Summed meter readings per step.imports (
ArrayLike) – Summed import register per step.generation (
ArrayLike) – Summed generation per step.demand (
ArrayLike) – The tampering meter’s demand per step.blocks (
ArrayLike) – Block label of each step; each block is held out in turn.hours (
ArrayLike) – Hour of the day of each step.sizes (
Sequence[float]) – Sizes of the tampering, astamper()reads them.models (
Sequence[str]) – Loss models compared, fromLOSSMODELSandLOSSALTERNATIVES.alpha (
float) – Target false-alarm rate of the daily test.steps (
int) – Steps per day.step (
float) – Length of a step in hours.contaminate (
int) – Calibration blocks in which the meter also tampers (the first ones after the held-out block, cyclically).
- Return type:
- Returns:
One row per held-out block, size and model –
block,size,model,flagged(the share of the block’s days whose unexplained energy exceeds the threshold) anddays(their number).
Examples
>>> rng = np.random.default_rng(0) >>> days, steps = 28, 48 >>> imports = 20 + 10 * np.sin(np.linspace(0, 2 * np.pi * days, days * steps)) + rng.normal(0, 1, days * steps) >>> head = imports + 0.002 * imports**2 + rng.normal(0, 0.05, imports.size) >>> hours = np.tile(np.arange(steps) / 2, days) >>> blocks = np.repeat(np.arange(4), 7 * steps) >>> result = balance(head, imports, imports, np.zeros_like(imports), imports / 10, blocks, hours, [0.0, 0.5]) >>> result.groupby("size")["flagged"].mean().round(2).tolist() [0.07, 1.0]
- milia.evaluate.anomaly.novelty(detector, training, seed=0)#
A novelty detector fitted on normal rows, whose
score_samplesis higher for more normal rows.- Parameters:
- Return type:
- Returns:
The fitted detector.
- Raises:
ValueError – If
detectoris unknown.
Examples
>>> model = novelty("lof", np.random.default_rng(0).normal(size=(60, 3))) >>> model.score_samples(np.zeros((1, 3))).shape (1,)
- milia.evaluate.anomaly.behaviour(series, kind, training, count=5, faults=('fragmented', 'offhours'), representations=('magnitude', 'events', 'eventscircular'), seed=0, detector='iforest', pooled=False)#
Compare representations of a day for detecting behaviour faults, each entity against its own past.
For each entity, an Isolation Forest is fitted on its first
trainingdays in each representation, faults are injected intocountlater days that show the behaviour (behaviourfault()), and the AUC of the forest’s scores on the later days is recorded.- Parameters:
series (
ArrayLike) – Readings per entity, shape(entities, days, steps).kind (
str) –"export"or"charging".training (
int) – Number of leading days each entity’s detector is fitted on.count (
int) – Days corrupted per entity and fault.faults (
Sequence[str]) – Faults injected, fromBEHAVIOURFAULTS.representations (
Sequence[str]) – Representations compared, fromREPRESENTATIONS.seed (
int) – Seed of the injections and the forests.detector (
str) –"iforest"(an Isolation Forest, which isolates points by axis-parallel splits and needs no scaling) or"lof"(the local outlier factor on standardised features, which compares a point’s density with that of its neighbours); comparing the two separates the effect of the representation from that of the detector.pooled (
bool) – Fit one detector on the training days of every entity, rather than one per entity (a global against a local model).
- Return type:
- Returns:
One row per entity, fault and representation –
entity,fault,representationandauc. Entities with fewer than two corrupted days, or none left normal, are skipped.
Examples
>>> hours = np.arange(48) / 2 >>> rng = np.random.default_rng(0) >>> clear = np.clip(np.sin(np.pi * (hours - 7) / 10), 0, None) * 4.0 >>> series = 0.5 + rng.normal(0, 0.02, (2, 80, 48)) - rng.uniform(0.4, 1.0, (2, 80, 1)) * clear >>> result = behaviour(series, "export", training=50, count=5, faults=("offhours",)) >>> sorted(result["representation"].unique()) ['events', 'eventscircular', 'magnitude']
- milia.evaluate.anomaly.profilerows(cube, config)#
Profile features of each meter of a cube of daily readings, one row per meter.
- Parameters:
cube (
ArrayLike) – Readings, shape(meters, days, steps), from 1 January.config (
ProfileConfig) – The profile features to build.
- Return type:
- Returns:
The features, shape
(meters, features).
Examples
>>> config = profilefeatures.ProfileConfig(normalized=True, period="seasonal", hemisphere="south") >>> rows = profilerows(np.random.default_rng(0).uniform(0.1, 2.0, (3, 366, 48)), config) >>> rows.shape[0] 3
- milia.evaluate.anomaly.adoption(cube, labels, pv, sessions, shares, config, mix=('ev1', 'ev2', 'pv', 'pvev'), alpha=0.1, holdout=0.3, seed=0)#
Share of meters still described by a segmentation, and still trusted, as a growing share adopts PV or EVs.
Each meter is described by its profile features (
config). The segments’ centroids are fitted on most meters and two split-conformal thresholds calibrated on the rest: the distance to the nearest centroid, which defines the membership sets (ConformalSets), and the distance to the nearest meter of the reference, the trust radius (KNNRetriever). Adoption (adopt()) then moves the adopters away from the reference.- Parameters:
cube (
ArrayLike) – Readings in kW, shape(meters, days, steps), from 1 January.labels (
ArrayLike) – Each meter’s segment.pv (
ArrayLike) – PV output per kWp, shape(days, steps).sessions (
ArrayLike) – Recorded charging of a session pool in kW, shape(vehicles, days, steps).config (
ProfileConfig) – The profile features describing a meter.mix (
Sequence[str]) – Technologies drawn for the adopters, uniformly; one entry gives every adopter the same technology.alpha (
float) – Miscoverage level of both thresholds.holdout (
float) – Share of meters held out to calibrate.seed (
int) – Seed of the split, the adopters, their technologies and their arrays and vehicles.
- Return type:
- Returns:
One row per share –
share;described, the share of meters whose set holds a segment, near1 - alphabefore adoption;adopterdescribed, the same among adopters;single;empty; andtrusted, the share within the trust radius. A meter whose profile features are not finite (a day whose maximum is zero cannot be peak-normalised) lies within no threshold, so it is neither described nor trusted.
Examples
>>> rng = np.random.default_rng(0) >>> cube = rng.uniform(0.2, 2.0, (30, 366, 48)) >>> config = profilefeatures.ProfileConfig(normalized=True, period="seasonal", hemisphere="south") >>> pv, pool = np.zeros((366, 48)), np.zeros((2, 366, 48)) >>> result = adoption(cube, np.repeat([0, 1], 15), pv, pool, [0.0], config) >>> list(result.columns) ['share', 'described', 'adopterdescribed', 'single', 'empty', 'trusted']
- class milia.evaluate.anomaly.Monitoring(frame, limit, statistic, profiles, centroids, baseline)#
Bases:
objectThe outcome of
monitor().- Variables:
frame – One row per monitored meter:
meter,segment,technology(empty for a non-adopter),alarm(week of the first alarm,-1if none) anddelay(weeks from the onset week to the alarm, for an adopter alarmed after the onset).limit (
float) – The calibrated decision limit.statistic (
ndarray) – The CUSUM statistic of the monitored meters, inframe’s order.profiles (
ndarray) – Their weekly profiles with adoption, shape(meters, weeks, steps).centroids (
ndarray) – Each segment’s weekly centroid, shape(segments, weeks, steps), in sorted segment order.baseline (
int) – The baseline weeks.
Examples
>>> outcome = Monitoring(pd.DataFrame(), 5.0, np.zeros((0, 3)), np.zeros((0, 4, 48)), np.zeros((1, 4, 48)), 1) >>> outcome.limit 5.0
- milia.evaluate.anomaly.monitor(cube, labels, pv, sessions, share, onset, mix=('ev1', 'ev2', 'pv', 'pvev'), baseline=16, drift=0.5, alpha=0.1, holdout=0.3, seed=0, statistic='cusum', nominal=None)#
Detect, week by week, the meters that stop resembling their segment after adopting a technology.
Each segment’s centroid of a week is the mean weekly profile (
weekly()) of its meters before adoption, so the distance of a meter to its segment carries no seasonal trend common to the segment. A one-sided CUSUM of each meter’s weekly distance (paths()) is compared with a limit calibrated on held-out meters without adoption (pathlimit()), because real meters drift from their segment through the year and the CUSUM’s nominal limit would alarm many that do not adopt. A share of the other meters adopts on dayonset.- Parameters:
cube (
ArrayLike) – Readings in kW, shape(meters, days, steps), from 1 January.labels (
ArrayLike) – Each meter’s segment.pv (
ArrayLike) – PV output per kWp, shape(days, steps).sessions (
ArrayLike) – Recorded charging of a session pool in kW, shape(vehicles, days, steps).share (
float) – Share of the monitored meters that adopt.onset (
int) – Day on which the adopters install their technology.mix (
Sequence[str]) – Technologies drawn for the adopters, uniformly.baseline (
int) – Initial weeks that define each meter’s normal distance; must end beforeonset.drift (
float) – CUSUM allowance in standard deviations.alpha (
float) – Target share of non-adopting meters alarmed within the horizon.holdout (
float) – Share of meters held out, without adoption, to calibrate the limit.seed (
int) – Seed of the split, the adopters and their technologies.statistic (
str) –"cusum"(paths()) or"ewma"(ewmapaths()).nominal (
float|None) – A fixed decision limit to compare with the calibrated one; when given,framealso holdsnominalalarm, the week of the first alarm at that limit.
- Return type:
- Returns:
The
Monitoring.- Raises:
ValueError – If the baseline does not end before the onset.
Examples
>>> rng = np.random.default_rng(0) >>> cube = rng.uniform(0.2, 2.0, (20, 140, 48)) >>> pv, pool = np.zeros((140, 48)), np.full((2, 140, 48), 7.0) >>> result = monitor(cube, np.repeat([0, 1], 10), pv, pool, share=0.5, onset=84, mix=("ev2",), baseline=8) >>> len(result.frame), result.statistic.shape[1] (14, 12)
- milia.evaluate.anomaly.pointadjust(flagged, segments)#
Point adjustment: a true anomalous segment counts as found in full if any one of its readings is flagged.
The adjustment is common in benchmarks of time-series detectors and inflates their scores, since a single lucky flag in a long segment credits the whole segment; it is reported only to show that inflation.
- Parameters:
- Return type:
- Returns:
The adjusted flags.
Examples
>>> pointadjust([False, True, False, False], [(0, 3)]).tolist() [True, True, True, False]
- milia.evaluate.anomaly.faultaucs(results, entities=None)#
Mean per-entity AUC of normal against injected scores, for each injection separately.
- Parameters:
- Return type:
- Returns:
Injection name to mean AUC.
Examples
>>> results = pd.DataFrame({"entity": 0, "set": ["normal", "normal", "stuck", "stuck"], "score": [0, 1, 2, 3]}) >>> faultaucs(results) {'stuck': 1.0}