milia.evaluate.anomaly#

Evaluating detectors by injection against each entity’s own normal baseline.

The protocol, for each entity (a meter with many days of readings):

  1. Split its days into training, calibration, held-out normal and an anomaly pool.

  2. Fit the detector on the training days and score every other day.

  3. Inject known anomalies into pool days and score them too.

  4. Calibrate a conformal threshold on the calibration days, then report the false-positive rate on held-out normal days, the recall on injected days, and the threshold-free AUC.

selfbaseline() produces every score once; summarize() turns them into metrics at any alpha without refitting, so an alpha sweep is cheap.

Functions

adoption(cube, labels, pv, sessions, shares, ...)

Share of meters still described by a segmentation, and still trusted, as a growing share adopts PV or EVs.

alphas(results[, levels])

summarize() at several alpha levels: target coverage against realised false-positive rate and recall.

auc(normal, anomalous)

ROC AUC of anomaly scores separating injected days (positive) from normal days.

balance(head, meters, imports, generation, ...)

Test the energy balance of a transformer block by block, with one meter tampering in the held-out block.

behaviour(series, kind, training[, count, ...])

Compare representations of a day for detecting behaviour faults, each entity against its own past.

drift(scores, threshold)

Share of scores still within a fixed threshold as some drift parameter grows.

faultaucs(results[, entities])

Mean per-entity AUC of normal against injected scores, for each injection separately.

features(days, index, featurize)

Feature matrix of the chosen days of one entity.

match(flagged, truth)

Precision and recall of a flagged set against a ground-truth set.

monitor(cube, labels, pv, sessions, share, onset)

Detect, week by week, the meters that stop resembling their segment after adopting a technology.

novelty(detector, training[, seed])

A novelty detector fitted on normal rows, whose score_samples is higher for more normal rows.

pointadjust(flagged, segments)

Point adjustment: a true anomalous segment counts as found in full if any one of its readings is flagged.

profilerows(cube, config)

Profile features of each meter of a cube of daily readings, one row per meter.

selfbaseline(entities, detector[, ...])

Score calibration, held-out normal and injected days for every entity.

split(n[, sizes, exclude, seed])

Shuffle day indices and split them into training, calibration, normal and anomaly-pool sets.

summarize(results[, alpha])

Detection metrics at one alpha: mean per-entity AUC, pooled false-positive rate and recall.

Classes

Monitoring(frame, limit, statistic, ...)

The outcome of monitor().

milia.evaluate.anomaly.split(n, sizes=(280, 40, 20), exclude=(), seed=1)#

Shuffle day indices and split them into training, calibration, normal and anomaly-pool sets.

Parameters:
  • n (int) – Number of days.

  • sizes (tuple[int, int, int]) – Training, calibration and held-out normal counts; the rest is the pool.

  • exclude (Sequence[int]) – Days kept out of every set (e.g. a known stress day).

  • seed (int) – Shuffle seed.

Return type:

dict[str, ndarray]

Returns:

{"train", "cal", "normal", "pool"} index arrays.

Raises:

ValueError – If the sizes leave no anomaly pool.

Examples

>>> sets = split(100, sizes=(50, 20, 10))
>>> [len(sets[k]) for k in ("train", "cal", "normal", "pool")]
[50, 20, 10, 20]
milia.evaluate.anomaly.features(days, index, featurize)#

Feature matrix of the chosen days of one entity.

Parameters:
  • days (ndarray) – The entity’s days, shape (days, steps).

  • index (ndarray) – Day indices.

  • featurize (Callable[[ndarray], ndarray]) – Maps one day’s readings to a feature vector.

Return type:

ndarray

Returns:

Array with one row per chosen day.

Examples

>>> features(np.arange(12.0).reshape(3, 4), np.array([0, 2]), lambda day: day[:2]).tolist()
[[0.0, 1.0], [8.0, 9.0]]
milia.evaluate.anomaly.auc(normal, anomalous)#

ROC AUC of anomaly scores separating injected days (positive) from normal days.

Parameters:
  • normal (ArrayLike) – Scores of normal days.

  • anomalous (ArrayLike) – Scores of injected days.

Return type:

float

Returns:

The AUC, 1 for perfect separation and 0.5 for chance.

Examples

>>> auc([0.1, 0.2], [0.9, 0.8])
1.0
milia.evaluate.anomaly.selfbaseline(entities, detector, featurize=None, injections=None, sets=None, count=5)#

Score calibration, held-out normal and injected days for every entity.

Each entity gets its own detector, fitted on its own training days, so the baseline is the entity’s normal behaviour rather than the population’s. Injection seeds are derived from the injection, entity, repeat and day, so results reproduce across processes.

Parameters:
Return type:

DataFrame

Returns:

One row per scored day – entity, set ("cal", "normal" or the injection name), day and score (higher = more anomalous).

Examples

>>> from milia.anomaly.ensemble import members
>>> from milia.data import profiles, synthetic
>>> cube = profiles.tensor(synthetic.generate(meters=2, days=60, steps=24).pivot)
>>> sets = split(60, sizes=(30, 15, 8))
>>> results = selfbaseline(cube, members(["ecod"])[0][1], sets=sets, count=2)
>>> sorted(results["set"].unique())
['cal', 'drift', 'dropout', 'normal', 'stuck']
milia.evaluate.anomaly.summarize(results, alpha=0.1)#

Detection metrics at one alpha: mean per-entity AUC, pooled false-positive rate and recall.

Each entity’s threshold is calibrated on its own "cal" scores, so the false-positive rate should not exceed alpha by more than sampling noise.

Parameters:
Return type:

dict[str, float]

Returns:

auc, fpr, recall and recall<kind> for each injection.

Examples

>>> from milia.anomaly.ensemble import members
>>> from milia.data import profiles, synthetic
>>> cube = profiles.tensor(synthetic.generate(meters=2, days=60, steps=24).pivot)
>>> sets = split(60, sizes=(30, 15, 8))
>>> results = selfbaseline(cube, members(["ecod"])[0][1], sets=sets, count=2)
>>> sorted(summarize(results, alpha=0.1))
['auc', 'fpr', 'recall', 'recalldrift', 'recalldropout', 'recallstuck']
milia.evaluate.anomaly.alphas(results, levels=(0.2, 0.1, 0.05))#

summarize() at several alpha levels: target coverage against realised false-positive rate and recall.

Parameters:
Return type:

DataFrame

Returns:

One row per level with alpha, coverage and the metrics.

Examples

>>> from milia.anomaly.ensemble import members
>>> from milia.data import profiles, synthetic
>>> cube = profiles.tensor(synthetic.generate(meters=2, days=60, steps=24).pivot)
>>> sets = split(60, sizes=(30, 15, 8))
>>> results = selfbaseline(cube, members(["ecod"])[0][1], sets=sets, count=2)
>>> alphas(results)["alpha"].tolist()
[0.2, 0.1, 0.05]
milia.evaluate.anomaly.drift(scores, threshold)#

Share of scores still within a fixed threshold as some drift parameter grows.

For example a retrieval trust radius calibrated before DER adoption, checked at increasing adoption levels: a falling share shows the threshold going stale.

Parameters:
  • scores (Mapping[Hashable, ArrayLike]) – Drift level to the scores observed at that level.

  • threshold (float) – The fixed threshold (scores at or below it are “within”).

Return type:

DataFrame

Returns:

One row per level with level and within.

Examples

>>> drift({0: [0.1, 0.2], 50: [0.1, 0.9]}, threshold=0.5)["within"].tolist()
[1.0, 0.5]
milia.evaluate.anomaly.match(flagged, truth)#

Precision and recall of a flagged set against a ground-truth set.

Parameters:
Return type:

dict[str, float]

Returns:

flagged, truth, hits, precision and recall.

Examples

>>> match(["a", "b"], ["b", "c"])
{'flagged': 2, 'truth': 2, 'hits': 1, 'precision': 0.5, 'recall': 0.5}
milia.evaluate.anomaly.balance(head, meters, imports, generation, demand, blocks, hours, sizes, kind='underreport', models=('head', 'components'), alpha=0.1, steps=48, step=0.5, contaminate=0)#

Test the energy balance of a transformer block by block, with one meter tampering in the held-out block.

For each held-out block (a week, say), the loss model (lossmodel()) is fitted and the conformal threshold of the daily unexplained energy calibrated on the other blocks; the meter then tampers throughout the held-out block (tamper()). A size of zero measures the false-alarm rate. Holding out a whole block, rather than random days, tests a threshold calibrated on other seasons, as an operator’s would be. With contaminate, the meter also tampers, at the same size, in that many calibration blocks, as it would if it had tampered before the operator calibrated.

Parameters:
  • head (ArrayLike) – Head power per step, whole days in time order.

  • meters (ArrayLike) – Summed meter readings per step.

  • imports (ArrayLike) – Summed import register per step.

  • generation (ArrayLike) – Summed generation per step.

  • demand (ArrayLike) – The tampering meter’s demand per step.

  • blocks (ArrayLike) – Block label of each step; each block is held out in turn.

  • hours (ArrayLike) – Hour of the day of each step.

  • sizes (Sequence[float]) – Sizes of the tampering, as tamper() reads them.

  • kind (str) – One of TAMPERS.

  • models (Sequence[str]) – Loss models compared, from LOSSMODELS and LOSSALTERNATIVES.

  • alpha (float) – Target false-alarm rate of the daily test.

  • steps (int) – Steps per day.

  • step (float) – Length of a step in hours.

  • contaminate (int) – Calibration blocks in which the meter also tampers (the first ones after the held-out block, cyclically).

Return type:

DataFrame

Returns:

One row per held-out block, size and model – block, size, model, flagged (the share of the block’s days whose unexplained energy exceeds the threshold) and days (their number).

Examples

>>> rng = np.random.default_rng(0)
>>> days, steps = 28, 48
>>> imports = 20 + 10 * np.sin(np.linspace(0, 2 * np.pi * days, days * steps)) + rng.normal(0, 1, days * steps)
>>> head = imports + 0.002 * imports**2 + rng.normal(0, 0.05, imports.size)
>>> hours = np.tile(np.arange(steps) / 2, days)
>>> blocks = np.repeat(np.arange(4), 7 * steps)
>>> result = balance(head, imports, imports, np.zeros_like(imports), imports / 10, blocks, hours, [0.0, 0.5])
>>> result.groupby("size")["flagged"].mean().round(2).tolist()
[0.07, 1.0]
milia.evaluate.anomaly.novelty(detector, training, seed=0)#

A novelty detector fitted on normal rows, whose score_samples is higher for more normal rows.

Parameters:
  • detector (str) – "iforest" (an Isolation Forest, which needs no scaling) or "lof" (the local outlier factor on standardised features).

  • training (ArrayLike) – Normal rows.

  • seed (int) – Seed of the Isolation Forest.

Return type:

BaseEstimator

Returns:

The fitted detector.

Raises:

ValueError – If detector is unknown.

Examples

>>> model = novelty("lof", np.random.default_rng(0).normal(size=(60, 3)))
>>> model.score_samples(np.zeros((1, 3))).shape
(1,)
milia.evaluate.anomaly.behaviour(series, kind, training, count=5, faults=('fragmented', 'offhours'), representations=('magnitude', 'events', 'eventscircular'), seed=0, detector='iforest', pooled=False)#

Compare representations of a day for detecting behaviour faults, each entity against its own past.

For each entity, an Isolation Forest is fitted on its first training days in each representation, faults are injected into count later days that show the behaviour (behaviourfault()), and the AUC of the forest’s scores on the later days is recorded.

Parameters:
  • series (ArrayLike) – Readings per entity, shape (entities, days, steps).

  • kind (str) – "export" or "charging".

  • training (int) – Number of leading days each entity’s detector is fitted on.

  • count (int) – Days corrupted per entity and fault.

  • faults (Sequence[str]) – Faults injected, from BEHAVIOURFAULTS.

  • representations (Sequence[str]) – Representations compared, from REPRESENTATIONS.

  • seed (int) – Seed of the injections and the forests.

  • detector (str) – "iforest" (an Isolation Forest, which isolates points by axis-parallel splits and needs no scaling) or "lof" (the local outlier factor on standardised features, which compares a point’s density with that of its neighbours); comparing the two separates the effect of the representation from that of the detector.

  • pooled (bool) – Fit one detector on the training days of every entity, rather than one per entity (a global against a local model).

Return type:

DataFrame

Returns:

One row per entity, fault and representation – entity, fault, representation and auc. Entities with fewer than two corrupted days, or none left normal, are skipped.

Examples

>>> hours = np.arange(48) / 2
>>> rng = np.random.default_rng(0)
>>> clear = np.clip(np.sin(np.pi * (hours - 7) / 10), 0, None) * 4.0
>>> series = 0.5 + rng.normal(0, 0.02, (2, 80, 48)) - rng.uniform(0.4, 1.0, (2, 80, 1)) * clear
>>> result = behaviour(series, "export", training=50, count=5, faults=("offhours",))
>>> sorted(result["representation"].unique())
['events', 'eventscircular', 'magnitude']
milia.evaluate.anomaly.profilerows(cube, config)#

Profile features of each meter of a cube of daily readings, one row per meter.

Parameters:
  • cube (ArrayLike) – Readings, shape (meters, days, steps), from 1 January.

  • config (ProfileConfig) – The profile features to build.

Return type:

ndarray

Returns:

The features, shape (meters, features).

Examples

>>> config = profilefeatures.ProfileConfig(normalized=True, period="seasonal", hemisphere="south")
>>> rows = profilerows(np.random.default_rng(0).uniform(0.1, 2.0, (3, 366, 48)), config)
>>> rows.shape[0]
3
milia.evaluate.anomaly.adoption(cube, labels, pv, sessions, shares, config, mix=('ev1', 'ev2', 'pv', 'pvev'), alpha=0.1, holdout=0.3, seed=0)#

Share of meters still described by a segmentation, and still trusted, as a growing share adopts PV or EVs.

Each meter is described by its profile features (config). The segments’ centroids are fitted on most meters and two split-conformal thresholds calibrated on the rest: the distance to the nearest centroid, which defines the membership sets (ConformalSets), and the distance to the nearest meter of the reference, the trust radius (KNNRetriever). Adoption (adopt()) then moves the adopters away from the reference.

Parameters:
  • cube (ArrayLike) – Readings in kW, shape (meters, days, steps), from 1 January.

  • labels (ArrayLike) – Each meter’s segment.

  • pv (ArrayLike) – PV output per kWp, shape (days, steps).

  • sessions (ArrayLike) – Recorded charging of a session pool in kW, shape (vehicles, days, steps).

  • shares (Sequence[float]) – Shares of meters that adopt.

  • config (ProfileConfig) – The profile features describing a meter.

  • mix (Sequence[str]) – Technologies drawn for the adopters, uniformly; one entry gives every adopter the same technology.

  • alpha (float) – Miscoverage level of both thresholds.

  • holdout (float) – Share of meters held out to calibrate.

  • seed (int) – Seed of the split, the adopters, their technologies and their arrays and vehicles.

Return type:

DataFrame

Returns:

One row per share – share; described, the share of meters whose set holds a segment, near 1 - alpha before adoption; adopterdescribed, the same among adopters; single; empty; and trusted, the share within the trust radius. A meter whose profile features are not finite (a day whose maximum is zero cannot be peak-normalised) lies within no threshold, so it is neither described nor trusted.

Examples

>>> rng = np.random.default_rng(0)
>>> cube = rng.uniform(0.2, 2.0, (30, 366, 48))
>>> config = profilefeatures.ProfileConfig(normalized=True, period="seasonal", hemisphere="south")
>>> pv, pool = np.zeros((366, 48)), np.zeros((2, 366, 48))
>>> result = adoption(cube, np.repeat([0, 1], 15), pv, pool, [0.0], config)
>>> list(result.columns)
['share', 'described', 'adopterdescribed', 'single', 'empty', 'trusted']
class milia.evaluate.anomaly.Monitoring(frame, limit, statistic, profiles, centroids, baseline)#

Bases: object

The outcome of monitor().

Variables:
  • frame – One row per monitored meter: meter, segment, technology (empty for a non-adopter), alarm (week of the first alarm, -1 if none) and delay (weeks from the onset week to the alarm, for an adopter alarmed after the onset).

  • limit (float) – The calibrated decision limit.

  • statistic (ndarray) – The CUSUM statistic of the monitored meters, in frame’s order.

  • profiles (ndarray) – Their weekly profiles with adoption, shape (meters, weeks, steps).

  • centroids (ndarray) – Each segment’s weekly centroid, shape (segments, weeks, steps), in sorted segment order.

  • baseline (int) – The baseline weeks.

Examples

>>> outcome = Monitoring(pd.DataFrame(), 5.0, np.zeros((0, 3)), np.zeros((0, 4, 48)), np.zeros((1, 4, 48)), 1)
>>> outcome.limit
5.0
milia.evaluate.anomaly.monitor(cube, labels, pv, sessions, share, onset, mix=('ev1', 'ev2', 'pv', 'pvev'), baseline=16, drift=0.5, alpha=0.1, holdout=0.3, seed=0, statistic='cusum', nominal=None)#

Detect, week by week, the meters that stop resembling their segment after adopting a technology.

Each segment’s centroid of a week is the mean weekly profile (weekly()) of its meters before adoption, so the distance of a meter to its segment carries no seasonal trend common to the segment. A one-sided CUSUM of each meter’s weekly distance (paths()) is compared with a limit calibrated on held-out meters without adoption (pathlimit()), because real meters drift from their segment through the year and the CUSUM’s nominal limit would alarm many that do not adopt. A share of the other meters adopts on day onset.

Parameters:
  • cube (ArrayLike) – Readings in kW, shape (meters, days, steps), from 1 January.

  • labels (ArrayLike) – Each meter’s segment.

  • pv (ArrayLike) – PV output per kWp, shape (days, steps).

  • sessions (ArrayLike) – Recorded charging of a session pool in kW, shape (vehicles, days, steps).

  • share (float) – Share of the monitored meters that adopt.

  • onset (int) – Day on which the adopters install their technology.

  • mix (Sequence[str]) – Technologies drawn for the adopters, uniformly.

  • baseline (int) – Initial weeks that define each meter’s normal distance; must end before onset.

  • drift (float) – CUSUM allowance in standard deviations.

  • alpha (float) – Target share of non-adopting meters alarmed within the horizon.

  • holdout (float) – Share of meters held out, without adoption, to calibrate the limit.

  • seed (int) – Seed of the split, the adopters and their technologies.

  • statistic (str) – "cusum" (paths()) or "ewma" (ewmapaths()).

  • nominal (float | None) – A fixed decision limit to compare with the calibrated one; when given, frame also holds nominalalarm, the week of the first alarm at that limit.

Return type:

Monitoring

Returns:

The Monitoring.

Raises:

ValueError – If the baseline does not end before the onset.

Examples

>>> rng = np.random.default_rng(0)
>>> cube = rng.uniform(0.2, 2.0, (20, 140, 48))
>>> pv, pool = np.zeros((140, 48)), np.full((2, 140, 48), 7.0)
>>> result = monitor(cube, np.repeat([0, 1], 10), pv, pool, share=0.5, onset=84, mix=("ev2",), baseline=8)
>>> len(result.frame), result.statistic.shape[1]
(14, 12)
milia.evaluate.anomaly.pointadjust(flagged, segments)#

Point adjustment: a true anomalous segment counts as found in full if any one of its readings is flagged.

The adjustment is common in benchmarks of time-series detectors and inflates their scores, since a single lucky flag in a long segment credits the whole segment; it is reported only to show that inflation.

Parameters:
  • flagged (ArrayLike) – One boolean per reading.

  • segments (Sequence[tuple[int, int]]) – (start, stop) indices of each true anomalous segment.

Return type:

ndarray

Returns:

The adjusted flags.

Examples

>>> pointadjust([False, True, False, False], [(0, 3)]).tolist()
[True, True, True, False]
milia.evaluate.anomaly.faultaucs(results, entities=None)#

Mean per-entity AUC of normal against injected scores, for each injection separately.

Parameters:
Return type:

dict[str, float]

Returns:

Injection name to mean AUC.

Examples

>>> results = pd.DataFrame({"entity": 0, "set": ["normal", "normal", "stuck", "stuck"], "score": [0, 1, 2, 3]})
>>> faultaucs(results)
{'stuck': 1.0}