milia.anomaly.group#

Anomaly scores for a group of meters rather than a single one.

Besides the coincidence of export, the module holds the energy balance of a transformer: its head power \(P^{\mathrm{head}}_t\) against the sum of its meters \(\sum_i P^{(i)}_t\). The difference is the technical loss of the feeders when no meter is tampered with. Technical loss is resistive and grows with the square of the branch currents, so it is modelled by least squares on days known to be normal (lossdesign(), fitloss()), and the residual left by the model (unexplained()) is the evidence of non-technical loss. Two loss models are compared: "head", \(a P^2 + c\) in the head power alone, and "components", \(a P^2 + b D^2 + g G^2 + c\), which adds the summed import \(D\) and generation \(G\) from meters with separate registers. Import and export cancel in the head power but both drive branch currents, so the head-only model is misspecified once PV exports.

Module Attributes

POOLINGS

Ways of pooling the members of a group into one score.

MEMBERTESTS

flag the group if any member is flagged, or control the false discovery rate across members (Benjamini-Hochberg).

LOSSMODELS

the head power alone, or with the summed import and generation.

LOSSALTERNATIVES

a robust (Huber) fit of the import-and-export model, which down-weights contaminated steps, and gradient-boosted trees on the head power, import, generation and hour, which assume no physics.

TAMPERS

a register that records part of the demand, or an appliance connected ahead of the meter.

BYPASS

an evening heating or hot water load.

Functions

coincidence(members)

Depth of the group's coincident export: minus the minimum of the summed net load.

exportscore(net[, how, weights])

Daily coincident-export score of a group of meters: the largest pooled export of each day.

fitloss(design, loss)

Fit a technical-loss model by least squares on steps known to be normal.

imbalance(head, meters)

Share of energy at the feeder head not seen by its meters, per step.

locate(residual, readings[, method])

Score each meter by how well a hidden share of its demand explains a residual.

lossdesign(head[, imports, generation, model])

Regressors of a technical-loss model, quadratic in the powers that drive the branch currents.

lossmodel(model, head, imports, generation, ...)

Fit a technical-loss model on steps known to be normal and return its prediction of the loss.

memberflags(calibration, scores[, alpha, test])

Flag a group's days by testing its members one by one against their own calibration days.

perday(values[, steps, step])

Energy per day from power per step, for a series of whole days.

setfeatures(net[, window])

Fixed set-function features of a group's day: summed export, summed squared export, summed midday export.

tamper(head, meters, imports, demand, hours, ...)

Tamper with the energy balance of a transformer or phase.

unexplained(head, meters, design, coefficients)

Power the meters and the technical-loss model leave unexplained, \(u_t\), per step.

milia.anomaly.group.POOLINGS: tuple[str, ...] = ('sum', 'max', 'weighted')#

Ways of pooling the members of a group into one score.

milia.anomaly.group.MEMBERTESTS: tuple[str, ...] = ('any', 'bh')#

flag the group if any member is flagged, or control the false discovery rate across members (Benjamini-Hochberg).

Type:

Ways of testing members one by one

milia.anomaly.group.LOSSMODELS: tuple[str, ...] = ('head', 'components')#

the head power alone, or with the summed import and generation.

Type:

Technical-loss models

milia.anomaly.group.LOSSALTERNATIVES: tuple[str, ...] = ('huber', 'boosted')#

a robust (Huber) fit of the import-and-export model, which down-weights contaminated steps, and gradient-boosted trees on the head power, import, generation and hour, which assume no physics.

Type:

Data-driven alternatives to the physical loss models

milia.anomaly.group.TAMPERS: tuple[str, ...] = ('underreport', 'bypass')#

a register that records part of the demand, or an appliance connected ahead of the meter.

Type:

Kinds of tampering

milia.anomaly.group.BYPASS: tuple[int, int] = (17, 22)#

an evening heating or hot water load.

Type:

Hours of the day (start inclusive, end exclusive) in which a bypassed appliance runs

milia.anomaly.group.coincidence(members)#

Depth of the group’s coincident export: minus the minimum of the summed net load.

Higher means more anomalous: many members exporting at once on one transformer or feeder is what drives over-voltage.

Parameters:

members (ArrayLike) – Member net-load profiles, shape (members, steps).

Return type:

float

Returns:

The export depth of the group, in the units of the profiles.

Examples

>>> coincidence([[1.0, -2.0, 0.5], [0.5, -1.0, 0.5]])
3.0
milia.anomaly.group.imbalance(head, meters)#

Share of energy at the feeder head not seen by its meters, per step.

A rise in this apparent loss, beyond the technical-loss baseline, is the signature of non-technical loss such as meter tampering.

Parameters:
  • head (ArrayLike) – Feeder-head power per step.

  • meters (ArrayLike) – Summed metered power per step.

Return type:

ndarray

Returns:

(head - meters) / head per step.

Examples

>>> imbalance([10.0, 10.0], [9.0, 8.0]).round(2).tolist()
[0.1, 0.2]
milia.anomaly.group.lossdesign(head, imports=None, generation=None, model='components')#

Regressors of a technical-loss model, quadratic in the powers that drive the branch currents.

Parameters:
  • head (ArrayLike) – Head power per step.

  • imports (ArrayLike | None) – Summed import of the meters per step; required for "components".

  • generation (ArrayLike | None) – Summed generation of the meters per step; required for "components".

  • model (str) – One of LOSSMODELS.

Return type:

ndarray

Returns:

An (n, k) design matrix whose last column is the constant.

Raises:

ValueError – If model is unknown, or "components" lacks the import or generation.

Examples

>>> lossdesign([2.0], [3.0], [1.0]).tolist()
[[4.0, 9.0, 1.0, 1.0]]
>>> lossdesign([2.0], model="head").tolist()
[[4.0, 1.0]]
milia.anomaly.group.fitloss(design, loss)#

Fit a technical-loss model by least squares on steps known to be normal.

Parameters:
Return type:

ndarray

Returns:

The coefficients, in the order of the design’s columns.

Examples

>>> head = np.linspace(1.0, 10.0, 20)
>>> fitloss(lossdesign(head, model="head"), 0.01 * head**2 + 0.2).round(3).tolist()
[0.01, 0.2]
milia.anomaly.group.unexplained(head, meters, design, coefficients)#

Power the meters and the technical-loss model leave unexplained, \(u_t\), per step.

Parameters:
Return type:

ndarray

Returns:

\(u_t = P^{\mathrm{head}}_t - \sum_i P^{(i)}_t - \hat P^{\mathrm{loss}}_t\) per step.

Examples

>>> unexplained([10.0], [9.0], [[100.0, 1.0]], [0.005, 0.2]).round(3).tolist()
[0.3]
milia.anomaly.group.perday(values, steps=48, step=0.5)#

Energy per day from power per step, for a series of whole days.

Parameters:
  • values (ArrayLike) – Power per step, a whole number of days in time order.

  • steps (int) – Steps per day.

  • step (float) – Length of a step in hours.

Return type:

ndarray

Returns:

Energy per day, in the power’s unit times hours.

Examples

>>> perday(np.ones(96)).tolist()
[24.0, 24.0]
milia.anomaly.group.locate(residual, readings, method='pursuit')#

Score each meter by how well a hidden share of its demand explains a residual.

For each meter the least-squares share \(\beta_i = \langle d_i, r \rangle / \langle d_i, d_i \rangle\) of its demand \(d_i\) in the residual \(r\) is clipped at zero (a tampered register hides demand, it does not add it) and multiplied by the correlation of \(d_i\) and \(r\). It is one step of matching pursuit over the meters’ demand, and a larger score ranks the meter as more likely to be the source.

Two alternatives fit all the meters jointly, \(r = \sum_i \beta_i d_i + \varepsilon\) with \(\beta_i \ge 0\): "nnls" by non-negative least squares, and "lasso" by the non-negative lasso with its penalty chosen by BIC, which also prefers few non-zero shares, as a single tampering meter implies. Joint fits separate meters whose demand is correlated, which the one-at-a-time score confuses, but they become unstable when the meters outnumber the independent variation in the steps.

Parameters:
  • residual (ArrayLike) – The residual per step, length n.

  • readings (ArrayLike) – Each meter’s demand per step, shape (meters, n).

  • method (str) – "pursuit" (one step of matching pursuit), "nnls" or "lasso".

Return type:

ndarray

Returns:

One score per meter; meters whose demand is constant score zero.

Examples

>>> rng = np.random.default_rng(1)
>>> readings = rng.uniform(0.2, 3.0, size=(8, 336))
>>> int(np.argmax(locate(0.4 * readings[5] + rng.normal(0, 0.05, 336), readings)))
5
milia.anomaly.group.tamper(head, meters, imports, demand, hours, kind, size, window=(17, 22))#

Tamper with the energy balance of a transformer or phase.

Under-recording lowers the meters and the import register by a share of one meter’s demand and leaves the head unchanged. A bypass adds an appliance’s power to the head during window and leaves the meters unchanged; the extra technical loss its current causes is second order and neglected.

Parameters:
  • head (ArrayLike) – Head power per step.

  • meters (ArrayLike) – Summed meter readings per step.

  • imports (ArrayLike) – Summed import register per step.

  • demand (ArrayLike) – The tampering meter’s demand per step.

  • hours (ArrayLike) – Hour of the day of each step.

  • kind (str) – One of TAMPERS.

  • size (float) – For "underreport", the share of the demand hidden; for "bypass", the appliance’s power.

  • window (tuple[int, int]) – Hours of the day (start inclusive, end exclusive) in which a bypassed appliance runs.

Return type:

tuple[ndarray, ndarray, ndarray]

Returns:

The tampered head, meters and imports.

Raises:

ValueError – If kind is unknown.

Examples

>>> head, meters, imports = tamper([5.0, 5.0], [4.0, 4.0], [4.0, 4.0], [2.0, 2.0], [12, 18], "bypass", 1.0)
>>> head.tolist(), meters.tolist()
([5.0, 6.0], [4.0, 4.0])
milia.anomaly.group.exportscore(net, how='sum', weights=None)#

Daily coincident-export score of a group of meters: the largest pooled export of each day.

Each member’s export is \(x_{i,t} = \max(0, -P^{(i)}_t)\). The members are pooled at every step and the day’s score is the largest pooled value. "sum" pools \(\sum_i x_{i,t}\), the likelihood-ratio statistic of a shift shared by the members; "max" pools \(\max_i x_{i,t}\), the statistic of a single member’s excursion; "weighted" pools \(\sum_i w_i x_{i,t}\), which with \(w_i\) the resistance of member \(i\)’s path approximates the voltage rise \(\sum_i R_i P_i / V\) its export causes. Every pooling is permutation invariant, \(\rho(\sum_i \phi(x_i))\) or a maximum, so the order of the members carries no information.

Parameters:
  • net (ArrayLike) – Net load of the members, negative while exporting, shape (members, days, steps).

  • how (str) – One of POOLINGS.

  • weights (ArrayLike | None) – One weight per member, required for "weighted".

Return type:

ndarray

Returns:

One score per day, in the unit of the net load (times the weights’ unit for "weighted").

Raises:

ValueError – If how is unknown, or "weighted" lacks weights.

Examples

>>> net = np.array([[[1.0, -2.0]], [[0.5, -1.0]]])
>>> exportscore(net, "sum").tolist(), exportscore(net, "max").tolist()
([3.0], [2.0])
>>> exportscore(net, "weighted", weights=[0.1, 0.3]).round(2).tolist()
[0.5]
milia.anomaly.group.memberflags(calibration, scores, alpha=0.1, test='any')#

Flag a group’s days by testing its members one by one against their own calibration days.

Each member’s daily score is compared with that member’s calibration scores through a conformal p-value. With "any", a day is flagged when any member’s p-value is at most alpha: on a normal day about n * alpha of n members are flagged by chance, so the group is flagged on most days. With "bh", the members of a day are tested together with Benjamini-Hochberg, which controls the false discovery rate across members at the cost of power, so a shift shared by many members, each small, is rarely found.

Parameters:
  • calibration (ArrayLike) – Calibration scores of each member, shape (members, days), higher being more anomalous.

  • scores (ArrayLike) – Scores of each member on the days tested, shape (members, days).

  • alpha (float) – Level of each member’s test, or the false discovery rate for "bh".

  • test (str) – One of MEMBERTESTS.

Return type:

ndarray

Returns:

One boolean per tested day.

Raises:

ValueError – If test is unknown.

Examples

>>> calibration = np.tile(np.arange(1.0, 20.0), (3, 1))
>>> scores = np.array([[5.0, 30.0], [5.0, 5.0], [5.0, 5.0]])
>>> memberflags(calibration, scores).tolist()
[False, True]
milia.anomaly.group.setfeatures(net, window=slice(22, 28, None))#

Fixed set-function features of a group’s day: summed export, summed squared export, summed midday export.

Each feature is a sum over the members of a feature of each member, \(\sum_i \phi(x_i)\), so a classifier \(\rho\) fitted on them is a set function of the Deep Sets form, invariant to the order of the members.

Parameters:
  • net (ArrayLike) – Net load of the members, negative while exporting, shape (members, days, steps).

  • window (slice) – Steps of the midday window (11:00 to 14:00 for half-hourly days).

Return type:

ndarray

Returns:

Shape (days, 3) – the summed daily export, the summed squared export and the summed export in the window.

Examples

>>> net = -np.ones((2, 1, 48))
>>> setfeatures(net).tolist()
[[96.0, 96.0, 12.0]]
milia.anomaly.group.lossmodel(model, head, imports, generation, hours, loss)#

Fit a technical-loss model on steps known to be normal and return its prediction of the loss.

"head" and "components" are the physical models of lossdesign() fitted by least squares (fitloss()). "huber" fits the components model with the Huber loss, so steps whose residual is far from the model, such as those of a meter already tampering, weigh less. "boosted" fits gradient-boosted trees to the head power, import, generation and hour of the day: it needs no physics, but it also fits any systematic part of the residual, including tampering present in the training steps.

Parameters:
Return type:

Callable[[ArrayLike, ArrayLike, ArrayLike, ArrayLike], ndarray]

Returns:

A function of (head, imports, generation, hours) giving the predicted loss per step.

Raises:

ValueError – If model is unknown.

Examples

>>> rng = np.random.default_rng(0)
>>> head, imports = rng.uniform(5, 50, 500), rng.uniform(5, 50, 500)
>>> generation, hours = np.zeros(500), np.tile(np.arange(48) / 2, 11)[:500]
>>> predict = lossmodel("huber", head, imports, generation, hours, 0.001 * head**2 + 0.002 * imports**2)
>>> bool(np.allclose(predict([10.0], [10.0], [0.0], [12.0]), [0.3], atol=0.01))
True