milia.anomaly.group#
Anomaly scores for a group of meters rather than a single one.
Besides the coincidence of export, the module holds the energy balance of a transformer: its head power
\(P^{\mathrm{head}}_t\) against the sum of its meters \(\sum_i P^{(i)}_t\). The difference is the technical
loss of the feeders when no meter is tampered with. Technical loss is resistive and grows with the square of the
branch currents, so it is modelled by least squares on days known to be normal (lossdesign(),
fitloss()), and the residual left by the model (unexplained()) is the evidence of non-technical loss.
Two loss models are compared: "head", \(a P^2 + c\) in the head power alone, and "components",
\(a P^2 + b D^2 + g G^2 + c\), which adds the summed import \(D\) and generation \(G\) from meters with
separate registers. Import and export cancel in the head power but both drive branch currents, so the head-only
model is misspecified once PV exports.
Module Attributes
Ways of pooling the members of a group into one score. |
|
flag the group if any member is flagged, or control the false discovery rate across members (Benjamini-Hochberg). |
|
the head power alone, or with the summed import and generation. |
|
a robust (Huber) fit of the import-and-export model, which down-weights contaminated steps, and gradient-boosted trees on the head power, import, generation and hour, which assume no physics. |
|
a register that records part of the demand, or an appliance connected ahead of the meter. |
|
an evening heating or hot water load. |
Functions
|
Depth of the group's coincident export: minus the minimum of the summed net load. |
|
Daily coincident-export score of a group of meters: the largest pooled export of each day. |
|
Fit a technical-loss model by least squares on steps known to be normal. |
|
Share of energy at the feeder head not seen by its meters, per step. |
|
Score each meter by how well a hidden share of its demand explains a residual. |
|
Regressors of a technical-loss model, quadratic in the powers that drive the branch currents. |
|
Fit a technical-loss model on steps known to be normal and return its prediction of the loss. |
|
Flag a group's days by testing its members one by one against their own calibration days. |
|
Energy per day from power per step, for a series of whole days. |
|
Fixed set-function features of a group's day: summed export, summed squared export, summed midday export. |
|
Tamper with the energy balance of a transformer or phase. |
|
Power the meters and the technical-loss model leave unexplained, \(u_t\), per step. |
- milia.anomaly.group.POOLINGS: tuple[str, ...] = ('sum', 'max', 'weighted')#
Ways of pooling the members of a group into one score.
- milia.anomaly.group.MEMBERTESTS: tuple[str, ...] = ('any', 'bh')#
flag the group if any member is flagged, or control the false discovery rate across members (Benjamini-Hochberg).
- Type:
Ways of testing members one by one
- milia.anomaly.group.LOSSMODELS: tuple[str, ...] = ('head', 'components')#
the head power alone, or with the summed import and generation.
- Type:
Technical-loss models
- milia.anomaly.group.LOSSALTERNATIVES: tuple[str, ...] = ('huber', 'boosted')#
a robust (Huber) fit of the import-and-export model, which down-weights contaminated steps, and gradient-boosted trees on the head power, import, generation and hour, which assume no physics.
- Type:
Data-driven alternatives to the physical loss models
- milia.anomaly.group.TAMPERS: tuple[str, ...] = ('underreport', 'bypass')#
a register that records part of the demand, or an appliance connected ahead of the meter.
- Type:
Kinds of tampering
- milia.anomaly.group.BYPASS: tuple[int, int] = (17, 22)#
an evening heating or hot water load.
- Type:
Hours of the day (start inclusive, end exclusive) in which a bypassed appliance runs
- milia.anomaly.group.coincidence(members)#
Depth of the group’s coincident export: minus the minimum of the summed net load.
Higher means more anomalous: many members exporting at once on one transformer or feeder is what drives over-voltage.
- Parameters:
members (
ArrayLike) – Member net-load profiles, shape(members, steps).- Return type:
- Returns:
The export depth of the group, in the units of the profiles.
Examples
>>> coincidence([[1.0, -2.0, 0.5], [0.5, -1.0, 0.5]]) 3.0
- milia.anomaly.group.imbalance(head, meters)#
Share of energy at the feeder head not seen by its meters, per step.
A rise in this apparent loss, beyond the technical-loss baseline, is the signature of non-technical loss such as meter tampering.
- Parameters:
- Return type:
- Returns:
(head - meters) / headper step.
Examples
>>> imbalance([10.0, 10.0], [9.0, 8.0]).round(2).tolist() [0.1, 0.2]
- milia.anomaly.group.lossdesign(head, imports=None, generation=None, model='components')#
Regressors of a technical-loss model, quadratic in the powers that drive the branch currents.
- Parameters:
- Return type:
- Returns:
An
(n, k)design matrix whose last column is the constant.- Raises:
ValueError – If
modelis unknown, or"components"lacks the import or generation.
Examples
>>> lossdesign([2.0], [3.0], [1.0]).tolist() [[4.0, 9.0, 1.0, 1.0]] >>> lossdesign([2.0], model="head").tolist() [[4.0, 1.0]]
- milia.anomaly.group.fitloss(design, loss)#
Fit a technical-loss model by least squares on steps known to be normal.
- Parameters:
design (
ArrayLike) – Regressors fromlossdesign().loss (
ArrayLike) – The observed loss per step, head power minus the sum of the meters.
- Return type:
- Returns:
The coefficients, in the order of the design’s columns.
Examples
>>> head = np.linspace(1.0, 10.0, 20) >>> fitloss(lossdesign(head, model="head"), 0.01 * head**2 + 0.2).round(3).tolist() [0.01, 0.2]
- milia.anomaly.group.unexplained(head, meters, design, coefficients)#
Power the meters and the technical-loss model leave unexplained, \(u_t\), per step.
- Parameters:
- Return type:
- Returns:
\(u_t = P^{\mathrm{head}}_t - \sum_i P^{(i)}_t - \hat P^{\mathrm{loss}}_t\) per step.
Examples
>>> unexplained([10.0], [9.0], [[100.0, 1.0]], [0.005, 0.2]).round(3).tolist() [0.3]
- milia.anomaly.group.perday(values, steps=48, step=0.5)#
Energy per day from power per step, for a series of whole days.
- Parameters:
- Return type:
- Returns:
Energy per day, in the power’s unit times hours.
Examples
>>> perday(np.ones(96)).tolist() [24.0, 24.0]
- milia.anomaly.group.locate(residual, readings, method='pursuit')#
Score each meter by how well a hidden share of its demand explains a residual.
For each meter the least-squares share \(\beta_i = \langle d_i, r \rangle / \langle d_i, d_i \rangle\) of its demand \(d_i\) in the residual \(r\) is clipped at zero (a tampered register hides demand, it does not add it) and multiplied by the correlation of \(d_i\) and \(r\). It is one step of matching pursuit over the meters’ demand, and a larger score ranks the meter as more likely to be the source.
Two alternatives fit all the meters jointly, \(r = \sum_i \beta_i d_i + \varepsilon\) with \(\beta_i \ge 0\):
"nnls"by non-negative least squares, and"lasso"by the non-negative lasso with its penalty chosen by BIC, which also prefers few non-zero shares, as a single tampering meter implies. Joint fits separate meters whose demand is correlated, which the one-at-a-time score confuses, but they become unstable when the meters outnumber the independent variation in the steps.- Parameters:
- Return type:
- Returns:
One score per meter; meters whose demand is constant score zero.
Examples
>>> rng = np.random.default_rng(1) >>> readings = rng.uniform(0.2, 3.0, size=(8, 336)) >>> int(np.argmax(locate(0.4 * readings[5] + rng.normal(0, 0.05, 336), readings))) 5
- milia.anomaly.group.tamper(head, meters, imports, demand, hours, kind, size, window=(17, 22))#
Tamper with the energy balance of a transformer or phase.
Under-recording lowers the meters and the import register by a share of one meter’s demand and leaves the head unchanged. A bypass adds an appliance’s power to the head during
windowand leaves the meters unchanged; the extra technical loss its current causes is second order and neglected.- Parameters:
head (
ArrayLike) – Head power per step.meters (
ArrayLike) – Summed meter readings per step.imports (
ArrayLike) – Summed import register per step.demand (
ArrayLike) – The tampering meter’s demand per step.hours (
ArrayLike) – Hour of the day of each step.size (
float) – For"underreport", the share of the demand hidden; for"bypass", the appliance’s power.window (
tuple[int,int]) – Hours of the day (start inclusive, end exclusive) in which a bypassed appliance runs.
- Return type:
- Returns:
The tampered head, meters and imports.
- Raises:
ValueError – If
kindis unknown.
Examples
>>> head, meters, imports = tamper([5.0, 5.0], [4.0, 4.0], [4.0, 4.0], [2.0, 2.0], [12, 18], "bypass", 1.0) >>> head.tolist(), meters.tolist() ([5.0, 6.0], [4.0, 4.0])
- milia.anomaly.group.exportscore(net, how='sum', weights=None)#
Daily coincident-export score of a group of meters: the largest pooled export of each day.
Each member’s export is \(x_{i,t} = \max(0, -P^{(i)}_t)\). The members are pooled at every step and the day’s score is the largest pooled value.
"sum"pools \(\sum_i x_{i,t}\), the likelihood-ratio statistic of a shift shared by the members;"max"pools \(\max_i x_{i,t}\), the statistic of a single member’s excursion;"weighted"pools \(\sum_i w_i x_{i,t}\), which with \(w_i\) the resistance of member \(i\)’s path approximates the voltage rise \(\sum_i R_i P_i / V\) its export causes. Every pooling is permutation invariant, \(\rho(\sum_i \phi(x_i))\) or a maximum, so the order of the members carries no information.- Parameters:
- Return type:
- Returns:
One score per day, in the unit of the net load (times the weights’ unit for
"weighted").- Raises:
ValueError – If
howis unknown, or"weighted"lacks weights.
Examples
>>> net = np.array([[[1.0, -2.0]], [[0.5, -1.0]]]) >>> exportscore(net, "sum").tolist(), exportscore(net, "max").tolist() ([3.0], [2.0]) >>> exportscore(net, "weighted", weights=[0.1, 0.3]).round(2).tolist() [0.5]
- milia.anomaly.group.memberflags(calibration, scores, alpha=0.1, test='any')#
Flag a group’s days by testing its members one by one against their own calibration days.
Each member’s daily score is compared with that member’s calibration scores through a conformal p-value. With
"any", a day is flagged when any member’s p-value is at mostalpha: on a normal day aboutn * alphaofnmembers are flagged by chance, so the group is flagged on most days. With"bh", the members of a day are tested together with Benjamini-Hochberg, which controls the false discovery rate across members at the cost of power, so a shift shared by many members, each small, is rarely found.- Parameters:
calibration (
ArrayLike) – Calibration scores of each member, shape(members, days), higher being more anomalous.scores (
ArrayLike) – Scores of each member on the days tested, shape(members, days).alpha (
float) – Level of each member’s test, or the false discovery rate for"bh".test (
str) – One ofMEMBERTESTS.
- Return type:
- Returns:
One boolean per tested day.
- Raises:
ValueError – If
testis unknown.
Examples
>>> calibration = np.tile(np.arange(1.0, 20.0), (3, 1)) >>> scores = np.array([[5.0, 30.0], [5.0, 5.0], [5.0, 5.0]]) >>> memberflags(calibration, scores).tolist() [False, True]
- milia.anomaly.group.setfeatures(net, window=slice(22, 28, None))#
Fixed set-function features of a group’s day: summed export, summed squared export, summed midday export.
Each feature is a sum over the members of a feature of each member, \(\sum_i \phi(x_i)\), so a classifier \(\rho\) fitted on them is a set function of the Deep Sets form, invariant to the order of the members.
- Parameters:
- Return type:
- Returns:
Shape
(days, 3)– the summed daily export, the summed squared export and the summed export in the window.
Examples
>>> net = -np.ones((2, 1, 48)) >>> setfeatures(net).tolist() [[96.0, 96.0, 12.0]]
- milia.anomaly.group.lossmodel(model, head, imports, generation, hours, loss)#
Fit a technical-loss model on steps known to be normal and return its prediction of the loss.
"head"and"components"are the physical models oflossdesign()fitted by least squares (fitloss())."huber"fits the components model with the Huber loss, so steps whose residual is far from the model, such as those of a meter already tampering, weigh less."boosted"fits gradient-boosted trees to the head power, import, generation and hour of the day: it needs no physics, but it also fits any systematic part of the residual, including tampering present in the training steps.- Parameters:
model (
str) – One ofLOSSMODELSorLOSSALTERNATIVES.head (
ArrayLike) – Head power per training step.imports (
ArrayLike) – Summed import per training step.generation (
ArrayLike) – Summed generation per training step.hours (
ArrayLike) – Hour of the day of each training step.loss (
ArrayLike) – The observed loss per training step.
- Return type:
Callable[[ArrayLike,ArrayLike,ArrayLike,ArrayLike],ndarray]- Returns:
A function of
(head, imports, generation, hours)giving the predicted loss per step.- Raises:
ValueError – If
modelis unknown.
Examples
>>> rng = np.random.default_rng(0) >>> head, imports = rng.uniform(5, 50, 500), rng.uniform(5, 50, 500) >>> generation, hours = np.zeros(500), np.tile(np.arange(48) / 2, 11)[:500] >>> predict = lossmodel("huber", head, imports, generation, hours, 0.001 * head**2 + 0.002 * imports**2) >>> bool(np.allclose(predict([10.0], [10.0], [0.0], [12.0]), [0.3], atol=0.01)) True