Foundations of Retrieval, Ranking and Recommendation

A Distribution System Operator (DSO) answers questions about individual customers and transformers every day, and most of them have the same form. A request to connect rooftop Photovoltaic (PV) arrives, and the operator must judge whether the network can host it, usually before any study has been run. The transformers of a network must be examined in some order, because the capacity to examine all of them at once does not exist. A transformer that violates its limits needs a remedy, and several are available, from settings on the customers’ inverters to a larger transformer. Part 5 showed that groups of customers answer none of these questions well. Archetypes of the shape of demand accounted for 5.3% of the variance of a customer’s contribution to the transformer peak, while the transformer itself accounted for 69.3% of the variance of the worst voltage. The answer has to be given for the individual case.

Questions about individual cases are not new to machine learning. Information retrieval finds the items most relevant to a query and orders them [1]. Recommender systems choose items for users of whom little may be known [2], [3]. Case-based reasoning solves a new problem by reusing the solution of a similar one [4], and causal inference asks what an action changes rather than what accompanies it [5], [6]. Each field supplies a formal statement of one step of a decision and a way to evaluate it. None was designed for a distribution network, however, and none assumes what this part can assume, namely a simulator that returns the outcome of every action for every case.

This chapter brings the four fields together for the Low Voltage (LV) network. It frames the decisions of a DSO as a chain of retrieval, ranking and recommendation (Figure 6.1), in which each step consumes the output of the one before. Retrieval estimates the outcome of a new case from similar ones, and raises the question of when such an estimate can be trusted. Ranking orders the cases by their estimated outcome, and raises the question of how an order is judged and learned. Recommendation assigns each case a mitigation measure, and raises the question of how the effect and the cost of a mitigation measure are estimated from records in which each transformer received only one. The chapter reviews the literature on each step, states the steps in one notation, and answers the three questions in turn on a synthetic case base whose outcome under every mitigation measure is known. Chapter 2 then applies retrieval and ranking to the AusNet networks, and Chapter 3 applies recommendation.

Figure 6.1: Schematic of the decision chain of Part 6. (a) A query, here a request to connect rooftop PV, described by its features. (b) Retrieval of the nearest simulated cases within a trust radius. (c) Ranking by estimated risk. (d) Recommendation of the action of least estimated cost. Slate ticks mark the simulated outcomes against which each step is scored.

Retrieval, Ranking and Recommendation in the Literature

The review follows the chain of Figure 6.1, from the retrieval of similar cases to the choice of an action, and closes with the work that has applied these ideas to energy networks.

Retrieval in its simplest form answers a query with the items nearest to it under a measure of similarity. The nearest-neighbour rule of Cover and Hart is its statistical foundation: as the number of stored cases grows, the error of the rule approaches a small multiple of the least achievable error [7]. A retrieval returns several items, and information retrieval therefore evaluates the answer as a ranked list, crediting relevant items more the nearer they are placed to the top [1]. Järvelin and Kekäläinen extended this credit to graded relevance through the discounted cumulative gain [8], now the standard measure of a ranked list.

Once a list is judged by its order, the order itself can be learned. Learning to rank fits a score whose order matches the relevance of the items, rather than the relevance of each item, and Liu classified its methods as pointwise, pairwise and listwise, according to whether the loss is computed on single items, on pairs or on whole lists [9]. RankNet introduced the pairwise logistic loss on the difference of two scores [10]. LambdaRank then weighted the gradient of each pair by the change in a listwise measure that swapping the pair would cause [11], and LambdaMART fitted these gradients with gradient-boosted trees [12], [13]. Chen and colleagues later showed that the pairwise losses of these rankers bound one minus the NDCG of the list, so that minimising them improves the measure by which a list is judged [14].

Recommender systems extend ranking from the items relevant to a query to the items suited to a user. They are commonly divided into content-based methods, which recommend from the features of an item, collaborative methods, which recommend from the behaviour of similar users, and hybrids of the two [2], [15], [16]. Matrix factorisation, the dominant collaborative method, learns latent factors from a large and sparse matrix of interactions [17], and a user or item without history therefore defeats it, a difficulty known as the cold start [18]. Recommendations are usually evaluated offline on logs of past interactions, by the accuracy of predicted ratings and of ranked lists [19], [20]. A log, however, records only the response to items that were shown, and unbiased offline evaluation reweights it by the probability with which each item was shown [21], [22].

The limitation of the log is a causal one, and it is shared by the oldest method of recommending a remedy. Case-based reasoning, as formalised by Aamodt and Plaza, solves a problem in a cycle of four steps: retrieve the most similar past case, reuse its solution, revise the solution by testing it, and retain the solved case [4]. The cycle assumes that the retained cases record what each solution achieved. Causal inference states what such a record can and cannot reveal. Rubin defined the effect of a treatment through its potential outcomes, of which only one can be observed for each unit [5], and Holland called this the fundamental problem of causal inference [23]. Rosenbaum and Rubin stated the conditions under which an effect can nevertheless be estimated from observational records [24], and Künzel, Sekhon, Bickel and Yu combined ordinary regressors into estimators of the effect on each unit, the meta-learners [25].

Two concerns cut across all three steps, the reliability of an estimate and the fairness of a list. Conformal prediction addresses the first: it turns any score into a set or an interval with a guaranteed coverage under exchangeability, without assumptions on the model [26], [27], [28], [29]. It was used for cluster membership in Part 5, Chapter 3 (Equation 3.2), and for intervals on electricity prices by Kath and Ziel [30]. Fairness of exposure addresses the second, by constraining how the attention of a ranked list, which falls with position, is shared among groups [31]; maximal marginal relevance similarly trades relevance against redundancy within a list [32].

In the energy domain these ideas have been applied separately. Qammar and colleagues estimated hosting capacity by clustering [33], Andagoya-Alba and colleagues estimated it without a network model [34], and Duran and Monti recommended where PV should be adopted [35]. Luo and colleagues built a recommender on disaggregated loads [36], Gangwar and Shaik used nearest neighbours for protection [37], and Parvez, Hu and Chen ranked floods of similar alarms for operators [38]. Each work addresses one step, and none evaluates against the outcome of every action. Assembling the steps into one chain first requires a common statement of the problem, which the next section provides.

Problem Formulation and Notation

The decisions of this part share one structure. A query \(q\), which may be a connection request, a customer, a transformer or a violation event, is described by a vector of features \(\mathbf{x}_q\). A case base \(\mathcal{D} = \{(\mathbf{x}_j, y_j)\}\) holds cases whose outcomes are known, and a set \(\mathcal{A}\) holds the actions available, including no action, written \(a = 0\). The potential outcome \(Y_q(a)\) is the outcome of query \(q\) had action \(a\) been taken. In this chapter it is the number of hours the voltage spends outside the statutory band. Each action also has a cost \(C_q(a)\), defined in Equation 6.26.

In this notation the three steps of the chain differ only in what they produce. Retrieval produces an estimate \(\hat y_q\) of the outcome of a new query from the cases nearest to it. Ranking produces an order of a set of queries from a score such as \(\hat y_q\). Recommendation produces for each query the action of least estimated cost,

\[ \hat a_q = \arg\min_{a \in \mathcal{A}} \hat C_q(a). \tag{6.1}\]

The three outputs are evaluated against different quantities, and here the simulator becomes essential. The field observes only \(Y_q(\hat a_q)\), the outcome of the action taken. The simulator observes \(Y_q(a)\) for every \(a\), so that each step can be scored exactly. Retrieval is scored by its error, ranking by the order of the true outcomes, and recommendation by the cost it forgoes. Table 6.1 maps the vocabulary of retrieval and recommendation onto the network, and Table 6.2 collects the notation, which extends that of Part 5 (Table 1.1).

Table 6.1: Vocabulary of retrieval and recommendation and its counterpart on the LV network.
Concept Retrieval and recommender systems Low-voltage network
Query A search or a user A connection request, transformer or violation event
Case or item A document or a product A simulated transformer with known outcomes
Relevance A graded judgement or a rating Hours outside the statutory band
Action An item shown to a user A mitigation measure applied to a transformer
Feedback Responses to the items shown The field record of the mitigation measure applied
Counterfactual Responses to items not shown Simulated outcomes of the other mitigation measures
Cold start A user or item without history A connection without a network study
Table 6.2: Notation of Part 6, extending Table 1.1 of Part 5.
Symbol Description
\(q\), \(\mathbf{x}_q\) Query and its features
\(\mathcal{D}_1\), \(\mathcal{D}_2\) Cases kept for retrieval, and \(n\) held-out calibration cases
\(\mathcal{A}\), \(a\) Set of actions (mitigation measures) and one action; \(a = 0\) is no action
\(Y_q(a)\) Potential outcome of \(q\) under \(a\), in hours outside the band
\(\tau_q(a)\) Uplift of \(a\), the hours it removes relative to no action
\(C_q(a)\), \(R_q\) Total cost of \(a\) for \(q\), and regret of a recommendation
\(\mathcal{N}_k(q)\), \(\hat y_q\) The \(k\) kept cases nearest to \(q\), and the estimated outcome
\(s(\mathbf{x})\), \(\hat r\) Distance to the nearest kept case, and trust radius
\(\gamma_i\), \(\varepsilon\) Residual score of a held-out case, and its conformal quantile
\(\hat\sigma(\mathbf{x})\), \(z_q\) Local scale of the residuals, and group of a query
\(m\), \(K\) Number of items in a list or of units sharing a burden, and cut-off of a list
\(\mathrm{rel}_i\), \(w_i\), \(f_i\) Relevance grade, weight and score of the item in position \(i\)
\(\rho_a(\mathbf{x})\), \(\omega\) Propensity of action \(a\), and share of treated cases
\(\pi\), \(V(\pi)\) Policy, and its mean cost
\(G(\mathbf{b})\) Gini coefficient of a burden \(\mathbf{b}\)

An Illustrative Case Base

Scoring every step exactly requires a population in which \(Y_q(a)\) is known for every mitigation measure, and a synthetic case base supplies one before the AusNet networks of Chapter 2. It contains 600 transformers on four networks. Transformer \(i\) serves a feeder of electrical length \(L_i\), with a share \(p_i\) of customers with rooftop PV, a share \(e_i\) charging an Electric Vehicle (EV) in the evening, and a level of demand \(d_i\). Its voltage at the end of the feeder over a clear day is

\[ v_i(t) = v_0 + L_i \bigl( \kappa_i p_i\, g(t) - \beta_d d_i\, u(t) - \beta_e e_i\, c(t) \bigr), \tag{6.2}\]

where \(v_0 = 1.03\) pu is the sending voltage, \(g\), \(u\) and \(c\) are the profiles of PV output, household demand and evening charging, \(\beta_d\) and \(\beta_e\) are sensitivities, and \(\kappa_i\) is the strength of the PV voltage rise, which varies between transformers and is not among the features. The four networks differ in the length of their feeders, from urban to rural.

The inverters of every case respond to voltage as AS/NZS 4777.2 requires of them, which no operator chooses. Their volt-watt response leaves PV output unrestricted up to 253 V (1.10 pu) and curtails it to 20% at 260 V, so it holds the voltage near the upper limit rather than inside the band; their volt-VAr response lies outside a model without reactive power. The mitigation measures of the network operator are stated as changes to the terms of Equation 6.2. Smart charging moves the charging term to the night, an export limit caps PV export at 70% of its peak, a tap change lowers the transformer’s off-load tap one step and with it \(v_0\) by 2.5%, and reinforcement with a larger transformer removes 40% of every voltage change. The outcome \(Y_i(a)\) is the number of hours \(v_i\) spends outside the AS 61000.3.100 band of 0.94 to 1.10 pu under mitigation measure \(a\).

Without action, 60.8% of the transformers left the band at some hour, 35.8% rising above it at midday and 47.5% falling below it in the evening (Table 7.2). Reinforcement reduced the mean hours outside the band from 2.72 to 0.45, while the tap change raised them to 3.02, because the evening hours it adds outweigh, on average, the midday hours it removes. No mitigation measure was the cheapest everywhere. Under the costs of Equation 6.26, no action was the cheapest for 41.3% of the transformers, reinforcement for 26.0%, smart charging for 19.8% and the tap change for 3.8%, those whose only violation is at midday. The choice of mitigation measure is therefore a decision for each transformer, and the case base is a fair test of the methods that make it.

Table 6.3: Outcomes of the five mitigation measures on the 600 transformers of the case base: mean hours outside the band, share of transformers kept within it, share for which the mitigation measure is cheapest, and annualised capital cost in arbitrary units.
Mitigation measure Mean hours outside the band Transformers within the band Cheapest for Capital cost
No action 2.72 0.39 0.41 0.00
Smart charging 1.89 0.58 0.20 0.80
Export limit 1.95 0.49 0.09 0.20
Tap change 3.02 0.23 0.04 0.10
Reinforce 0.45 0.85 0.26 1.50

The case base is split at random into 300 transformers kept for retrieval, 150 held out for calibration and 150 for testing. The four features \(L\), \(p\), \(e\) and \(d\) are standardised on the kept transformers. The kept transformers play the role of the studies a utility has already run, and the test transformers the role of new requests, which is where the chain begins.

Retrieval Under a Trust Radius

A new connection request is the first decision of the chain, and it usually arrives without a network study. This section estimates its outcome from the case base, and then asks when that estimate deserves to be used.

Nearest Neighbours and the Trust Radius

A request without a study is a cold start in the sense of recommender systems. The query has no outcome of its own, and with no history of interactions its outcome can only be estimated from the content of similar cases. The estimate of this part is the mean outcome of the \(k\) nearest kept cases,

\[ \hat y_q = \frac{1}{k} \sum_{j \in \mathcal{N}_k(q)} y_j, \tag{6.3}\]

with \(k = 10\) and the Euclidean distance on the standardised features. Equation 6.3 returns an estimate for every query, including one unlike any case in \(\mathcal{D}_1\), whose nearest neighbours describe it poorly. A request from a household unlike any simulated, or from a network with longer feeders, is such a query. The estimate therefore needs a rule that decides when to answer from the case base and when to run a study instead, and the trust radius of Figure 6.2 is that rule.

Figure 6.2: Schematic of routing by the trust radius. (a) Distance of each held-out case to the nearest kept case; the 90% conformal quantile of these distances is the radius. (b) Two PV requests among the kept cases, ringed at the radius. (c) Request A is estimated from its neighbours; request B is simulated and added to the case base.

The radius has to refuse few queries that resemble the case base, and conformal prediction sets it with that guarantee. It follows the split-conformal construction of Part 5, Chapter 3 (Equation 3.2), with the distance to the case base as the nonconformity score [26], [27]. The score of a query is its distance to the nearest kept case,

\[ s(\mathbf{x}) = \min_{j \in \mathcal{D}_1} \lVert \mathbf{x} - \mathbf{x}_j \rVert. \tag{6.4}\]

The held-out cases calibrate this score. Let \(s_{(1)} \le \dots \le s_{(n)}\) be the ordered scores of the \(n\) cases of \(\mathcal{D}_2\). The trust radius at level \(\alpha\) is the order statistic

\[ \hat r = s_{(\lceil (n + 1)(1 - \alpha) \rceil)}. \tag{6.5}\]

If a query is exchangeable with the held-out cases, it lies within the radius with probability

\[ 1 - \alpha \le \Pr\{ s(\mathbf{x}_q) \le \hat r \} \le 1 - \alpha + \frac{1}{n + 1}, \tag{6.6}\]

the upper bound holding when the scores have no ties [28], [29]. At most a share \(\alpha\) of the queries that resemble the case base is therefore refused. The converse is also informative, since a refusal rate well above \(\alpha\) on a new population measures how far that population has shifted. The same test can be written with the conformal p-value

\[ p(\mathbf{x}_q) = \frac{1 + \#\{ i \in \mathcal{D}_2 : s_i \ge s(\mathbf{x}_q) \}}{n + 1}, \tag{6.7}\]

with refusal when \(p(\mathbf{x}_q) \le \alpha\) [26].

Error Intervals

The radius guarantees that a trusted query is close to some case, but closeness does not make its estimate accurate. A second calibration, on the outcomes rather than the features, supplies an interval with its own guarantee. The residual score of a held-out case is

\[ \gamma_i = \frac{|y_i - \hat y_i|}{\hat\sigma(\mathbf{x}_i)}, \tag{6.8}\]

and \(\varepsilon\) is its order statistic by the rule of Equation 6.5. The interval

\[ \mathcal{C}(\mathbf{x}_q) = \bigl[ \hat y_q - \varepsilon\, \hat\sigma(\mathbf{x}_q),\ \hat y_q + \varepsilon\, \hat\sigma(\mathbf{x}_q) \bigr] \tag{6.9}\]

then covers a new outcome with probability at least \(1 - \alpha\). The scale \(\hat\sigma\) determines the shape of the interval. With \(\hat\sigma \equiv 1\) the interval has a fixed width. With \(\hat\sigma = \hat\sigma_k\), the standard deviation of the outcomes of the \(k\) nearest cases, it is the locally weighted interval of Lei and colleagues [28], wider where the neighbours disagree (Figure 6.3).

Figure 6.3: Schematic of conformal error intervals for two PV requests estimated from their nearest cases (darker). The fixed interval (grey) has one width; the adaptive interval (orange) is narrower where the neighbours agree and wider where they disagree.

Both guarantees hold on average over all queries, whereas a utility may need them within each part of its network. Both calibrations can therefore be made separately for each group \(z\) of cases, such as a network or a regime of violation, and the guarantee then holds within each group,

\[ \Pr\{ Y_q \in \mathcal{C}(\mathbf{x}_q) \mid z_q = z \} \ge 1 - \alpha, \tag{6.10}\]

rather than only on average over the groups [26].

Retrieval on the Case Base

The estimate, the radius and the interval were applied to the test transformers, and to a second population drawn to differ from the case base. Retrieval reduced the mean absolute error from 2.61 hours, that of the mean of the kept cases, to 1.17 hours for the queries it trusted (Table 6.4). The radius, calibrated at \(\alpha = 0.1\), trusted 87.3% of the test transformers. Queries from a population whose feeders were drawn between 1.0 and 1.8 rather than between 0.5 and 1.2, before the factor of each network, were trusted in only 51.3% of cases. Those refused were estimated with an error of 2.04 hours against 1.53 hours for those trusted, so the radius both detected the shift and identified the queries it affected most.

Table 6.4: Retrieval of hours outside the band by the ten nearest kept cases, with the trust radius and the adaptive interval calibrated at 90%, for queries from the same networks and from networks with longer feeders. Errors are mean absolute errors in hours; coverage and width refer to trusted queries.
Queries Trusted Error when trusted Error when refused Interval coverage Interval width
Same networks 0.87 1.17 1.22 0.85 3.93
Longer feeders 0.51 1.53 2.04 0.82 5.22

The interval appeared less reliable on this split. The adaptive interval covered 84.7% of the trusted test outcomes, below the nominal 90%. The guarantee of Equation 6.6, however, is an average over random splits, and a single split of 150 test cases varies around it by several points. Over 200 random splits the trust radius admitted 89.9% of the test transformers, the fixed-width interval covered 90.1% and the adaptive interval 89.9%, against a finite-sample floor of 90.07% (Table 6.5). The adaptive interval reached this coverage with a mean width of 4.41 hours against 5.82 hours for the fixed width, 24% narrower, because it concentrated its width on the queries whose neighbours disagreed.

Table 6.5: Coverage of the trust radius and of the fixed and adaptive error intervals over 200 random splits at a nominal 90%, with the mean interval width in hours.
Interval Trusted Coverage Standard deviation of coverage Mean width
Adaptive 0.899 0.899 0.033 4.406
Fixed width 0.899 0.901 0.031 5.818

A conformal guarantee is a property of the procedure, averaged over the random choice of the calibration and test cases, and a single evaluation can fall below it by chance. A coverage reported for one split should be read with its sampling variation, here a standard deviation of about 3 points. A claim that a method under-covers requires repeated splits.

A trusted estimate with a calibrated interval answers a single request. A utility, however, holds many transformers at once and can study only some of them, so the estimates must next be compared across transformers.

Ranking and Its Evaluation

Comparing transformers turns retrieval into ranking, and the quality of a ranking depends on where its errors fall. This section first defines how an order is judged, then how it can be learned, and then compares both on the case base.

Measures of a Ranked List

A DSO that examines transformers in order of estimated risk needs the order to be correct at the top, where the examinations happen. Let a score place the \(m\) transformers of one list in positions \(i = 1, \dots, m\), and let \(\mathrm{rel}_i \ge 0\) be the relevance grade of the item in position \(i\). Here the grade is the hours outside the band; Figure 6.4 uses four illustrative levels instead. The simplest measures look only at the first \(K\) positions. With \(S_K\) the items in those positions and \(T_K\) the \(K\) most relevant items, precision at \(K\) is

\[ \mathrm{P}_K = \frac{|S_K \cap T_K|}{K}. \tag{6.11}\]

The reciprocal rank looks only at the most relevant item, placed at position \(i^\star\),

\[ \mathrm{RR} = \frac{1}{i^\star}, \tag{6.12}\]

and its mean over the lists is the mean reciprocal rank, Mean Reciprocal Rank (MRR). Precision counts an item as relevant or not, and the reciprocal rank follows a single item, so neither distinguishes a near miss from a gross error. The discounted cumulative gain does, by weighting the grade of each position [8],

\[ \mathrm{DCG}_K = \sum_{i=1}^{K} \mathrm{rel}_i\, w_i, \qquad w_i = \frac{1}{\log_2(i + 1)}. \tag{6.13}\]

The gain is linear in the grade. The exponential gain \(2^{\mathrm{rel}_i} - 1\), common in web search [11], is appropriate for grades on a short ordinal scale and would make hours outside the band meaningless as a gain. Because \(\mathrm{DCG}_K\) grows with the grades of the list, lists are compared after dividing by the gain of the ideal order, \(\mathrm{IDCG}_K\),

\[ \mathrm{NDCG}_K = \frac{\mathrm{DCG}_K}{\mathrm{IDCG}_K}, \tag{6.14}\]

which equals one for a perfect ranking (Figure 6.4).

Figure 6.4: Schematic of NDCG for a model ranking of six transformers against the ideal order of their relevance grades. Position i carries the weight 1/log2(i + 1), and NDCG is the ratio of the two discounted cumulative gains.

The top of the list is not the only concern of a planner, who may study the remaining transformers in order as well. Kendall’s coefficient measures the agreement of two orders over the whole list [39]. Many transformers of a list share the same grade, since 39% have no hours outside the band, and the coefficient is therefore taken in its form corrected for ties,

\[ \tau_{\mathrm{b}} = \frac{\nu_{\mathrm{c}} - \nu_{\mathrm{d}}}{\sqrt{(\nu_0 - \nu_{\mathrm{rel}})(\nu_0 - \nu_f)}}, \tag{6.15}\]

where \(\nu_{\mathrm{c}}\) and \(\nu_{\mathrm{d}}\) count the concordant and discordant pairs, \(\nu_0 = m(m - 1)/2\) is the number of pairs, and \(\nu_{\mathrm{rel}}\) and \(\nu_f\) count the pairs tied in grade and in score. With one list per network, each measure is averaged over the four networks.

Learning to Rank

The measures judge a score, and the next question is where the score comes from. It can be the estimate of Equation 6.3, which is fitted to the outcome of each transformer and ranks only as a consequence. A pointwise loss of this kind fits each grade separately, as a regression does,

\[ \ell_i = (f_i - \mathrm{rel}_i)^2, \tag{6.16}\]

where \(f_i\) is the score of item \(i\). The score can instead be learned to order the list directly. A pairwise loss penalises each pair with \(\mathrm{rel}_i > \mathrm{rel}_j\) that the scores order wrongly, through the logistic loss of RankNet [10],

\[ \ell_{ij} = \log\bigl(1 + e^{-(f_i - f_j)}\bigr). \tag{6.17}\]

The pairwise loss counts every misordered pair equally, whereas Equation 6.14 weights errors by position. LambdaRank closes this gap by weighting the gradient of each pair by the change in Normalised Discounted Cumulative Gain (NDCG) that swapping the two items would cause [11],

\[ \lambda_{ij} = -\frac{|\Delta \mathrm{NDCG}_{ij}|}{1 + e^{f_i - f_j}}, \tag{6.18}\]

so that errors near the top of a list cost most (Figure 6.5). The loss remains pairwise, but its weights come from a listwise measure, which places LambdaRank between the pairwise and listwise families of Liu [9]. LambdaMART fits these gradients with boosted regression trees [12]. The ranker of this part grades the outcome of each list into five quantile levels and fits LambdaMART on them.

Figure 6.5: Schematic of a pairwise loss against LambdaRank on six ranked transformers. Step height is the weight of a position and blocks are the grade. Of two misordered pairs, the pairwise loss counts both alike, while LambdaRank weights the swap at the top about twelve times the swap at the bottom.

Ranking on the Case Base

The two sources of a score, the pointwise estimate and the learned ranker, were compared with two baselines on the test transformers of each network. A single split of the case base varies too much to separate them, so the comparison was repeated over the 200 random splits used for coverage, and each pair of scores was compared split by split with a Wilcoxon signed-rank test. The nearest kept cases ranked best on every measure, with an NDCG at 10 of 0.858 and \(\tau_{\mathrm{b}}\) of 0.633, against 0.841 and 0.588 for LambdaMART (Table 6.6). The advantage in NDCG was small, 0.016 on average and present in 71% of the splits, but significant (\(p < 10^{-9}\)). The advantage in \(\tau_{\mathrm{b}}\) was larger and more consistent, present in 93% of the splits (\(p < 10^{-30}\)), so the estimate ordered the whole list better than the ranker trained to order it. The MRR differed least, 0.533 against 0.500, with the nearest cases ahead in 59% of the splits (\(p = 0.008\)).

Table 6.6: Ranking of the test transformers of each network by hours outside the band, averaged over the four networks and over 200 random splits of the case base.
Score NDCG at 10 Standard deviation of NDCG Precision at 10 MRR Kendall's τ-b
Nearest cases 0.858 0.039 0.706 0.533 0.633
LambdaMART 0.841 0.042 0.684 0.500 0.588
PV share 0.649 0.065 0.520 0.317 0.305
Random 0.368 0.067 0.273 0.105 0.004

Ranking by the PV share alone, a heuristic available to any utility, reached an NDCG of 0.649 and an MRR of 0.317, and a random order 0.368 and 0.105. The PV share therefore captured part of the risk but ordered neither the top nor the rest of the list as well as the case base, since the evening under-voltage of a transformer depends on its charging and demand rather than its PV. With 300 kept transformers and four features, a ranker learned for the purpose did not improve on the estimate it ranks. Whether learning to rank helps on larger and more varied lists is examined in Chapter 2.

A ranked list tells a utility where to act but not what to do. The last step of the chain chooses the mitigation measure, and it requires a different kind of estimate.

Recommendation as a Causal Question

Retrieval and ranking estimate the outcome of a transformer as it is. Recommendation must estimate how the outcome would change under each mitigation measure, which is a causal question. This section starts from the practice of reusing past remedies, shows why field records bias it, states the estimators that correct the bias, and then judges recommendations by their cost.

Utilities already recommend remedies from experience, and case-based reasoning formalises that practice (Figure 6.6). A planner retrieves the transformer most like the new one, reuses its remedy, revises the remedy if a study shows that it fails, and retains the solved case. The cycle is sound when the cases record what each remedy did. A field record, however, holds one outcome per transformer, that of the action taken, and the next subsection states what this omission costs.

Figure 6.6: Schematic of the case-based reasoning cycle on a transformer with midday over-voltage. (a) The new problem. (b) Retrieval of the most similar solved case by its daily voltage. (c) Reuse and revision of its mitigation measure, a tap change, which keeps the voltage within the limit before the case is retained.

Potential Outcomes and Confounding

What a remedy did is a comparison between two potential outcomes. The effect of action \(a\) on query \(q\) is its uplift, the hours it removes relative to no action,

\[ \tau_q(a) = Y_q(0) - Y_q(a). \tag{6.19}\]

Figure 6.7: Schematic of the potential outcomes of five mitigation measures on one transformer with midday over-voltage. (a) The transformer. (b) Hours above the limit under each mitigation measure; only the action taken (solid) is observed in the field. (c) Uplift of each mitigation measure relative to no action.

Only one of the two potential outcomes is observed in the field (Figure 6.7). An estimate from a field record must therefore compare different transformers, those that received the action, \(A = a\), and those that did not. The naive comparison of their means decomposes as

\[ \mathbb{E}[Y \mid A = 0] - \mathbb{E}[Y \mid A = a] = \tau_{\mathrm{T}} + B, \tag{6.20}\]

where \(\tau_{\mathrm{T}} = \mathbb{E}[\tau(a) \mid A = a]\) is the effect on the treated transformers and

\[ B = \mathbb{E}[Y(0) \mid A = 0] - \mathbb{E}[Y(0) \mid A = a] \tag{6.21}\]

is the selection bias [6]. A utility applies a mitigation measure where the problem is worst, so the treated transformers have the larger \(Y(0)\), \(B\) is negative, and the naive comparison understates the effect (Figure 6.8), in the case base enough to make a mitigation measure that changes little appear harmful.

Figure 6.8: Schematic of selection bias in the case base when the tap change follows the PV share. The treated feeders would have had more hours outside the band without the mitigation measure (dashed), so the naive comparison of observed means misstates the effect on the treated by the selection bias, here even in sign.

The bias arises because the choice of action depends on the transformer, and it can be removed when that dependence is captured by the features. The effect can then be estimated from observational records under two conditions [24]. The first is that every feature that influenced the choice of action is among the features,

\[ \{ Y(0), Y(a) \} \perp\!\!\!\perp A \mid \mathbf{x}. \tag{6.22}\]

The second is that every transformer could have received either action, with propensity

\[ 0 < \rho_a(\mathbf{x}) = \Pr\{ A = a \mid \mathbf{x} \} < 1. \tag{6.23}\]

Meta-Learners

Under these conditions the effect can be estimated by comparing transformers with the same features, and the meta-learners of Künzel and colleagues do so with ordinary regressors [25]. The T-learner fits a model \(\hat\mu_0\) of the outcome on the untreated cases and a model \(\hat\mu_1\) on the treated cases, and estimates

\[ \hat\tau(\mathbf{x}) = \hat\mu_0(\mathbf{x}) - \hat\mu_1(\mathbf{x}), \tag{6.24}\]

with the sign of Equation 6.19. When one group is small, its model is poor and so is the difference. The X-learner reduces this dependence by imputing an effect for each case from the model of the other group, \(\tilde\tau_i = \hat\mu_0(\mathbf{x}_i) - y_i\) for a treated case and \(\tilde\tau_i = y_i - \hat\mu_1(\mathbf{x}_i)\) for an untreated one. It regresses the imputed effects on the features within each group, giving \(\hat\tau_1\) and \(\hat\tau_0\), and combines them with the share \(\omega\) of treated cases,

\[ \hat\tau(\mathbf{x}) = \omega\, \hat\tau_0(\mathbf{x}) + (1 - \omega)\, \hat\tau_1(\mathbf{x}), \tag{6.25}\]

so that the regression built on the model of the larger group receives the larger weight (Figure 6.9).

Figure 6.9: Schematic of the T-learner and the X-learner on the field record of the case base, where the tap was lowered (orange) more often at high PV. The T-learner takes the gap between the models fitted to each group; the X-learner compares each case with the model of the other group.

To test the estimators against a known effect, a field record was generated in which the tap was lowered with a probability that rose with the PV share, reaching 49.1% of the kept and held-out transformers. The effect of the tap change differs in sign between transformers, removing midday hours where PV dominates and adding evening hours where charging does, and its simulated uplift on the test transformers averaged \(-0.29\) hours. The naive comparison estimated \(-1.26\) hours, four times the harm, because the treated transformers were those with the most PV and the worst baseline, as Equation 6.21 predicts (Table 6.7). The T-learner estimated \(-0.35\) hours and the X-learner \(-0.29\) hours. For individual transformers the X-learner was the more accurate, with an error of 1.44 hours against 1.69 hours and a correlation with the simulated effect of 0.48 against 0.45.

Table 6.7: Estimates of the hours removed by a tap change from a confounded field record, against the simulated effect: average effect and its error, and for the meta-learners the error and correlation of the effect on each transformer.
Estimator Average effect Error of the average RMSE per transformer Correlation
Difference in means −1.26 −0.98
T-learner −0.35 −0.06 1.69 0.45
X-learner −0.29 0.00 1.44 0.48

Cost and Regret

An estimated effect does not yet choose a mitigation measure, because the mitigation measures that remove the most hours also cost the most. A recommendation is therefore judged by its total cost. The total cost of mitigation measure \(a\) for query \(q\) is

\[ C_q(a) = c_{\mathrm{cap}}(a) + \lambda E_q(a) + \mu Y_q(a), \tag{6.26}\]

where \(c_{\mathrm{cap}}(a)\) is the annualised capital cost of the mitigation measure, \(E_q(a)\) the PV energy it curtails at a value \(\lambda\), and \(Y_q(a)\) the hours outside the band that remain at a penalty \(\mu\). The regret of a recommendation is the cost it forgoes relative to the best mitigation measure,

\[ R_q = C_q(\hat a_q) - \min_{a \in \mathcal{A}} C_q(a). \tag{6.27}\]

Figure 6.10 shows the parts of the cost and the regret for one transformer of the case base.

Figure 6.10: Schematic of the total cost of each mitigation measure for one transformer of the case base, by capital, curtailed PV and the penalty on the hours that remain. A tap change has the least cost; choosing reinforcement forgoes the difference, the regret.

Four policies were compared on the test transformers by their regret (Table 6.8). The case-based policy estimated the cost of every mitigation measure from the ten nearest kept cases and chose the least. The regime rule of Part 5 applied a tap change to over-voltage, smart charging to under-voltage, reinforcement to both and no action to neither. The remaining two applied a tap change everywhere and nothing anywhere. The regime rule had the least mean regret, 0.19, and chose a mitigation measure of least cost for 87% of the transformers. The case-based policy had a regret of 0.25 and chose optimally for 54%. Applying a tap change everywhere cost 1.93 in regret, more than taking no action, 1.55, because it deepens every evening under-voltage.

Table 6.8: Regret of four policies on the test transformers: mean regret, share of least-cost choices, and mean hours outside the band that remain.
Policy Mean regret Optimal choices Mean hours outside the band
Case-based 0.25 0.54 0.63
Regime rule 0.19 0.87 0.73
Always tap change 1.93 0.05 2.92
No action 1.55 0.45 2.63

The regime rule performed well because the mitigation measures of the case base act on the regimes by construction, the tap change on the midday rise and smart charging on the evening charging. Chapter 3 examines whether the rule holds on a network whose violations depend on more than these two terms, where the case-based policy has the advantage of requiring no such rule.

A policy that minimises the cost of each transformer separately says nothing about how its benefits and burdens are shared across the network, which the next section addresses.

Equity of Targeting

Recommendations made transformer by transformer can concentrate their effects. A programme that enrols the transformers of greatest benefit may serve a few networks only, and the volt-watt response of inverters curtails the customers at the ends of feeders more than those near the transformer. Measuring this concentration requires an index of inequality. For a burden \(\mathbf{b} = (b_1, \dots, b_m)\) shared among \(m\) units with mean \(\bar b\), the Gini coefficient is [40]

\[ G(\mathbf{b}) = \frac{1}{2 m^2 \bar b} \sum_{i=1}^{m} \sum_{j=1}^{m} |b_i - b_j|. \tag{6.28}\]

It is zero when the burden is shared equally and approaches one when a single unit bears it all. For ranked lists, fairness of exposure applies the same idea to the attention \(w_i\) of Equation 6.13 that each group receives [31].

An index only describes the concentration, and equity becomes an objective when the index is bounded. A programme of \(M\) places then chooses the set \(S\) of units of largest benefit subject to a bound \(\bar g\) on the Gini coefficient of the enrolment rates \(\boldsymbol{\phi}_S\) of the groups,

\[ \begin{aligned} S^\star = {} & \arg\max_{|S| = M} \textstyle\sum_{i \in S} \tau_i \\ & \text{subject to } G(\boldsymbol{\phi}_S) \le \bar g. \end{aligned} \tag{6.29}\]

Sweeping \(\bar g\) traces the price of equity, the benefit forgone for each reduction of \(G\).

The price was measured on a smart-charging programme that enrolled 20 of the 150 test transformers, with the hours removed as its benefit. The benefit rose with the length of the feeders, from a mean of 0.27 hours on the urban network to 1.17 hours on the most rural, so the unconstrained programme enrolled the rural networks, at a Gini coefficient of 0.258 across the four networks (Figure 8.5, Table 8.3). Bounding the coefficient at 0.1 reduced it to 0.095 and retained 96.3% of the hours removed. The price of near-equal enrolment was therefore 3.7% of the benefit.

Figure 6.11: Schematic of a smart-charging programme of 20 test transformers, one dot per transformer ordered by benefit within its network. (a) Enrolled by largest benefit, the programme favours the rural network. (b) Under a bound of 0.1 on the Gini coefficient of the enrolment rates, it spreads across the networks.
Table 6.9: Price of equity in a smart-charging programme of 20 test transformers: hours removed under a bound on the Gini coefficient of the enrolment rates of the four networks, relative to no bound.
Gini bound Hours removed Share of the unconstrained Gini achieved
0.0 64.500 0.963 0.095
0.1 64.500 0.963 0.095
0.2 65.500 0.978 0.183
0.3 67.000 1.000 0.258
0.5 67.000 1.000 0.258
1.0 67.000 1.000 0.258

Every result of this chapter, from the coverage of the radius to the price of equity, was computed against simulated outcomes. The final section makes explicit what this changes relative to the evaluation practised in recommender systems.

Evaluation Against a Simulator

The steps of the chain combine into a policy \(\pi\), which maps the features of a query to an action, and a policy is judged by its value, its mean cost over \(N\) queries. Recommender systems estimate this value offline from a log in which action \(a_i\) was taken with propensity \(\rho_{a_i}(\mathbf{x}_i)\), by inverse propensity weighting [21], [22],

\[ \hat V(\pi) = \frac{1}{N} \sum_{i=1}^{N} \frac{\mathbb{1}\{\pi(\mathbf{x}_i) = a_i\}}{\rho_{a_i}(\mathbf{x}_i)}\, C_i(a_i). \tag{6.30}\]

The estimate uses only the queries on which the policy agrees with the log, and its variance grows as the propensities fall. A simulator removes both limitations, because every action can be applied to every query,

\[ V(\pi) = \frac{1}{N} \sum_{i=1}^{N} C_i\bigl(\pi(\mathbf{x}_i)\bigr). \tag{6.31}\]

Figure 6.12 contrasts the two on six transformers of the case base.

Figure 6.12: Schematic of scoring a policy, here the regime rule (ringed), on six test transformers. (a) A field record holds one outcome per transformer, so inverse propensity weighting scores the policy only where its choice matches the record. (b) A simulator knows every outcome and scores every choice.

The mean regret of Table 6.8 is \(V(\pi)\) less the value of the policy that always chooses the cheapest mitigation measure, and the error of an uplift estimate in Table 6.7 is computed in the same way. Exact evaluation is not complete evaluation, however, and two limitations remain. A simulated outcome is only as accurate as the network model and the profiles that drive it, so a method is validated against the simulator rather than against the network. The methods are also trained on simulated cases, which a utility would replace with its records, and performance on records that mix field outcomes with confounded choices of action is the harder test, which Table 6.7 only begins.

Conclusion

This chapter stated retrieval, ranking and recommendation for the LV network as one decision chain. A query is described by its features, its outcome is estimated from the cases nearest to it, the queries are placed in an order, and each is assigned the mitigation measure of least estimated cost. Each step was given a formal statement, drawn from information retrieval, recommender systems and causal inference, and an evaluation that a simulator makes exact. Each was illustrated on a synthetic case base of 600 transformers whose outcome under five mitigation measures was known.

The case base answered the three questions of the introduction in turn. An estimate from similar cases can be trusted within a conformal trust radius, which admitted 89.9% of in-distribution queries over 200 splits at a nominal 90%, and 51.3% of queries from networks with longer feeders, whose refused queries were estimated with errors a third larger than the trusted ones. An adaptive error interval held the same coverage with intervals 24% narrower than a fixed width. An order is judged by measures that weight the top of the list, and by these the estimate of the nearest cases reached an NDCG at 10 of 0.858 over 200 splits, and a ranker learned for the purpose ranked significantly worse, with the largest difference over the whole list. A tap change that adds 0.29 hours on average appeared to add 1.26 hours when estimated naively from a record in which it followed the PV share, and the meta-learners removed most of this bias. Judged by cost, the regime rule of Part 5 chose the least-cost mitigation measure for 87% of transformers, in a case base where the mitigation measures act on the regimes by construction.

The limitations are those of a synthetic case base. Its outcomes follow a stated voltage model with four features and one unobserved strength of the voltage rise, and its networks differ only in the length of their feeders. The relative performance of the methods on it is therefore illustrative, not a result about real networks. Chapter 2 applies retrieval and ranking to the AusNet networks, where the outcomes come from power-flow simulation and the features from smart-meter data.

Exercises

  1. Change \(\alpha\) in the setup cell from 0.1 to 0.2 and to 0.05. How do the trusted share and the interval width of Table 6.4 follow \(\alpha\), and does the coverage over repeated splits in Table 6.5 follow \(1 - \alpha\)?
  2. Draw the queries of the shifted population with feeders between 1.2 and 2.4 times the original range. How do the trusted share and the error of the refused queries change?
  3. Make the choice of the tap change in the uplift cell depend on \(\kappa\), the unobserved strength of the voltage rise, as well as on the PV share. Which condition of Equation 6.22 is then violated, and which estimators of Table 6.7 remain close to the simulated effect?
  4. Raise the capital cost of reinforcement in umeme.casebase.LEVERS from 1.5 to 3.0. How do the regret and the optimal share of the regime rule and of the case-based policy change?

References