Segmentation for Network Studies

Chapter 4 located the weakness of the shape archetypes in their input rather than in the clustering. The archetypes describe when a household consumes, whereas the thermal risk of a transformer depends on how much its customers draw, and peak normalisation removes exactly that quantity. This finding raises three questions for the uses of segmentation in network operation. The first is whether the same pipeline, applied to an input that retains the level of demand, produces segments that retain the risk. The second is whether representative profiles drawn from such segments can replace the measured readings of every customer in a network study. The third is whether the identity of the customers who adopt Photovoltaic (PV) or an Electric Vehicle (EV) charger, described by these segments, changes the response of the network.

Segmentations that Retain the Level of Demand

The weakness of the archetypes thus lies in their input rather than in the clustering, which suggests that the same pipeline, applied to an input that retains the level of demand, should capture the risk that the archetypes miss. This section tests that proposition with three alternatives to the peak-normalised profile. The first appends three magnitude features, the logarithms of mean and peak demand and the load factor, to the normalised profile, with both blocks standardised and given equal weight. The second clusters the seasonal daily profiles in kW, so that level and timing enter together. The third describes each household by its coincident demand, its mean demand during the violations of each of the three extreme scenarios and the ratio of that demand to its usual demand at the same hour. Each input passes through the pipeline of the archetypes, a t-SNE embedding clustered with k-means. Every segmentation is compared at the \(K = 3\) of the archetypes, which is also the number of groups of the tercile reference, so that differences in the variance explained do not arise from different numbers of groups; Figure 5.2 compares every \(K\) from 2 to 12.

A segmentation fitted on the violations it is asked to explain would explain them by construction. The coincident-demand features are therefore computed from the violations of the summer half-year only, from October to March, and every segmentation is evaluated on household demand during the violations of the winter half-year, which none of them saw, together with the simulated outcomes of Attribution of Risk to Archetypes averaged over the customers assigned each household’s profile. The terciles of annual demand serve as a reference that requires no clustering. Figure 5.1 illustrates the design.

Figure 5.1: Schematic of the held-out evaluation of the segmentations. (a) The year from April: the coincident-demand features are computed from the violations of the summer half-year, and every segmentation is evaluated on household demand during the violations of the winter half-year. (b) The quantity evaluated, the share of the variance of winter event-time demand that lies between segments, large when the segments differ in level and small when they overlap. The strips are stylised.
Table 5.1: Four segmentations of the 342 AusNet households, each built with the pipeline of the archetypes (t-SNE and k-means, K chosen by the trust-gated composite score), against the terciles of annual mean demand. The last five columns give the share of the variance across households explained by the segmentation (eta squared) of the household’s mean demand during the violations of the winter half-year at 80% penetration, which no segmentation saw, and of its simulated contribution to the network peak and worst voltage. The coincident-demand segmentation is built from the violations of the summer half-year.
Segmentation K Silhouette Balance Event-time demand, winter (held out) Simulated outcome
Over-voltage, PV Under-voltage, EV Overload, EV Peak contribution Worst voltage
Shape (archetypes) 3 0.404 0.994 0.136 0.057 0.048 0.068 0.000
Shape and size 3 0.399 0.995 0.458 0.370 0.336 0.147 0.004
Demand profile (kW) 3 0.516 0.995 0.539 0.448 0.410 0.188 0.007
Event role 3 0.453 0.994 0.095 0.082 0.078 0.396 0.003
Demand tercile 3 1.000 0.536 0.454 0.421 0.180 0.005

Table 5.1 reports the comparison, each share computed with Equation 4.4. The clustering of profiles in kW reached a silhouette of 0.516 and a balance of 0.995, better separated than the archetypes at 0.404. Its three segments explained 53.9% of the variance of winter demand during over-voltage, 44.8% during under-voltage and 41.0% during overload, against 13.6%, 5.7% and 4.8% for the archetypes, and 18.8% of the contribution to the network peak against 6.8%. Shape and size together explained 33.6 to 45.8% of the winter event-time demand. The demand clusters matched the terciles of annual demand, which explained 53.6%, 45.4% and 42.1%, so that clustering in kW recovers the level of demand without being given it as a feature. No segmentation explained more than 0.7% of the worst voltage, which confirms that voltage risk under PV resides in the point of connection rather than in the household.

The coincident-demand segmentation illustrates the hazard against which the held-out evaluation guards. It explained 39.6% of the contribution to the peak on a summer evening, more than any other segmentation, but only 7.8 to 9.5% of the winter event-time demand, because the summer violations from which it was built describe one season’s evening peaks rather than a persistent property of the household. A segmentation derived from network outcomes thus requires the same out-of-sample evaluation as a forecast before it can inform a decision.

Figure 5.2: Share of the variance across households explained by each segmentation as the number of clusters rises: (a) of the mean demand during EV-driven under-voltage in the held-out winter half-year, and (b) of the simulated contribution to the network peak.

This ranking held at every number of clusters (Figure 5.2). For winter demand during under-voltage, the demand-profile segmentation rose from 39.2% at \(K = 2\) to 64.0% at \(K = 10\), whereas the shape archetypes reached at most 15.3% for any \(K\) up to 12. For the contribution to the peak, the coincident-demand segmentation led at every \(K\), reaching 59.3% at \(K = 11\), consistent with the season of its construction, and the demand-profile segmentation rose to 31.1% at \(K = 12\), against at most 11.8% for the archetypes.

Table 5.2: The demand clusters, found on the seasonal daily profiles in kW, numbered by mean demand: mean demand over the year, mean daily peak, mean demand in the evening and around midday, and the share of their households in each shape archetype.
Cluster Households Demand (kW) Share of households
Mean Daily peak 17:00 to 20:00 10:00 to 15:00 Archetype 1 Archetype 2 Archetype 3
Demand cluster 1 98 0.27 1.26 0.42 0.26 0.63 0.16 0.20
Demand cluster 2 122 0.44 1.75 0.69 0.40 0.43 0.28 0.29
Demand cluster 3 122 0.81 2.69 1.26 0.82 0.15 0.42 0.43

The three demand clusters are ordered by level (Table 5.2, Figure 5.3). Their mean demand is 0.27, 0.44 and 0.81 kW and their demand between 17:00 and 20:00 is 0.42, 0.69 and 1.26 kW, a threefold range, while all three share the evening peak of the archetypes. The demand clusters therefore cut across the archetypes rather than replacing them. The most peaked Archetype 1 makes up 63% of the lowest cluster and 15% of the highest, whereas the flattest Archetype 3 makes up 20% and 43%, so that a flatter use of the day is partly a property of larger households, yet every cluster contains all three archetypes.

Figure 5.3: Mean daily profiles in kW of the AusNet households by demand cluster. Thin grey lines are households; the orange line is the cluster mean. The vertical axis is cut at 3 kW, which the evening profiles of two households exceed.

Representative Profiles in Network Simulation

The preceding section measured how much of the risk a segmentation explains; network planning uses segmentations more directly. Planning studies seldom simulate the measured readings of every customer. They assign each customer a typical profile, often scaled to its billed consumption, and the accuracy of the study depends on how well the typical profiles stand in for the individual ones. This use therefore offers the most direct test of what a segmentation contributes to network analysis. HPK11 is simulated over the four seasonal weeks with random adoption of PV or EV chargers by 20 to 80% of customers, once with the measured readings and once with each of five sets of representative profiles. For a grouping \(g\) of the households, with \(g(i)\) the group of household \(i\) and \(E_i = \sum_t x_{i,t}\) its consumption over the year, the representative and scaled representative readings are

\[ r_{i,t} = \frac{1}{\lvert g(i) \rvert} \sum_{j \in g(i)} x_{j,t}, \qquad \tilde{r}_{i,t} = E_i \, \frac{1}{\lvert g(i) \rvert} \sum_{j \in g(i)} \frac{x_{j,t}}{E_j}, \tag{5.1}\]

so that the scaled profile keeps the shape of the group and restores the level of the household. The five sets are a single average profile for every household (one group), the mean profile of the household’s archetype, the same scaled, the mean profile of its demand cluster, and the same scaled. The adopters, array sizes and charging sessions are identical in every run, so that differences in outcome arise from the household profiles alone.

Table 5.3: Simulation of HPK11 with representative profiles in place of the measured readings of every household, over four weeks, one per season, with PV or EV adoption from 20 to 80% of customers at random: root-mean-square error of every LV transformer’s half-hourly loading and lowest and highest customer voltage against the measured replay, ratio of total violation hours to those of the measured replay, and the share of transformer, level and constraint combinations whose status against the criterion of violations in at most 1% of the time matches the measured replay.
Household profiles RMSE of the half-hourly series Violation hours, ratio to measured Criterion
Loading (pu) Lowest voltage (mpu) Highest voltage (mpu) PV over-voltage PV overload EV under-voltage EV overload Agreement
Single average 0.022 6.759 2.171 0.925 0.995 0.588 0.956 0.960
Archetypes 0.021 6.578 2.142 0.929 0.999 0.611 0.994 0.963
Archetypes, scaled 0.020 5.713 1.898 0.949 1.013 0.929 1.070 0.972
Demand clusters 0.020 6.251 1.941 0.942 1.001 0.671 0.946 0.969
Demand clusters, scaled 0.020 5.696 1.894 0.945 1.002 0.799 1.031 0.969

Table 5.3 compares the outcomes with those of the simulation with measured profiles, both as time series and as events. For a series \(y\) such as the loading of a transformer, with \(\hat{y}\) its value under representative profiles, the error is the root-mean-square difference over all transformers \(j\) and half-hours \(t\) of the simulated weeks; the violation hours are compared by the ratio of totals over the transformers; and the classification is compared by the share of transformers, levels and constraints on which the two simulations agree about the criterion of Equation 4.1, applied to the simulated period:

\[ \text{RMSE} = \sqrt{\frac{1}{\lvert \Omega \rvert} \sum_{(j, t) \in \Omega} (\hat{y}_{j,t} - y_{j,t})^2}, \qquad R = \frac{\sum_j \hat{D}_j}{\sum_j D_j}, \qquad A = \frac{1}{\lvert \mathcal{J} \rvert} \sum_{\mathcal{J}} \mathbb{1}\left[ \mathbb{1}[\hat{D}_j \le \theta] = \mathbb{1}[D_j \le \theta] \right], \tag{5.2}\]

where \(\Omega\) is the set of transformer half-hours, \(\mathcal{J}\) the set of transformer, level and constraint combinations, and \(\theta\) 1% of the 672 simulated hours. As time series, every set of representative profiles reproduced the loading of the transformers closely, with a root-mean-square error between 0.020 and 0.022 pu, and the lowest customer voltage within 5.7 to 6.8 thousandths of a per unit. The demand clusters reduced each error relative to the single average profile, and scaling to each household’s consumption reduced it further, to 0.0197 pu of loading and 5.70 thousandths of lowest voltage for the scaled demand clusters.

As events, the representative profiles differed markedly (Figure 5.4). The midday regimes were reproduced within 8%, with ratios \(R\) of over-voltage hours to the simulation with measured profiles between 0.925 and 0.949 and of PV-driven overload between 0.995 and 1.013. EV-driven under-voltage, the brief evening regime, was underestimated by every set: the single average profile produced 58.8% of the measured hours, the archetypes 61.1%, the demand clusters 67.1%, the scaled demand clusters 79.9% and the scaled archetypes 92.9%. Without Distributed Energy Resources (DER) the measured readings produced 14.0 hours of under-voltage over the four weeks, which the unscaled representatives reduced to at most 1.0 hour.

Figure 5.4: Total violation hours across the LV transformers of HPK11 over four simulated weeks, one per season, as PV or EV penetration rises, when every household replays its measured readings or a representative profile. Each panel has its own vertical scale.

The difference lies in what averaging removes. A representative profile is the mean of about a hundred households and is therefore smooth, whereas an evening under-voltage occurs when the peaks of many customers on one feeder coincide. Retaining the level of demand, through clusters in kW or through scaling to each household’s consumption, restores the spread of demand across customers and recovers part of the under-voltage, but the diversity of timing within a cluster is lost in every case (Figure 5.5). The two segmentations are complementary in this use. Where only profiles are available, the demand clusters reproduced the evening regime better than the archetypes or a single average, and where billed consumption supplies the level of each customer, the scaled shape archetypes reproduced it most closely. On the classification that informs hosting decisions, whether a transformer remains within the criterion of violations in at most 1% of the time, all sets agreed with the simulation with measured profiles in 96.0 to 97.2% of cases.

Figure 5.5: Schematic of why a representative profile misses the evening violations of coincident peaks. (a) The households of one cluster, each with an evening peak at its own time, and their mean, the representative profile. (b) The summed demand of six households on one transformer whose peaks coincide exceeds the demand \(L\) at which the voltage leaves the band, whereas the representative profile assigned to every household does not; scaling it to each household’s consumption restores the level of the sum but not the coincidence of the peaks.

Figure 5.6 shows this loss on the evening of lowest voltage in the simulated weeks, 16 January, with EV chargers at 80% of customers. Every set of representative profiles reproduced the timing of the evening and the loading of the transformer, which peaked at 1.35 to 1.39 times its rating against 1.38 for the measured readings, and every set breached the lower voltage limit. The measured readings, however, reached a lowest voltage of 0.841 pu, against 0.863 to 0.884 pu for the representative profiles, so that the representatives reproduce the occurrence of the violation but underestimate its depth.

Figure 5.6: Loading and lowest customer voltage of the HPK11 transformer whose customers reached the lowest voltage in the four simulated weeks, on that evening, 16 January, with EV chargers at 80% of customers, when every household replays its measured readings or a representative profile. The red lines mark the rating and the lower voltage limit of 0.94 pu; each panel has its own vertical scale.

Effect of Who Adopts

In every analysis so far, including the simulations of the preceding section, the adopters are fixed by a random draw, as in DSS-AU. Adoption in practice is neither random nor uniform, since it varies between areas with their building stock and socio-economic characteristics [5], and segments offer a natural means of describing who adopts. This section therefore holds the number of installations fixed and varies their allocation. For adoption by a share \(\ell\) of the \(N\) residential customers, from 10 to 25%, the \(n = \ell N\) adopters are drawn without replacement from an eligible set \(\mathcal{S}\): all customers for a random allocation, the customers of one archetype or of one demand cluster for a segment allocation, and, for a concentrated allocation, the customers of whole transformers taken in a random order until \(\lvert \mathcal{S} \rvert \ge n\), of whom the first \(n\) adopt, so that every chosen transformer but the last is fully equipped (Figure 5.7). The level of 25% is the largest that the smallest segment, with 1,514 customers, can supply. Each allocation is solved over the four seasonal weeks with two seeds.

Figure 5.7: Schematic of the four allocations of adoption compared in this section, each with four PV adopters among fifteen households on three transformers, in which the size of a house is its level of demand. (a) At random; (b) within one archetype, households of one shape at every level of demand; (c) within one demand cluster, here the highest; (d) concentrated on one transformer. The eligible households are boxed in the colour of their segmentation in Figure 5.8.
Figure 5.8: Total duration of violations across the LV transformers of HPK11 over four simulated weeks, one per season, as the share of customers who adopt PV or an EV charger rises, for seven ways of choosing the adopters. Each point is the mean of two seeds; each panel has its own vertical scale.

The geographical concentration of adoption changed the network response more than any behavioural allocation (Figure 5.8, Table 5.4). With PV at 25% of customers, random placement produced 1,515.5 hours of over-voltage on 40 transformers and 9.8 hours of overload on 3 transformers. Concentrated placement reduced the over-voltage to 1,298.0 hours, because it reached only 10.5 transformers on average, but raised the overload to 1,119.0 hours on 14.5 transformers. Concentrated EV adoption likewise raised under-voltage from 59.2 to 91.2 hours and overload from 90.8 to 115.0 hours, although the range between seeds of 70.5 to 159.5 hours for overload indicates that the effect depends on which transformers receive the chargers.

Table 5.4: Violations over the four simulated weeks with 25% of customers adopting, by allocation: total hours across the LV transformers of HPK11 and the number of transformers with at least one violation, each the mean of two seeds.
Allocation PV over-voltage PV overload EV under-voltage EV overload
Hours Transformers Hours Transformers Hours Transformers Hours Transformers
Random 1,515.5 40.0 9.8 3.0 59.2 14.5 90.8 12.5
Archetype 1 1,579.0 39.0 9.5 3.0 56.8 18.0 90.5 12.5
Archetype 2 1,453.8 37.0 6.5 3.0 63.5 13.5 89.5 12.0
Archetype 3 1,405.8 39.0 8.2 3.0 71.8 15.0 88.5 14.0
Demand cluster 1 1,794.8 40.5 9.5 3.0 56.5 16.0 91.2 12.0
Demand cluster 2 1,722.5 38.5 9.5 3.0 73.0 15.0 91.8 11.5
Demand cluster 3 1,206.2 40.0 8.8 3.0 66.5 15.0 91.2 13.5
Concentrated 1,298.0 10.5 1,119.0 14.5 91.2 15.0 115.0 13.5

Among the behavioural allocations, the demand clusters separated the outcomes far more than the archetypes. PV installed only by the households of the highest-demand cluster produced 1,206.2 hours of over-voltage, 20.4% less than random placement, whereas PV installed only by the lowest-demand cluster produced 1,794.8 hours, 18.4% more. The three archetypes spanned 1,405.8 to 1,579.0 hours, a range of 173.2 hours against 588.6 hours across the demand clusters. The direction agrees with the event-time demand of Demand at the Time of Violation, since households with a larger midday demand consume more of their own generation and export less of it (Figure 5.9). For EVs the allocation by segment made no consistent difference to overload, which ranged from 88.5 to 91.8 hours across the random and segment allocations, and only a modest one to under-voltage, from 56.5 to 73.0 hours, against a range between seeds of 51.0 to 67.5 hours under random placement.

Figure 5.9: Schematic of how the midday demand of a household offsets the export of its PV array. Two households share one array: the one with a low daytime demand exports most of its generation, and the one with a higher daytime demand uses more of it on site and exports less.

Implications for Network Operation

Taken together, the results bear on four activities of a Distribution System Operator (DSO), summarised in Figure 5.10. For connection assessment and reinforcement planning, the exposure of a transformer was determined by its loading ratio and its number of customers, which correlated with overload and over-voltage hours at 0.943 and 0.743, and by the geographical concentration of adoption, which raised PV-driven overload at 25% adoption from 9.8 to 1,119.0 hours over four weeks. The regimes of violation add a temporal dimension to these decisions. On HPK11, reverse-flow overload is absent below 40% PV penetration and becomes the largest regime at 80%, so that an assessment made at a low penetration does not reveal the constraint that later adoption will impose.

For network studies that cannot use every customer’s readings, clustering supplies the representative profiles, and the choice of segmentation matters more for the evening regime than for the midday ones. Profiles that retained each customer’s level of demand, through clusters in kW or through scaling to billed consumption, recovered 67.1 to 92.9% of the EV-driven under-voltage hours, against 58.8% for a single average profile. Since every set of representatives underestimated the brief evening violations, studies of EV hosting based on typical profiles should be read as lower bounds on the duration of under-voltage.

For tariff design and customer engagement, segments defined by the level of demand identify the households that matter. Confining PV adoption to the highest-demand cluster reduced over-voltage by 20.4% relative to random placement, which suggests that combining PV with daytime consumption, for instance through time-of-use incentives or storage, is most effective when aimed at larger households. The demand present at evening violations likewise followed the level of household demand rather than its shape, so that flexibility programmes for EV-driven under-voltage should select customers by consumption and location rather than by profile shape.

For monitoring, two uses of clustering complement the time-series record. The regimes of violation summarise a year of events per transformer into a small number of recognisable types, whose shares indicate which remedy a transformer requires as penetration changes. The shape archetypes, although they carry little information about risk, remain useful for detecting new behaviour, since forty households given an EV charging profile formed a pure archetype of their own, displacing an existing archetype unless the number of archetypes was allowed to grow by one. This use requires that the clustering be repeated on a schedule and with at least a season of history.

Table 5.5 summarises which product of clustering serves each decision, and the evidence for it in this chapter and the last.

Table 5.5: Products of clustering and the network decisions they serve, with the evidence for each.
Decision Product of clustering Evidence
Connection assessment and reinforcement Regimes of violation per transformer Reverse-flow overload absent below 40% PV and the largest regime at 80% (Regimes of Violation)
Network studies without every customer’s readings Representative profiles that retain the level of demand 67.1 to 92.9% of EV under-voltage hours recovered, against 58.8% for one average profile (Representative Profiles in Network Simulation)
Tariffs and engagement programmes Demand clusters (segments of the level of demand) 20.4% less over-voltage with PV in the highest-demand cluster (Effect of Who Adopts)
Flexibility programmes for EV charging Annual demand and location, not profile shape Event-time demand correlated with annual demand at 0.806 during under-voltage (Demand at the Time of Violation)
Monitoring of behaviour Shape archetypes, recomputed with \(K + 1\) evaluated EV charging formed a pure archetype, displacing an existing one at a fixed \(K\) (Chapter 3)
Figure 5.10: What clustering supplies to four activities of a DSO. (a) Planning: the same number of arrays concentrated on one transformer overloads it. (b) Network studies: the measured evening peak crosses a limit that representative profiles stay below. (c) Tariffs and engagement: the households to target are the larger ones, for daytime use with PV or flexibility with an EV. (d) Monitoring: the shares of the regimes of violation as adoption grows, with reverse-flow overload emerging. The quantities of (a) and (d) are schematic.

Conclusion

This chapter evaluated three uses of segmentation in network operation. Applied to profiles in kW, the pipeline of Chapter 2 produced three demand clusters that explained 41.0 to 53.9% of the demand during the violations of a held-out half-year and 18.8% of the peak contribution, as much as the terciles of annual demand. A segmentation built from summer violations explained the summer peak contribution best but failed in winter, which shows that a segmentation derived from network outcomes requires out-of-sample evaluation. In network simulation, representative profiles that retained the level of demand reproduced the evening under-voltage better than a single average profile, although every set underestimated its depth. As allocations of adoption, the demand clusters separated the over-voltage outcome 3.4 times more widely than the archetypes, while the geographical concentration of adoption changed the response more than any behavioural allocation.

These conclusions are subject to several limitations. The households are placed on the network by a random draw, so the analysis cannot represent geographical clustering of behaviour, and the network is a single urban feeder with a pseudo LV topology. The direct simulations cover four weeks, one per season, with two seeds per configuration, and the attribution scenarios of Chapter 4 equip every customer, which isolates demand and location at the cost of realism. The regimes of violation are derived from one network and one year of weather, and all analyses use the demand of one population.

The results also define what the next chapter must address. The demand clusters reached only the explanatory power of a simple tercile of annual demand, and no segmentation approached the 69.6% of the peak contribution that the household itself explains. Representative profiles, moreover, lost the diversity of timing that produces brief evening violations. Chapter 6, in preparation, examines representations of a household that retain magnitude and describe its demand across several time scales, and asks whether the groupings they produce close the gap between segment and household.

Key findings

Segments that retain the level of demand captured the risk that shape archetypes miss: three demand clusters explained about half of the demand present during held-out violations. Representative profiles reproduced the midday regimes closely and the brief evening under-voltage only in part, best when they retained each customer’s level. Where adoption concentrated mattered more than which segment adopted, and among segments, the level of demand mattered more than its shape. A segmentation derived from network outcomes must be evaluated on outcomes it did not see.

Exercises

  1. In the segmentations cell of the companion notebook, evaluate every segmentation at \(K = 5\) instead of the common \(K\). Does the ranking of Table 5.1 change?
  2. Using the simulated records of the replay cell, find the evening of lowest voltage under EV charging at 40% rather than 80%. Do the representative profiles still breach the lower limit on that evening?
  3. Add to the adoption cell an allocation confined to the households with the highest demand at the half-hour of the network peak. How does it compare with the highest-demand cluster?

References