| Dataset | Country | Households | Resolution (min) | First day | Days |
|---|---|---|---|---|---|
| London | United Kingdom | 1,284 | 30 | 2013-01-01 | 360 |
| GoiEner | Spain | 1,000 | 60 | 2021-06-06 | 360 |
| AusNet | Australia | 342 | 30 | 2020-01-01 | 360 |
Validating a Clustering Pipeline
Chapter 1 left four questions open: what makes two customers members of the same group, whether a load curve needs a specialised distance and representation, how a grouping can be validated without labels, and how physical knowledge can constrain the clustering itself. On a synthetic population with planted archetypes each question had a definite answer, because the answer was known in advance. On metered data it is not, and this chapter takes up the first three questions there.
The chapter follows the sequence in which a clustering pipeline is built. It first clusters the raw daily curves, then settles a feature-based pipeline, and then tests that pipeline with every instrument of Chapter 1: agreement between algorithms, resampling stability, calibration against data without structure, graph connectivity and, where the data permit, an external attribute. A recurring result is that each quantitative rule applied along the way misled at some point about this population, and that each correction came from outside the rule: from the cluster sizes, the archetype curves, a connectivity check or a tariff column. This observation, rather than any particular procedure, is the principal finding of the chapter.
The three populations of the part are used throughout (Table 2.1): 1,284 households from the Low Carbon London trial, 1,000 from the Spanish cooperative GoiEner and 342 from the AusNet network in Victoria. Applying each comparison to all three populations makes generalisation a property that each finding either has or lacks, reported where the finding is made rather than in a closing section.
Clustering the Raw Curves
The simplest starting point, before any feature engineering, is to cluster the load curves themselves, which tests the second question of Chapter 1 directly. Two distances are compared on the peak-normalised daily profiles of the London households: Euclidean distance, which compares hour with hour, and dynamic time warping, which first aligns the two curves and so tolerates a household that performs the same activity at a different time [1].
Warping performed worse on every separation index (Table 2.2). At the number of clusters each distance recommends, Euclidean k-means reached a density silhouette of 0.259 against 0.178 for dynamic time warping, and a Davies-Bouldin index of 1.32 against 1.93. The cluster means in Figure 2.1 indicate the reason. Household demand is anchored to the clock by work, school and tariff periods, so a peak at seven and a peak at nine are two behaviours rather than one behaviour observed with an offset, and aligning them removes the distinction that the clustering is intended to find. Dynamic time warping is suited to sequences in which phase is a nuisance; in a daily load profile, phase is the signal. This confirms on metered data the result that Chapter 1 obtained on the synthetic population.
A Feature-Based Pipeline
With the Euclidean distance retained, the remaining choice is the representation to which it is applied. The features used from here on are seasonal profiles, four seasons by twenty-four hours, computed after dividing each meter’s readings by its own daily maximum, so that each of the 96 features describes when a household consumes rather than how much.
Pitfall: inheriting a normalisation decision from a library default
Whether features are peak-normalised changes what a clustering finds. During the work behind this chapter, a library default for that option changed, nothing in the calling code set it explicitly, and every profile feature set silently began to describe how much each household consumed rather than when. Every validity index still returned a plausible number. A decision of this consequence belongs in the calling code, stated explicitly, where a change to a library cannot revoke it.
The features are projected to two dimensions with t-SNE at perplexity thirty, within the range van der Maaten and Hinton suggest for a population of this size [2]; a later section tests this choice. t-SNE places each household at a point \(\mathbf{z}_i\) of the plane so that the similarities of the households in the embedding match those in the feature space, by minimising the Kullback-Leibler divergence
\[ \mathrm{KL}(P \,\|\, Q) = \sum_{i \neq j} p_{ij} \log \frac{p_{ij}}{q_{ij}}, \qquad q_{ij} = \frac{(1 + \lVert \mathbf{z}_i - \mathbf{z}_j \rVert^2)^{-1}}{\sum_{k \neq l} (1 + \lVert \mathbf{z}_k - \mathbf{z}_l \rVert^2)^{-1}}, \tag{2.1}\]
where \(p_{ij}\) is the symmetrised similarity of households \(i\) and \(j\) under a Gaussian kernel whose bandwidth for each household is set by the perplexity. The heavy-tailed kernel of \(q_{ij}\) preserves neighbourhoods rather than global distances, which is why the embedding is clustered rather than the distances read from it. Figure 3.1 shows how eight internal indices rank the number of clusters on the resulting embedding. Each preferred a different value, which is the usual situation and the reason Chapter 1 cautioned against relying on any single index [3], [4], [5], [6]. A weighted Borda ranking combines eleven indices and down-weights those that correlate strongly with one another, so that a group of near-identical separation measures cannot outvote an index that measures something different. For indices \(m = 1, \dots, M\), each oriented so that higher is better, let \(\pi_m(K)\) be the rank of \(K\) among the candidate values \(\mathcal{K}\) under index \(m\), largest for the best, and \(\bar{\varrho}_m = \frac{1}{M} \sum_{m' \neq m} \lvert \varrho_{m m'} \rvert\) the mean absolute Spearman correlation of index \(m\) with the others across the candidates. Each candidate scores
\[ B(K) = \sum_{m=1}^{M} w_m \, \pi_m(K), \qquad w_m = \frac{\max(1 - \bar{\varrho}_m,\ 0.05)}{\sum_{m'} \max(1 - \bar{\varrho}_{m'},\ 0.05)}, \tag{2.2}\]
and the recommended \(K\) maximises \(B\). It recommended \(K = 11\), ahead of 12 and 8 (Table 8.2).
At \(K = 11\) the clusters contain between 56 and 144 households. Figure 2.3 places them in the embedding and Figure 2.4 shows their daily profiles. The profiles provide the check that no index performs: a high Borda score indicates that the groups are separated in the embedding, whereas only the curves show whether the separation corresponds to behaviour that a network planner would recognise.
What the Feature Step Changes
Whether the feature step improves on the raw curves cannot be decided by comparing silhouettes, because the two are computed in different spaces. The comparison that remains valid is between the partitions themselves, through the adjusted Rand index, which requires no shared space. The feature partition agreed with the Euclidean raw-curve partition at an ARI of 0.070 and with the warped one at 0.063, while the two raw-curve partitions agreed with each other at 0.226 (Table 2.4). Figure 2.5 colours the same embedding by each partition, so that every difference between its panels is a difference in labelling. The pipelines therefore produce different segmentations rather than two readings of one segmentation that differ at the margins, and the seasonal dimension is what separates them.
Agreement Between Algorithms
The pipeline so far fixes k-means. Seven algorithms are therefore evaluated for \(K\) from 2 to 12 on the same embedding, each scored by the same Borda ranking (Table 2.5). Five of them, k-means, mini-batch k-means, the Gaussian mixture, Ward linkage and BIRCH, settled on \(K = 11\), whereas average linkage and spectral clustering settled on \(K = 2\). The division follows the geometry that each algorithm assumes. Methods that partition on distance to a centre find many roughly equal regions in an embedding of a continuous manifold, whereas average linkage merges on the smallest mean pairwise distance and spectral clustering cuts a nearest-neighbour graph, so both are drawn to the largest-scale division available.
The agreement of five algorithms is weaker evidence than it appears. The five share an objective and are scored by the same ensemble on the same embedding, so their agreement is closer to a restatement than to five independent confirmations. The two dissenting algorithms are the more informative, and the remainder of the chapter returns to them.
Breaking the Tie with Resampling
Internal indices cannot settle a disagreement between partitions that all score well, because they all measure the same two quantities on the same embedding. The resampling measures of Chapter 1 supply evidence of a different kind. Tibshirani and Walther’s prediction strength (Equation 1.10) asks whether a clustering fitted on one half of the population predicts the co-membership of the other half [7], and Hennig’s cluster-wise stability (Equation 1.11) asks whether each cluster is recovered under bootstrap resampling [8]. The trust-gated composite score combines this evidence with separation. It multiplies a separation score by a trust factor,
\[ C(K) = S(K) \cdot T(K), \qquad S(K) = \frac{B_{\text{sep}}(K)}{\lvert \mathcal{K} \rvert}, \qquad T(K) = \min \Big\{ H(\mathbf{p}_K),\ \min_{k} \operatorname{stability}(C_k) \Big\}, \tag{2.3}\]
where \(B_{\text{sep}}\) is the weighted Borda score of Equation 2.2 over the silhouette, Caliński-Harabasz and Davies-Bouldin indices, \(H(\mathbf{p}_K)\) the balance of Equation 1.8 at \(K\), and the inner minimum runs over the clusters of the partition. A candidate therefore scores well only if it separates well, is balanced and reproduces, and no amount of separation compensates for a missing trust signal.
Read without calibration, the result appeared decisive (Table 2.6). Every algorithm that chose \(K = 11\) reached a prediction strength between 0.35 and 0.42, well below the 0.8 that Tibshirani and Walther suggest, while spectral clustering and average linkage at \(K = 2\) reached 0.959 and 0.891. The trust-gated composite score accordingly ranked the two dissenting algorithms first, which suggested that the consensus of five does not reproduce and that the dissenters had found the only division that does.
Calibration Against a Structureless Reference
That reading assumes that prediction strength is comparable across \(K\), and there is a structural reason to doubt it. Prediction strength is a minimum of co-membership agreement over the test clusters, and both parts of that construction depend on \(K\). Under random assignment two points share a cluster with probability of roughly \(1/K\), so the chance level is near 0.5 at \(K = 2\) and near 0.09 at \(K = 11\), and a minimum over eleven clusters gives eleven opportunities for one weak cluster to set the score, against two.
The dependence can be measured directly by running the same checks on reference data with the size and extent of the embedding but no cluster structure, uniform over its bounding box, which is the reference Tibshirani, Walther and Hastie use for the gap statistic [9]. A third check joins the two: Ben-Hur, Elisseeff and Guyon’s subsample stability, the mean adjusted Rand index (Equation 1.12) between the partitions of pairs of random subsamples on the households they share, which is chance-corrected within the index itself [10]. For each check \(m\), with value \(m(K)\) on the data and \(m_0^{(r)}(K)\) on the \(r\)-th of \(R\) reference draws, the excess over the reference is
\[ \Delta_m(K) = m(K) - \frac{1}{R} \sum_{r=1}^{R} m_0^{(r)}(K), \qquad \bar{\Delta}(K) = \frac{1}{3} \sum_{m} \Delta_m(K), \tag{2.4}\]
the mean excess being taken over prediction strength, cluster-wise stability and subsample agreement. Figure 2.6 shows each check on the London embedding and on the reference.
On structureless data, prediction strength reached 0.949 at \(K = 2\) and 0.376 at \(K = 11\). The threshold of 0.8 is therefore not one standard applied to different candidates: noise clears it comfortably at small \(K\) and cannot clear it at large \(K\).
Remark: comparing resampling evidence across K
Subtracting the value a check attains on a structureless reference of the same shape converts an uninterpretable level into an interpretable difference. The excess has the same meaning at every \(K\): how much more reliably this population partitions than data without structure. It costs a few additional fits, and it is the form in which these checks can be compared between different values of \(K\).
Corrected in this way, the verdict reversed. At \(K = 2\) the excess was indistinguishable from zero on all three checks, between \(-0.009\) and \(0.001\): a two-way split of the London embedding reproduces as reliably as a two-way split of random points, and its reproducibility is therefore uninformative. Six values of \(K\) had a positive excess on all three checks at once, namely 5 and 7 to 11. Applied to each algorithm at its own recommended \(K\) (Table 2.7), the correction left k-means and the Gaussian mixture at \(K = 11\) as the only candidates with a positive excess on all three checks, while both \(K = 2\) candidates lost cluster-wise stability relative to the reference, by 0.147 for average linkage and 0.251 for spectral clustering.
The reference also qualifies what stability means. On structureless data, subsample agreement never fell below 0.738. Two independent subsamples of uniform noise, clustered separately, agree substantially about which points belong together, because k-means partitions noise as reproducibly as it partitions anything. Stability therefore measures reproducibility rather than structure, and the two coincide only when there is structure to reproduce.
The Overnight Cohort
The corrected evidence scored the \(K = 2\) split at zero excess, and its balance of 0.259 marks it as isolating a corner of the population rather than dividing it. The split nevertheless merits examination before that conclusion is accepted, because what the two dissenting algorithms separated can be checked directly.
The nearest-neighbour graph of the embedding has two connected components, and still has two at fifty neighbours; only at a hundred does the smaller one join the rest (Table 2.8). Spectral clustering was therefore not misconfigured. A graph with two disconnected components has two zero eigenvalues in its Laplacian, so a spectral method must separate them before it can find anything else [11]. The smaller component holds 56 households and coincides exactly with the \(K = 2\) partition of the dissenting algorithms, at an ARI of 1.000. Nor is the split an artefact of t-SNE: it is better separated in the original 96-dimensional feature space, with a silhouette of 0.414, than in the two-dimensional embedding, at 0.369, so the reduction did not create it.
The 56 households form a recognisable behavioural group (Table 2.9, Figure 2.7). Fifty of them reach their daily peak between midnight and one in the morning, against fewer than 1% of the rest of the population, whose median peak falls at seven in the evening. Sixty per cent of their daily consumption falls before six in the morning, against 15% for the rest, and their mean daily demand of 9.7 kWh is roughly double. A sharp overnight block followed by little consumption during the day is the signature of electric storage heating charged on an off-peak tariff.
The trial’s own tariff column supports this reading independently (Table 2.10). Every one of the 56 households is on the standard tariff and none is in the dynamic time-of-use arm, against a time-of-use rate of 18.6% across the rest of the population, which would have predicted about ten of them; Fisher’s exact test gives \(p \approx 2 \times 10^{-5}\). Their overnight peak therefore cannot be a response to the trial’s price signal, because none of them received it. It is pre-existing behaviour, consistent with a legacy off-peak meter of the kind on which a dynamic tariff cannot be layered, which is the most likely reason that none of them was enrolled.
Pitfall: trusting a correction beyond the problem it corrected
Excess over a structureless reference scored this split at zero, and the arithmetic was correct. Any two-way split reproduces well, so a genuine two-way structure earns no more excess than an arbitrary cut, and a balance gate rejects a real minority archetype exactly as it would reject a fabricated one. Three defensible criteria all dismissed a group that a single tariff column confirms.
Detecting a genuine minority archetype therefore requires a different instrument from those that rank partitions: graph connectivity, a density method, or inspection of the curves. For a distribution operator, a load that already runs overnight and is thermally buffered is among the most valuable sources of flexibility on the network, and a centroid method asked for eleven balanced regions cannot return a group that holds 4% of the population.
Four Objective Families on Metered Data
The seven algorithms compared above do not represent seven independent definitions of a cluster. Chapter 1 distinguished four objective families, and two of them, density-connectivity and graph partitioning, were represented only by the dissenting algorithms. Both are now applied at the settled \(K\) on the same embedding, with DBSCAN’s radius \(\varepsilon\) read at the elbow of the sorted distances from each household to its \(k\)-th nearest neighbour, as Ester, Kriegel, Sander and Xu recommend [12].
The density method stood apart (Figure 2.8). With its radius at the k-distance elbow, DBSCAN found two clusters and no noise points, and its partition agreed with every other family at an ARI between 0.019 and 0.021, the level expected of an independent partition. The partition, probabilistic, hierarchical and graph methods agreed with one another at 0.666 to 0.839, and the eigengap of the spectral method’s graph Laplacian independently estimated eleven clusters. Under a density definition of a cluster, a t-SNE embedding of a continuous behavioural manifold contains few separated dense regions, and DBCV, a density-based validity index, was negative at the recommended \(K\), at \(-0.275\) [13]. The separation indices of the earlier sections did not register this, because they measure separation rather than density.
Three Populations
Every result so far was obtained on London. Table 2.11 applies the same pipeline, unchanged, to GoiEner and AusNet, and Figure 2.9 shows the three embeddings; the number of clusters that each algorithm recommends on each population is listed in the companion notebook.
The results fall into three groups. The independence of DBSCAN generalised fully: its agreement with k-means was 0.019 on London, 0.011 on GoiEner and 0.000 on AusNet, and the density-based DBCV index was negative at the recommended \(K\) on all three populations [13]. The recommended number of clusters did not generalise, and neither did the confidence behind it. London and AusNet settled on eleven clusters and GoiEner on ten, an apparent agreement that the per-algorithm results qualify: five of the seven algorithms agreed on the modal value for London, three for GoiEner and only two for AusNet, whose 342 households provide the least evidence from which to form a consensus. One London finding did not survive at all. The division between algorithms that prefer a fine partition and a pair that prefers the coarsest was particular to London: average linkage chose 2, 7 and 2 clusters across the three populations, and spectral clustering 2, 11 and 11. The coarse split was a response to London’s disconnected cohort rather than a standing preference of either algorithm. Chapter 3 returns to the number of clusters on AusNet with calibrated evidence.
Auditing the Fixed Choices
Every result above came from one pipeline whose feature set, absence of a scaling step and reduction method were fixed at the outset and not revisited. They can be audited now because the first half of the chapter produced the instruments needed to do so. Silhouette cannot compare these conditions, since each changes the space in which the clustering takes place. Three measures can: the adjusted Rand index (Equation 1.12) against the settled partition of each population, cluster balance (Equation 1.8) and size, and the recall (Equation 1.14) of the overnight cohort on London, a structure established independently of any of the pipelines being compared. Each audit varies one choice and holds the rest at the settled pipeline.
Scaling
The pipeline applies no scaling to the features. Since each feature is an hourly mean of a peak-normalised series, the columns already share a bounded scale, but their spreads differ, and a distance treats low-variance hours as less important.
The scaling step left every separation index nearly unchanged and rewrote a large part of the partition (Figure 2.10). Across the six conditions, the embedded silhouette varied by at most 0.023 on London, 0.013 on GoiEner and 0.011 on AusNet, and balance stayed above 0.96 throughout, so a choice made from a table of validity indices would treat the six as equivalent. Their partitions are not equivalent: standardising the features lowered the agreement with the unscaled partition to 0.549 on London, 0.624 on GoiEner and 0.424 on AusNet, and the quantile transform reached 0.374 on AusNet. The indices do not measure what the scaling step changes.
The unscaled pipeline scores 1.000 against itself by construction, so the agreement column cannot rank the conditions. Cohort recovery can, because it measures agreement with a structure established independently of all six. No scaling, min-max scaling and robust scaling each recovered the overnight cohort exactly; standardising recovered 95% of it, the power transform 91% and the quantile transform 83%. The ordering has a mechanism: a quantile transform maps each feature onto a uniform distribution and so compresses the extreme overnight values that make the cohort distinctive. Since the peak-normalised features already share a bounded scale, no scaler improves on omitting the step, and the pipeline keeps the unscaled features, now as a choice supported by evidence. The conclusion is specific to this representation; a feature set that mixes kilowatt-hours with ratios or hours would reopen the question.
Feature Sets
The seasonal profile is one of ten feature sets. Three are whole-profile representations that differ in what they average over (the seasonal profile, a weekly profile by day of week and hour, and both together), and seven compress a household to a few summary statistics of its magnitude, volatility, trend, shape, autocorrelation, calendar pattern or distribution. These are not competing estimates of one quantity: a representation determines which structure a clustering can find.
Ranked by silhouette, the summary statistics led (Table 2.12): shape reached 0.667 and magnitude 0.521, against 0.402 to 0.412 for the three whole-profile representations. Much of that ranking reflects dimensionality rather than quality. shape has three columns and volatility two, against 96 to 264 for the profiles, and distances concentrate as dimension grows. The adjusted Rand index shows that the summary statistics are not weaker versions of the profile partition but are nearly unrelated to it, at no more than 0.071 on London and similarly low values on the other two populations (Figure 2.11), whereas the weekly and combined profiles agreed with the seasonal partition at 0.569 and 0.625. magnitude sorts households by how much they consume and volatility by how erratic they are, and neither has any reason to reproduce a grouping by when a household uses electricity.
Cohort recovery inverted the silhouette ranking. Only the three whole-profile representations recovered the overnight cohort intact; shape recovered two-thirds of it, and the remaining six recovered between 7% and 25%. The two representations with the highest silhouette are among those that dissolve a group for which independent evidence exists. A feature set is therefore not better or worse in the abstract: it determines which question a clustering answers, and where a known structure exists, testing whether a representation preserves it is more informative than any index.
Dimensionality Reduction
The reduction step was fixed to t-SNE, and the overnight cohort was found because t-SNE’s neighbourhood graph left its households in a component of their own. Six conditions are compared: no reduction, principal component analysis, kernel PCA [14], t-SNE, UMAP [15] and PaCMAP [16]. Because each produces a different space, the silhouette is reported both within each embedding and in the shared feature space.
The two silhouettes disagreed (Table 2.13). Measured in each embedding, reduction helped: no reduction scored 0.097 on London against 0.348 to 0.427 for the reduced embeddings. Measured in the shared feature space, the order reversed, with no reduction highest at 0.097 and the three manifold methods lowest at 0.051 to 0.062. The embedded silhouette rewards having fewer dimensions in which to separate, the same effect that favoured the three-column shape, so it cannot settle the question. Cluster sizes can. A linear projection asked for eleven groups returned clusters of five households on London and two on AusNet, while the three manifold methods returned no cluster smaller than 19 on any population. No separation index objects to a cluster of two households, because two points far from everything else are perfectly compact and perfectly separated.
The connectivity check of the ten-nearest-neighbour graph, tabulated in the companion notebook, explains the difference. Only t-SNE, UMAP and PaCMAP left the overnight cohort as a disconnected component of the ten-nearest-neighbour graph, and each isolated exactly the same 56 households, at an ARI of 1.000 against the cohort. Without reduction, and under PCA and kernel PCA, the graph has a single component. The cohort is not invisible without a manifold method, since k-means recovered 91% of it on the unreduced features and 88% under PCA, but only the manifold methods make it topologically distinct. Figure 2.12 shows the six embeddings coloured by the settled partition. Among the manifold methods the choice is nearly free, since they agreed on the cohort exactly and their balance on London differed by 0.003; the decision that matters is between manifold and linear reduction.
Retained Components
Each reduction above kept two components, the number a scatter plot can show, which is a presentation constraint rather than an argument. Chapter 1 introduced two rules for choosing the number: the 90% variance rule and Zhu and Ghodsi’s profile likelihood [17].
The two rules disagreed widely. The variance rule asked for 11, 8 and 14 components on the three populations, and profile likelihood for one on all three. The disagreement had little practical consequence (Figure 2.13). Under UMAP, increasing the retained components from two to twenty changed the balance on London by at most 0.002, cohort recovery stayed at 1.000 except at ten components, where it was 0.982, and the smallest cluster held 55 or 56 households throughout. Under PCA the result was worse rather than different: the smallest cluster on AusNet held two households at two components and again at twenty, and cohort recovery never exceeded 91%. The number of retained components is a minor choice; the family of reduction is the major one. Profile likelihood’s answer of one component reflects the shape of these spectra, a single dominant direction followed by a smoothly decaying tail; the first two components already hold 63.8% of the variance on London, 73.8% on GoiEner and 53.8% on AusNet.
Interaction Between Choices
Each audit varied one choice with the others fixed, so the feature-set ranking assumed t-SNE and the reduction ranking assumed the seasonal profile. Crossing four feature sets with three reducers tests whether the two choices interact. They interacted little (Figure 2.14). shape ranked first and volatility second under every reducer on every population; on London the order was identical under all three reducers, and on GoiEner and AusNet the only change was the seasonal and weekly profiles exchanging third and fourth place under one reducer.
The stability of this ranking is less reassuring than it appears. The ranking is measured by silhouette in the shared feature space, which the feature-set audit showed to be dominated by the number of columns, and the number of columns does not change with the reducer. A confounded measure produces a stable ranking whether the choices interact or not. The more telling observation is that the ranking is both stable and inverted with respect to cohort recovery: the profile sets ranked third and fourth here are the only representations that preserve the cohort. Where settling one choice at a time failed, it failed because the objective optimised at each step was inappropriate, not because the choices interact.
A Joint Search
A search over the whole space at once is the usual response to interacting choices, and its value here is narrower than is often claimed. A genetic algorithm explores combinations that a grid would miss, but its main advantage is that it optimises a deliberately chosen objective across the whole space. NSGA-II searches over feature set, scaler, reduction method with its own hyperparameters, clustering algorithm and \(K\) from 2 to 12, maximising three objectives together: the density silhouette, the balance of the cluster sizes and a resampling stability proxy [18], [19]. None of the three is trustworthy alone, which is the reason for optimising them jointly rather than combining them into one weighted score.
Three hundred trials produced a Pareto front of 22 trials (Table 2.14, Figure 2.15), and none of them improved on the settled pipeline. Most of the front consists of two-way splits of almost exactly half the population each, with a balance of 1.000: asked to maximise balance, the search cut the population in half along whichever direction was available. The other end of the front reproduced the limitation identified in Chapter 1. Trial 109, the shape features through kernel PCA at \(K = 2\), reached a silhouette of 1.000 with a balance of 0.009, and trials 234 and 132 lay beside it at 0.992 and 0.984 with balances of 0.017 and 0.021. They belong to the front because no trial exceeds them on separation, which is what non-domination implies when one objective is insensitive to population share [20].
The audit in Table 2.15 applies the calibrated checks to the settled pipeline and to the five most balanced trials of the front. The settled pipeline reached a cluster-wise stability excess of 0.174 and recovered the overnight cohort exactly. The five trials split the population into two groups of 634 to 640 against the rest, recovered between 8% and 9% of the cohort, and four of the five had a negative prediction-strength excess, meaning that they reproduce no better than uniform noise at their own \(K\).
The importance decomposition explains the shape of the front (Figure 2.16). Functional analysis of variance over a random-forest surrogate attributed to \(K\) the largest share of the variation in the silhouette, 0.46, and most of the variation in the stability proxy, 0.77, more than all other parameters together [21]. Balance depended instead on the representation: fANOVA attributed 0.46 to the choice of reducer, while PED-ANOVA, an evaluator designed to be steadier under uneven trial counts, placed kernel PCA’s kernel first at 0.28 and the reducer second at 0.18 [22]. The two evaluators agree that \(K\) and the reduction step dominate and differ in how they divide the credit, so the ranking is better determined than any single value.
The search found the non-dominated set of the objectives it was given, over a space that no grid could cover. Its result shows that a search inherits the blind spots of its objectives and amplifies them, because it optimises exactly the quantities it is given.
Conclusion
This chapter applied the instruments of Chapter 1 to three metered populations and answered three of its questions. On the second question, a load curve does not need a specialised distance: dynamic time warping performed worse than Euclidean distance on London because the timing of demand is the information that distinguishes households. The representation that preserved the population’s one verifiable structure was a peak-normalised seasonal profile, unscaled, passed through a neighbourhood-preserving reduction at two components; within that choice the specific reducer and the number of components mattered little, while the choice between manifold and linear reduction mattered greatly.
On the third question, no single index, and no set of indices sharing the same limitation, validated a grouping on its own. Uncalibrated resampling favoured the coarsest split for structural reasons, and calibration against a structureless reference reversed that verdict, yet even calibrated evidence scored a genuine minority archetype at zero. The overnight cohort of 56 London households is the most instructive instance. Silhouette ranked the representations that dissolve it above those that preserve it, the calibrated excess scored its split at zero, and balance rejected it, while graph connectivity and the trial’s tariff column established that it is real. The first question, which criterion defines a group, therefore received a qualified answer: centroid methods at \(K = 11\) provide a reproducible segmentation of the London population, and a connectivity or density method is needed alongside them to find the minority groups that such a segmentation absorbs.
A joint search over the whole pipeline did not improve on these choices, because it optimised the same objectives whose limitations the chapter had exposed. The conclusion is not that the measures should be abandoned but that each number should be read together with what it describes: the cluster sizes, the archetype curves and, where available, an external attribute. Table 2.16 condenses this practice into a protocol. Chapter 3 applies the settled pipeline, together with this protocol, to the AusNet population, and Chapters 4 and 5 to the network that its households supply.
| Step | Check | Failure it guards against |
|---|---|---|
| 1 | State the representation, including its normalisation, in the calling code | A silent change in what is clustered |
| 2 | Rank \(K\) with several internal indices, combined by a decorrelated ranking | Reliance on a single index |
| 3 | Inspect the cluster sizes and their balance | Compact, distant minorities rewarded by separation indices |
| 4 | Calibrate each resampling check against a structureless reference | Resampling evidence that favours small \(K\) by construction |
| 5 | Check the connectivity of the nearest-neighbour graph and apply a density method | Genuine minority groups absorbed by centroid methods |
| 6 | Compare pipelines by the agreement of their partitions, not by indices computed in different spaces | Comparisons that change the space they measure |
| 7 | Validate against an external attribute or an independently established structure | Partitions that are reproducible but not meaningful |
| 8 | Repeat the analysis on a second population | Findings particular to one population |
Key findings
On metered data, dynamic time warping performed worse than Euclidean distance, because the timing of demand is the signal. Internal indices disagreed with one another, and uncalibrated resampling favoured a two-way split that reproduced no better than noise; calibrated against a structureless reference, the evidence supported eleven clusters on London. A cohort of 56 households whose overnight peak is the signature of storage heating, confirmed by the trial’s tariff column, was dismissed by every ranking criterion and found only by graph connectivity. Manifold reduction preserved this cohort where linear reduction and summary features did not, and a joint search over the whole pipeline found nothing better than the settled one.
Exercises
- Compute the connectivity table of the
tbl-connectivitycell on the GoiEner and AusNet embeddings. Does either population contain a disconnected group, and if so, what does its mean daily profile show? - Repeat the scaler audit of the
scalerscell with the weekly profile instead of the seasonal profile. Does the ordering of the scalers by cohort recovery persist? - Restrict the joint search of the
searchcell to \(K \ge 4\). What does the Pareto front then consist of, and does any trial recover the overnight cohort?