Validating a Clustering Pipeline

Chapter 1 left four questions open: what makes two customers members of the same group, whether a load curve needs a specialised distance and representation, how a grouping can be validated without labels, and how physical knowledge can constrain the clustering itself. On a synthetic population with planted archetypes each question had a definite answer, because the answer was known in advance. On metered data it is not, and this chapter takes up the first three questions there.

The chapter follows the sequence in which a clustering pipeline is built. It first clusters the raw daily curves, then settles a feature-based pipeline, and then tests that pipeline with every instrument of Chapter 1: agreement between algorithms, resampling stability, calibration against data without structure, graph connectivity and, where the data permit, an external attribute. A recurring result is that each quantitative rule applied along the way misled at some point about this population, and that each correction came from outside the rule: from the cluster sizes, the archetype curves, a connectivity check or a tariff column. This observation, rather than any particular procedure, is the principal finding of the chapter.

The three populations of the part are used throughout (Table 2.1): 1,284 households from the Low Carbon London trial, 1,000 from the Spanish cooperative GoiEner and 342 from the AusNet network in Victoria. Applying each comparison to all three populations makes generalisation a property that each finding either has or lacks, reported where the finding is made rather than in a closing section.

Table 2.1: The three metered populations of this chapter, after removing meters with insufficient coverage.
Dataset Country Households Resolution (min) First day Days
London United Kingdom 1,284 30 2013-01-01 360
GoiEner Spain 1,000 60 2021-06-06 360
AusNet Australia 342 30 2020-01-01 360

Clustering the Raw Curves

The simplest starting point, before any feature engineering, is to cluster the load curves themselves, which tests the second question of Chapter 1 directly. Two distances are compared on the peak-normalised daily profiles of the London households: Euclidean distance, which compares hour with hour, and dynamic time warping, which first aligns the two curves and so tolerates a household that performs the same activity at a different time [1].

Table 2.2: k-means on the raw daily curves of the 1,284 London households under Euclidean distance and dynamic time warping, at the K each recommends.
Distance Recommended K Density silhouette Calinski-Harabasz Davies-Bouldin1 Balance Smallest cluster
Euclidean 3 0.259 478.785 1.321 0.794 73
DTW 2 0.178 296.246 1.926 0.984 548
1 Lower is better; for every other index higher is better.

Warping performed worse on every separation index (Table 2.2). At the number of clusters each distance recommends, Euclidean k-means reached a density silhouette of 0.259 against 0.178 for dynamic time warping, and a Davies-Bouldin index of 1.32 against 1.93. The cluster means in Figure 2.1 indicate the reason. Household demand is anchored to the clock by work, school and tariff periods, so a peak at seven and a peak at nine are two behaviours rather than one behaviour observed with an offset, and aligning them removes the distinction that the clustering is intended to find. Dynamic time warping is suited to sequences in which phase is a nuisance; in a daily load profile, phase is the signal. This confirms on metered data the result that Chapter 1 obtained on the synthetic population.

Figure 2.1: Cluster means of the raw daily curves under (a) Euclidean distance and (b) dynamic time warping.

A Feature-Based Pipeline

With the Euclidean distance retained, the remaining choice is the representation to which it is applied. The features used from here on are seasonal profiles, four seasons by twenty-four hours, computed after dividing each meter’s readings by its own daily maximum, so that each of the 96 features describes when a household consumes rather than how much.

Pitfall: inheriting a normalisation decision from a library default

Whether features are peak-normalised changes what a clustering finds. During the work behind this chapter, a library default for that option changed, nothing in the calling code set it explicitly, and every profile feature set silently began to describe how much each household consumed rather than when. Every validity index still returned a plausible number. A decision of this consequence belongs in the calling code, stated explicitly, where a change to a library cannot revoke it.

The features are projected to two dimensions with t-SNE at perplexity thirty, within the range van der Maaten and Hinton suggest for a population of this size [2]; a later section tests this choice. t-SNE places each household at a point \(\mathbf{z}_i\) of the plane so that the similarities of the households in the embedding match those in the feature space, by minimising the Kullback-Leibler divergence

\[ \mathrm{KL}(P \,\|\, Q) = \sum_{i \neq j} p_{ij} \log \frac{p_{ij}}{q_{ij}}, \qquad q_{ij} = \frac{(1 + \lVert \mathbf{z}_i - \mathbf{z}_j \rVert^2)^{-1}}{\sum_{k \neq l} (1 + \lVert \mathbf{z}_k - \mathbf{z}_l \rVert^2)^{-1}}, \tag{2.1}\]

where \(p_{ij}\) is the symmetrised similarity of households \(i\) and \(j\) under a Gaussian kernel whose bandwidth for each household is set by the perplexity. The heavy-tailed kernel of \(q_{ij}\) preserves neighbourhoods rather than global distances, which is why the embedding is clustered rather than the distances read from it. Figure 3.1 shows how eight internal indices rank the number of clusters on the resulting embedding. Each preferred a different value, which is the usual situation and the reason Chapter 1 cautioned against relying on any single index [3], [4], [5], [6]. A weighted Borda ranking combines eleven indices and down-weights those that correlate strongly with one another, so that a group of near-identical separation measures cannot outvote an index that measures something different. For indices \(m = 1, \dots, M\), each oriented so that higher is better, let \(\pi_m(K)\) be the rank of \(K\) among the candidate values \(\mathcal{K}\) under index \(m\), largest for the best, and \(\bar{\varrho}_m = \frac{1}{M} \sum_{m' \neq m} \lvert \varrho_{m m'} \rvert\) the mean absolute Spearman correlation of index \(m\) with the others across the candidates. Each candidate scores

\[ B(K) = \sum_{m=1}^{M} w_m \, \pi_m(K), \qquad w_m = \frac{\max(1 - \bar{\varrho}_m,\ 0.05)}{\sum_{m'} \max(1 - \bar{\varrho}_{m'},\ 0.05)}, \tag{2.2}\]

and the recommended \(K\) maximises \(B\). It recommended \(K = 11\), ahead of 12 and 8 (Table 8.2).

Figure 2.2: Internal validity indices of k-means on the t-SNE embedding of the London seasonal profiles for K from 2 to 12, one panel per index. The filled point is the K each index prefers, and the dotted line marks K = 11, recommended by the weighted Borda ranking over eleven indices.
Table 2.3: The five values of K with the highest weighted Borda score over the eleven indices.
K Weighted Borda score
11 8.799
12 8.090
8 7.202
7 6.976
10 6.837

At \(K = 11\) the clusters contain between 56 and 144 households. Figure 2.3 places them in the embedding and Figure 2.4 shows their daily profiles. The profiles provide the check that no index performs: a high Borda score indicates that the groups are separated in the embedding, whereas only the curves show whether the separation corresponds to behaviour that a network planner would recognise.

Figure 2.3: The 1,284 London households in the two-dimensional t-SNE embedding of their seasonal profiles, coloured and numbered by the k-means cluster at the recommended K.
Figure 2.4: Peak-normalised daily profiles of the London households by cluster. Thin gray lines are households; the blue line is the cluster mean.

What the Feature Step Changes

Whether the feature step improves on the raw curves cannot be decided by comparing silhouettes, because the two are computed in different spaces. The comparison that remains valid is between the partitions themselves, through the adjusted Rand index, which requires no shared space. The feature partition agreed with the Euclidean raw-curve partition at an ARI of 0.070 and with the warped one at 0.063, while the two raw-curve partitions agreed with each other at 0.226 (Table 2.4). Figure 2.5 colours the same embedding by each partition, so that every difference between its panels is a difference in labelling. The pipelines therefore produce different segmentations rather than two readings of one segmentation that differ at the margins, and the seasonal dimension is what separates them.

Table 2.4: Agreement between the partitions of the raw-curve and feature pipelines, which needs no shared feature space.
Comparison ARI NMI Purity
Euclidean curves against features 0.070 0.185 0.234
DTW curves against features 0.063 0.140 0.190
Euclidean curves against DTW curves 0.226 0.208 0.751
Figure 2.5: The same t-SNE embedding coloured by (a) the feature pipeline, (b) the Euclidean raw-curve pipeline and (c) the DTW raw-curve pipeline. Every difference between the panels is a difference in labelling.

Agreement Between Algorithms

The pipeline so far fixes k-means. Seven algorithms are therefore evaluated for \(K\) from 2 to 12 on the same embedding, each scored by the same Borda ranking (Table 2.5). Five of them, k-means, mini-batch k-means, the Gaussian mixture, Ward linkage and BIRCH, settled on \(K = 11\), whereas average linkage and spectral clustering settled on \(K = 2\). The division follows the geometry that each algorithm assumes. Methods that partition on distance to a centre find many roughly equal regions in an embedding of a continuous manifold, whereas average linkage merges on the smallest mean pairwise distance and spectral clustering cuts a nearest-neighbour graph, so both are drawn to the largest-scale division available.

Table 2.5: The number of clusters each of seven algorithms recommends on the same embedding, under the same weighted Borda ranking.
Algorithm K Weighted Borda score
Gaussian mixture 11 8.781
K-means 11 8.553
Mini-batch k-means 11 8.488
Average linkage 2 8.115
Spectral 2 8.078
Ward linkage 11 7.854
BIRCH 11 7.420

The agreement of five algorithms is weaker evidence than it appears. The five share an objective and are scored by the same ensemble on the same embedding, so their agreement is closer to a restatement than to five independent confirmations. The two dissenting algorithms are the more informative, and the remainder of the chapter returns to them.

Breaking the Tie with Resampling

Internal indices cannot settle a disagreement between partitions that all score well, because they all measure the same two quantities on the same embedding. The resampling measures of Chapter 1 supply evidence of a different kind. Tibshirani and Walther’s prediction strength (Equation 1.10) asks whether a clustering fitted on one half of the population predicts the co-membership of the other half [7], and Hennig’s cluster-wise stability (Equation 1.11) asks whether each cluster is recovered under bootstrap resampling [8]. The trust-gated composite score combines this evidence with separation. It multiplies a separation score by a trust factor,

\[ C(K) = S(K) \cdot T(K), \qquad S(K) = \frac{B_{\text{sep}}(K)}{\lvert \mathcal{K} \rvert}, \qquad T(K) = \min \Big\{ H(\mathbf{p}_K),\ \min_{k} \operatorname{stability}(C_k) \Big\}, \tag{2.3}\]

where \(B_{\text{sep}}\) is the weighted Borda score of Equation 2.2 over the silhouette, Caliński-Harabasz and Davies-Bouldin indices, \(H(\mathbf{p}_K)\) the balance of Equation 1.8 at \(K\), and the inner minimum runs over the clusters of the partition. A candidate therefore scores well only if it separates well, is balanced and reproduces, and no amount of separation compensates for a missing trust signal.

Table 2.6: Separation, balance and resampling evidence for each algorithm at its recommended K, ranked by the trust-gated composite score: prediction strength over ten random halvings and the smallest cluster-wise stability over 100 bootstrap resamples.
Algorithm K Density silhouette Balance Smallest cluster Resampling Composite score
Prediction strength Minimum stability
Spectral 2 0.839 0.259 56 0.959 0.699 0.481
Average linkage 2 0.839 0.259 56 0.891 0.646 0.444
K-means 11 0.417 0.988 56 0.418 0.704 0.303
Gaussian mixture 11 0.412 0.981 55 0.352 0.480 0.215
Mini-batch k-means 11 0.416 0.988 56 0.365 0.500 0.196
Ward linkage 11 0.379 0.969 33 0.355 0.289 0.120
BIRCH 11 0.380 0.977 56 0.346 0.310 0.104

Read without calibration, the result appeared decisive (Table 2.6). Every algorithm that chose \(K = 11\) reached a prediction strength between 0.35 and 0.42, well below the 0.8 that Tibshirani and Walther suggest, while spectral clustering and average linkage at \(K = 2\) reached 0.959 and 0.891. The trust-gated composite score accordingly ranked the two dissenting algorithms first, which suggested that the consensus of five does not reproduce and that the dissenters had found the only division that does.

Calibration Against a Structureless Reference

That reading assumes that prediction strength is comparable across \(K\), and there is a structural reason to doubt it. Prediction strength is a minimum of co-membership agreement over the test clusters, and both parts of that construction depend on \(K\). Under random assignment two points share a cluster with probability of roughly \(1/K\), so the chance level is near 0.5 at \(K = 2\) and near 0.09 at \(K = 11\), and a minimum over eleven clusters gives eleven opportunities for one weak cluster to set the score, against two.

The dependence can be measured directly by running the same checks on reference data with the size and extent of the embedding but no cluster structure, uniform over its bounding box, which is the reference Tibshirani, Walther and Hastie use for the gap statistic [9]. A third check joins the two: Ben-Hur, Elisseeff and Guyon’s subsample stability, the mean adjusted Rand index (Equation 1.12) between the partitions of pairs of random subsamples on the households they share, which is chance-corrected within the index itself [10]. For each check \(m\), with value \(m(K)\) on the data and \(m_0^{(r)}(K)\) on the \(r\)-th of \(R\) reference draws, the excess over the reference is

\[ \Delta_m(K) = m(K) - \frac{1}{R} \sum_{r=1}^{R} m_0^{(r)}(K), \qquad \bar{\Delta}(K) = \frac{1}{3} \sum_{m} \Delta_m(K), \tag{2.4}\]

the mean excess being taken over prediction strength, cluster-wise stability and subsample agreement. Figure 2.6 shows each check on the London embedding and on the reference.

Figure 2.6: Resampling checks of k-means on the London embedding (blue) and on uniform reference data of the same size and extent (gray), for K from 2 to 12. The segment between the points is the excess attributable to cluster structure.

On structureless data, prediction strength reached 0.949 at \(K = 2\) and 0.376 at \(K = 11\). The threshold of 0.8 is therefore not one standard applied to different candidates: noise clears it comfortably at small \(K\) and cannot clear it at large \(K\).

Remark: comparing resampling evidence across K

Subtracting the value a check attains on a structureless reference of the same shape converts an uninterpretable level into an interpretable difference. The excess has the same meaning at every \(K\): how much more reliably this population partitions than data without structure. It costs a few additional fits, and it is the form in which these checks can be compared between different values of \(K\).

Corrected in this way, the verdict reversed. At \(K = 2\) the excess was indistinguishable from zero on all three checks, between \(-0.009\) and \(0.001\): a two-way split of the London embedding reproduces as reliably as a two-way split of random points, and its reproducibility is therefore uninformative. Six values of \(K\) had a positive excess on all three checks at once, namely 5 and 7 to 11. Applied to each algorithm at its own recommended \(K\) (Table 2.7), the correction left k-means and the Gaussian mixture at \(K = 11\) as the only candidates with a positive excess on all three checks, while both \(K = 2\) candidates lost cluster-wise stability relative to the reference, by 0.147 for average linkage and 0.251 for spectral clustering.

Table 2.7: Raw prediction strength and the calibrated excess of each resampling check, per algorithm at its recommended K.
Algorithm K Raw strength Excess over the reference
Strength Stability Subsample Mean
Average linkage 2 0.891 0.148 −0.147 0.382 0.128
K-means 11 0.405 0.029 0.174 0.057 0.087
Gaussian mixture 11 0.355 0.046 0.053 0.109 0.069
Mini-batch k-means 11 0.340 −0.008 0.123 0.054 0.056
BIRCH 11 0.356 0.027 −0.021 0.020 0.008
Ward linkage 11 0.362 0.036 −0.113 0.059 −0.006
Spectral 2 0.960 0.036 −0.251 0.066 −0.049

The reference also qualifies what stability means. On structureless data, subsample agreement never fell below 0.738. Two independent subsamples of uniform noise, clustered separately, agree substantially about which points belong together, because k-means partitions noise as reproducibly as it partitions anything. Stability therefore measures reproducibility rather than structure, and the two coincide only when there is structure to reproduce.

The Overnight Cohort

The corrected evidence scored the \(K = 2\) split at zero excess, and its balance of 0.259 marks it as isolating a corner of the population rather than dividing it. The split nevertheless merits examination before that conclusion is accepted, because what the two dissenting algorithms separated can be checked directly.

Table 2.8: Connected components of the k-nearest-neighbour graph of the London embedding as the number of neighbours grows.
Neighbours Connected components Smallest component
10 2 56
15 2 56
20 2 56
30 2 56
50 2 56
100 1 1,284
200 1 1,284

The nearest-neighbour graph of the embedding has two connected components, and still has two at fifty neighbours; only at a hundred does the smaller one join the rest (Table 2.8). Spectral clustering was therefore not misconfigured. A graph with two disconnected components has two zero eigenvalues in its Laplacian, so a spectral method must separate them before it can find anything else [11]. The smaller component holds 56 households and coincides exactly with the \(K = 2\) partition of the dissenting algorithms, at an ARI of 1.000. Nor is the split an artefact of t-SNE: it is better separated in the original 96-dimensional feature space, with a silhouette of 0.414, than in the two-dimensional embedding, at 0.369, so the reduction did not create it.

Table 2.9: The isolated component of the London embedding against the rest of the population.
Group Households Share of demand 00:00 to 06:00 Mean daily demand (kWh) Peaking 00:00 to 01:00 Median peak hour
Overnight cohort 56 0.603 9.747 0.893 0
Rest of the population 1,228 0.150 4.774 0.008 19

The 56 households form a recognisable behavioural group (Table 2.9, Figure 2.7). Fifty of them reach their daily peak between midnight and one in the morning, against fewer than 1% of the rest of the population, whose median peak falls at seven in the evening. Sixty per cent of their daily consumption falls before six in the morning, against 15% for the rest, and their mean daily demand of 9.7 kWh is roughly double. A sharp overnight block followed by little consumption during the day is the signature of electric storage heating charged on an off-peak tariff.

Figure 2.7: Peak-normalised daily profiles of the overnight cohort and of the rest of the London population.

The trial’s own tariff column supports this reading independently (Table 2.10). Every one of the 56 households is on the standard tariff and none is in the dynamic time-of-use arm, against a time-of-use rate of 18.6% across the rest of the population, which would have predicted about ten of them; Fisher’s exact test gives \(p \approx 2 \times 10^{-5}\). Their overnight peak therefore cannot be a response to the trial’s price signal, because none of them received it. It is pre-existing behaviour, consistent with a legacy off-peak meter of the kind on which a dynamic tariff cannot be layered, which is the most likely reason that none of them was enrolled.

Table 2.10: Tariff arm of the Low Carbon London trial for the overnight cohort and the rest of the population.
Group Standard tariff Time-of-use tariff1
Overnight cohort 56 0
Rest of the population 1,000 228
1 Expected in the cohort at the population rate: 10.4; Fisher exact test p = 2.0e-05.

Pitfall: trusting a correction beyond the problem it corrected

Excess over a structureless reference scored this split at zero, and the arithmetic was correct. Any two-way split reproduces well, so a genuine two-way structure earns no more excess than an arbitrary cut, and a balance gate rejects a real minority archetype exactly as it would reject a fabricated one. Three defensible criteria all dismissed a group that a single tariff column confirms.

Detecting a genuine minority archetype therefore requires a different instrument from those that rank partitions: graph connectivity, a density method, or inspection of the curves. For a distribution operator, a load that already runs overnight and is thermally buffered is among the most valuable sources of flexibility on the network, and a centroid method asked for eleven balanced regions cannot return a group that holds 4% of the population.

Four Objective Families on Metered Data

The seven algorithms compared above do not represent seven independent definitions of a cluster. Chapter 1 distinguished four objective families, and two of them, density-connectivity and graph partitioning, were represented only by the dissenting algorithms. Both are now applied at the settled \(K\) on the same embedding, with DBSCAN’s radius \(\varepsilon\) read at the elbow of the sorted distances from each household to its \(k\)-th nearest neighbour, as Ester, Kriegel, Sander and Xu recommend [12].

Figure 2.8: Pairwise adjusted Rand index between partitions of the London embedding from one or more methods of each objective family, at the recommended K (DBSCAN determines its own number of clusters).

The density method stood apart (Figure 2.8). With its radius at the k-distance elbow, DBSCAN found two clusters and no noise points, and its partition agreed with every other family at an ARI between 0.019 and 0.021, the level expected of an independent partition. The partition, probabilistic, hierarchical and graph methods agreed with one another at 0.666 to 0.839, and the eigengap of the spectral method’s graph Laplacian independently estimated eleven clusters. Under a density definition of a cluster, a t-SNE embedding of a continuous behavioural manifold contains few separated dense regions, and DBCV, a density-based validity index, was negative at the recommended \(K\), at \(-0.275\) [13]. The separation indices of the earlier sections did not register this, because they measure separation rather than density.

Three Populations

Every result so far was obtained on London. Table 2.11 applies the same pipeline, unchanged, to GoiEner and AusNet, and Figure 2.9 shows the three embeddings; the number of clusters that each algorithm recommends on each population is listed in the companion notebook.

Table 2.11: The London pipeline applied unchanged to the three populations.
Dataset Households Recommended K K range Algorithms at the modal K DBCV at K ARI against k-means
DBSCAN Gaussian mixture Spectral
London 1,284 11 2 to 11 5 −0.275 0.019 0.833 0.757
GoiEner 1,000 10 7 to 12 3 −0.331 0.011 0.499 0.567
AusNet 342 11 2 to 12 2 −0.097 0.000 0.614 0.705
Figure 2.9: The t-SNE embeddings of the three populations, coloured and numbered by the k-means cluster at each population’s recommended K.

The results fall into three groups. The independence of DBSCAN generalised fully: its agreement with k-means was 0.019 on London, 0.011 on GoiEner and 0.000 on AusNet, and the density-based DBCV index was negative at the recommended \(K\) on all three populations [13]. The recommended number of clusters did not generalise, and neither did the confidence behind it. London and AusNet settled on eleven clusters and GoiEner on ten, an apparent agreement that the per-algorithm results qualify: five of the seven algorithms agreed on the modal value for London, three for GoiEner and only two for AusNet, whose 342 households provide the least evidence from which to form a consensus. One London finding did not survive at all. The division between algorithms that prefer a fine partition and a pair that prefers the coarsest was particular to London: average linkage chose 2, 7 and 2 clusters across the three populations, and spectral clustering 2, 11 and 11. The coarse split was a response to London’s disconnected cohort rather than a standing preference of either algorithm. Chapter 3 returns to the number of clusters on AusNet with calibrated evidence.

Auditing the Fixed Choices

Every result above came from one pipeline whose feature set, absence of a scaling step and reduction method were fixed at the outset and not revisited. They can be audited now because the first half of the chapter produced the instruments needed to do so. Silhouette cannot compare these conditions, since each changes the space in which the clustering takes place. Three measures can: the adjusted Rand index (Equation 1.12) against the settled partition of each population, cluster balance (Equation 1.8) and size, and the recall (Equation 1.14) of the overnight cohort on London, a structure established independently of any of the pipelines being compared. Each audit varies one choice and holds the rest at the settled pipeline.

Scaling

The pipeline applies no scaling to the features. Since each feature is an hourly mean of a peak-normalised series, the columns already share a bounded scale, but their spreads differ, and a distance treats low-variance hours as less important.

Figure 2.10: Effect of the scaling step: (a) agreement of each scaled pipeline with the unscaled partition on the three populations, and (b) the share of the overnight cohort recovered as one cluster on London (Jaccard overlap).

The scaling step left every separation index nearly unchanged and rewrote a large part of the partition (Figure 2.10). Across the six conditions, the embedded silhouette varied by at most 0.023 on London, 0.013 on GoiEner and 0.011 on AusNet, and balance stayed above 0.96 throughout, so a choice made from a table of validity indices would treat the six as equivalent. Their partitions are not equivalent: standardising the features lowered the agreement with the unscaled partition to 0.549 on London, 0.624 on GoiEner and 0.424 on AusNet, and the quantile transform reached 0.374 on AusNet. The indices do not measure what the scaling step changes.

The unscaled pipeline scores 1.000 against itself by construction, so the agreement column cannot rank the conditions. Cohort recovery can, because it measures agreement with a structure established independently of all six. No scaling, min-max scaling and robust scaling each recovered the overnight cohort exactly; standardising recovered 95% of it, the power transform 91% and the quantile transform 83%. The ordering has a mechanism: a quantile transform maps each feature onto a uniform distribution and so compresses the extreme overnight values that make the cohort distinctive. Since the peak-normalised features already share a bounded scale, no scaler improves on omitting the step, and the pipeline keeps the unscaled features, now as a choice supported by evidence. The conclusion is specific to this representation; a feature set that mixes kilowatt-hours with ratios or hours would reopen the question.

Feature Sets

The seasonal profile is one of ten feature sets. Three are whole-profile representations that differ in what they average over (the seasonal profile, a weekly profile by day of week and hour, and both together), and seven compress a household to a few summary statistics of its magnitude, volatility, trend, shape, autocorrelation, calendar pattern or distribution. These are not competing estimates of one quantity: a representation determines which structure a clustering can find.

Table 2.12: The ten feature sets on London, ordered by the silhouette of their t-SNE embedding: dimensionality, agreement with the seasonal-profile partition and recovery of the overnight cohort.
Feature set Columns Silhouette ARI against seasonal Cohort recovery
Shape 3 0.667 0.071 0.667
Magnitude 5 0.521 0.019 0.171
Volatility 2 0.467 0.007 0.108
Autocorr 5 0.412 0.053 0.094
Seasonal 96 0.412 1.000 1.000
Calendar 4 0.411 0.017 0.252
Distribution 6 0.408 0.023 0.247
Profile 264 0.408 0.625 1.000
Trend 5 0.404 0.003 0.068
Weekly 168 0.402 0.569 1.000

Ranked by silhouette, the summary statistics led (Table 2.12): shape reached 0.667 and magnitude 0.521, against 0.402 to 0.412 for the three whole-profile representations. Much of that ranking reflects dimensionality rather than quality. shape has three columns and volatility two, against 96 to 264 for the profiles, and distances concentrate as dimension grows. The adjusted Rand index shows that the summary statistics are not weaker versions of the profile partition but are nearly unrelated to it, at no more than 0.071 on London and similarly low values on the other two populations (Figure 2.11), whereas the weekly and combined profiles agreed with the seasonal partition at 0.569 and 0.625. magnitude sorts households by how much they consume and volatility by how erratic they are, and neither has any reason to reproduce a grouping by when a household uses electricity.

Figure 2.11: Agreement of each feature set’s partition with the seasonal-profile partition on the three populations.

Cohort recovery inverted the silhouette ranking. Only the three whole-profile representations recovered the overnight cohort intact; shape recovered two-thirds of it, and the remaining six recovered between 7% and 25%. The two representations with the highest silhouette are among those that dissolve a group for which independent evidence exists. A feature set is therefore not better or worse in the abstract: it determines which question a clustering answers, and where a known structure exists, testing whether a representation preserves it is more informative than any index.

Dimensionality Reduction

The reduction step was fixed to t-SNE, and the overnight cohort was found because t-SNE’s neighbourhood graph left its households in a component of their own. Six conditions are compared: no reduction, principal component analysis, kernel PCA [14], t-SNE, UMAP [15] and PaCMAP [16]. Because each produces a different space, the silhouette is reported both within each embedding and in the shared feature space.

Table 2.13: The six reduction conditions: silhouette measured in each embedding and in the shared feature space, balance and cohort recovery on London, and the smallest cluster on each population.
Reducer Silhouette (London) Smallest cluster Balance Cohort recovery
In the embedding In the feature space London GoiEner AusNet
None 0.097 0.097 11 23 13 0.947 0.911
PCA 0.348 0.090 5 23 2 0.899 0.875
Kernel PCA 0.348 0.074 23 24 2 0.941 0.875
t-SNE 0.412 0.054 56 36 24 0.988 1.000
UMAP 0.427 0.062 56 24 21 0.991 1.000
PaCMAP 0.417 0.051 56 36 19 0.991 1.000

The two silhouettes disagreed (Table 2.13). Measured in each embedding, reduction helped: no reduction scored 0.097 on London against 0.348 to 0.427 for the reduced embeddings. Measured in the shared feature space, the order reversed, with no reduction highest at 0.097 and the three manifold methods lowest at 0.051 to 0.062. The embedded silhouette rewards having fewer dimensions in which to separate, the same effect that favoured the three-column shape, so it cannot settle the question. Cluster sizes can. A linear projection asked for eleven groups returned clusters of five households on London and two on AusNet, while the three manifold methods returned no cluster smaller than 19 on any population. No separation index objects to a cluster of two households, because two points far from everything else are perfectly compact and perfectly separated.

The connectivity check of the ten-nearest-neighbour graph, tabulated in the companion notebook, explains the difference. Only t-SNE, UMAP and PaCMAP left the overnight cohort as a disconnected component of the ten-nearest-neighbour graph, and each isolated exactly the same 56 households, at an ARI of 1.000 against the cohort. Without reduction, and under PCA and kernel PCA, the graph has a single component. The cohort is not invisible without a manifold method, since k-means recovered 91% of it on the unreduced features and 88% under PCA, but only the manifold methods make it topologically distinct. Figure 2.12 shows the six embeddings coloured by the settled partition. Among the manifold methods the choice is nearly free, since they agreed on the cohort exactly and their balance on London differed by 0.003; the decision that matters is between manifold and linear reduction.

Figure 2.12: The London seasonal profiles under each reduction condition (the first two dimensions where there are more), coloured and numbered by the settled partition.

Retained Components

Each reduction above kept two components, the number a scatter plot can show, which is a presentation constraint rather than an argument. Chapter 1 introduced two rules for choosing the number: the 90% variance rule and Zhu and Ghodsi’s profile likelihood [17].

The two rules disagreed widely. The variance rule asked for 11, 8 and 14 components on the three populations, and profile likelihood for one on all three. The disagreement had little practical consequence (Figure 2.13). Under UMAP, increasing the retained components from two to twenty changed the balance on London by at most 0.002, cohort recovery stayed at 1.000 except at ten components, where it was 0.982, and the smallest cluster held 55 or 56 households throughout. Under PCA the result was worse rather than different: the smallest cluster on AusNet held two households at two components and again at twenty, and cohort recovery never exceeded 91%. The number of retained components is a minor choice; the family of reduction is the major one. Profile likelihood’s answer of one component reflects the shape of these spectra, a single dominant direction followed by a smoothly decaying tail; the first two components already hold 63.8% of the variance on London, 73.8% on GoiEner and 53.8% on AusNet.

Figure 2.13: Balance of the k-means partition as the number of retained components grows, under (a) PCA and (b) UMAP.

Interaction Between Choices

Each audit varied one choice with the others fixed, so the feature-set ranking assumed t-SNE and the reduction ranking assumed the seasonal profile. Crossing four feature sets with three reducers tests whether the two choices interact. They interacted little (Figure 2.14). shape ranked first and volatility second under every reducer on every population; on London the order was identical under all three reducers, and on GoiEner and AusNet the only change was the seasonal and weekly profiles exchanging third and fourth place under one reducer.

Figure 2.14: Silhouette in the shared feature space for four feature sets crossed with three reducers, on each population.

The stability of this ranking is less reassuring than it appears. The ranking is measured by silhouette in the shared feature space, which the feature-set audit showed to be dominated by the number of columns, and the number of columns does not change with the reducer. A confounded measure produces a stable ranking whether the choices interact or not. The more telling observation is that the ranking is both stable and inverted with respect to cohort recovery: the profile sets ranked third and fourth here are the only representations that preserve the cohort. Where settling one choice at a time failed, it failed because the objective optimised at each step was inappropriate, not because the choices interact.

Conclusion

This chapter applied the instruments of Chapter 1 to three metered populations and answered three of its questions. On the second question, a load curve does not need a specialised distance: dynamic time warping performed worse than Euclidean distance on London because the timing of demand is the information that distinguishes households. The representation that preserved the population’s one verifiable structure was a peak-normalised seasonal profile, unscaled, passed through a neighbourhood-preserving reduction at two components; within that choice the specific reducer and the number of components mattered little, while the choice between manifold and linear reduction mattered greatly.

On the third question, no single index, and no set of indices sharing the same limitation, validated a grouping on its own. Uncalibrated resampling favoured the coarsest split for structural reasons, and calibration against a structureless reference reversed that verdict, yet even calibrated evidence scored a genuine minority archetype at zero. The overnight cohort of 56 London households is the most instructive instance. Silhouette ranked the representations that dissolve it above those that preserve it, the calibrated excess scored its split at zero, and balance rejected it, while graph connectivity and the trial’s tariff column established that it is real. The first question, which criterion defines a group, therefore received a qualified answer: centroid methods at \(K = 11\) provide a reproducible segmentation of the London population, and a connectivity or density method is needed alongside them to find the minority groups that such a segmentation absorbs.

A joint search over the whole pipeline did not improve on these choices, because it optimised the same objectives whose limitations the chapter had exposed. The conclusion is not that the measures should be abandoned but that each number should be read together with what it describes: the cluster sizes, the archetype curves and, where available, an external attribute. Table 2.16 condenses this practice into a protocol. Chapter 3 applies the settled pipeline, together with this protocol, to the AusNet population, and Chapters 4 and 5 to the network that its households supply.

Table 2.16: A protocol for validating a clustering pipeline on smart-meter data.
Step Check Failure it guards against
1 State the representation, including its normalisation, in the calling code A silent change in what is clustered
2 Rank \(K\) with several internal indices, combined by a decorrelated ranking Reliance on a single index
3 Inspect the cluster sizes and their balance Compact, distant minorities rewarded by separation indices
4 Calibrate each resampling check against a structureless reference Resampling evidence that favours small \(K\) by construction
5 Check the connectivity of the nearest-neighbour graph and apply a density method Genuine minority groups absorbed by centroid methods
6 Compare pipelines by the agreement of their partitions, not by indices computed in different spaces Comparisons that change the space they measure
7 Validate against an external attribute or an independently established structure Partitions that are reproducible but not meaningful
8 Repeat the analysis on a second population Findings particular to one population

Key findings

On metered data, dynamic time warping performed worse than Euclidean distance, because the timing of demand is the signal. Internal indices disagreed with one another, and uncalibrated resampling favoured a two-way split that reproduced no better than noise; calibrated against a structureless reference, the evidence supported eleven clusters on London. A cohort of 56 households whose overnight peak is the signature of storage heating, confirmed by the trial’s tariff column, was dismissed by every ranking criterion and found only by graph connectivity. Manifold reduction preserved this cohort where linear reduction and summary features did not, and a joint search over the whole pipeline found nothing better than the settled one.

Exercises

  1. Compute the connectivity table of the tbl-connectivity cell on the GoiEner and AusNet embeddings. Does either population contain a disconnected group, and if so, what does its mean daily profile show?
  2. Repeat the scaler audit of the scalers cell with the weekly profile instead of the seasonal profile. Does the ordering of the scalers by cohort recovery persist?
  3. Restrict the joint search of the search cell to \(K \ge 4\). What does the Pareto front then consist of, and does any trial recover the overnight cohort?

References