| k | separation_score | trust_factor | composite_score |
|---|---|---|---|
| 2 | 0.768 | 0.937 | 0.72 |
| 3 | 0.808 | 0.835 | 0.675 |
| 4 | 0.896 | 0.747 | 0.67 |
| 6 | 0.619 | 0.833 | 0.515 |
| 5 | 0.48 | 0.833 | 0.4 |
| 7 | 0.286 | 0.859 | 0.245 |
| 8 | 0.143 | 0.687 | 0.098 |
Chapter 4: Alternative Representations Across Two Populations
A tariff or a DER-risk tool built on Chapter 3’s archetypes is only as trustworthy as the assumption that those archetypes are a property of customer behaviour, not a coincidence of one utility’s metering setup. Chapter 1 closed with a promise, stated plainly rather than tucked into a footnote: later chapters would test whether the settled recipe, together with four different data representations, a two-stage summary of behaviour, tensor decomposition, a learned foundation-model embedding, and a diffusion-map perspective, produces consistent definitions of real structure across independent populations, including households in London and Spain. That is not a hypothetical. It is what this chapter checks, against GoiEner and London households, using the same trust-gated composite score Chapter 2 built and Chapter 3 already relies on, not a fresh, more forgiving standard invented for the occasion.
The standard way a new representation earns its place in this literature is familiar: propose it, show it beats a baseline on one dataset, publish. That approach quietly assumes a representation’s own apparent success on one population is evidence of something real and transferable, not a coincidence of that population’s quirks. Chapter 1’s review already found this gap directly: not one of Tucker decomposition, diffusion maps, a learned zero-shot embedding, or CROCS’s own two-stage design has been checked against a second, independently collected population, in this literature or in this book, until now. Checked honestly, four questions follow, and together they are this chapter’s structure. Does each representation find genuinely trustworthy structure at all, on either population, once judged by the same resampling standard as everything else in this part? When it does, does it agree with what the settled recipe already found, or does it reveal something the settled recipe cannot see? Does a finding on one population survive contact with a second, or is it an idiosyncrasy that does not travel? And after checking all four, honestly, does the settled shape+PaCMAP+GMM recipe remain the strongest known option, or does a alternative earn a place beside it?
Borrow, do not invent
Every mechanism this chapter reaches for is a published method, already named and formally introduced in Chapter 1, restated here in miniature rather than re-derived from scratch. Tucker’s three-way factorisation approximates a genuine households-by-days-by-time-of-day tensor \(\mathcal{X} \in \mathbb{R}^{I \times J \times K}\) as a smaller core tensor and one factor matrix per mode, \(\mathcal{X} \approx \mathcal{G} \times_1 A \times_2 B \times_3 C\) [1]; the household-mode factor matrix \(A\) is the low-dimensional embedding clustered on below. Coifman and Lafon’s diffusion maps build a Markov transition matrix \(P = D^{-1}W\) from a kernel similarity and embed each point by its top eigenvectors scaled by eigenvalue, \(\Psi_t(i) = (\lambda_1^t \psi_1(i), \lambda_2^t \psi_2(i), \dots)\), reading the eigenvalue spectrum itself for a gap before trusting any resulting partition [2]. Yerbury et al.’s CROCS summarises each household’s days into a small Representative Load Set, then compares households by a Weighted Sum of Minimum Distances between their own RLSs, crediting a match by how often each representative shape actually occurs, not just whether the single closest pair happens to align [3]. Chronos-2, already validated in this book’s foundation-model forecasting chapter, runs each household’s sequence through a pretrained encoder with no fitting step at all and returns one embedding vector per input patch, mean-pooled here into a single per-household vector [4].
The reference point
Before checking anything new, this chapter re-anchors on what is already known. Chapter 2 settled shape+PaCMAP+GMM on London; Chapter 3 already extended the same no-reduction convention to GoiEner directly, settling on its own \(K=4\) there. This chapter’s baseline rerun of that same peak-normalised, no-reduction pipeline on 2,000 GoiEner households, run independently for this comparison, reconfirms the broader shape of that finding rather than one fixed number: \(k{=}2\), \(k{=}3\), and \(k{=}4\) all clear the trust gate, prediction strength 0.839, 0.728, and 0.719 respectively, minimum cluster stability 0.937, 0.881, and 0.796, a genuine landscape rather than one winner, the same honest shape of result Chapter 2 already found for London’s richer resolutions.
That landscape, not a single number, is the baseline every representation below gets checked against.
Tucker tensor decomposition
Raw Tucker decomposition, with no correction for a household’s own scale, fails on both populations, and fails the same way. On GoiEner, its own household-mode factors turn out to correlate directly with each household’s raw peak consumption, so Euclidean k-means ends up isolating the household with the most extreme peak at each \(k\), not finding archetypes; the trust gate confirms it, prediction strength 0.998 (trivial for a split this lopsided) against a cluster-wise stability of only 0.47, deep in Hennig’s dissolved band. On London the same mechanism recurs, a suspiciously high silhouette at \(k{=}2\) (0.813) collapsing sharply by \(k{=}4\), and the trust gate again refuses it: prediction strength 0.96, stability just 0.177. The two validity curves below make the warning sign visible before the trust gate ever runs: a silhouette this high, this early, is itself a clue worth reading, not just a number worth maximising.
Peak-normalising each household’s season before decomposing, this book’s magnitude-invariance convention, is where the two populations genuinely part ways. On GoiEner it produces a trustworthy \(k{=}2\) split, 1,960 households against 40, prediction strength 0.976, minimum cluster stability 0.858, as trustworthy on paper as anything found anywhere in this book. Yet the trust-gated composite score ranks it last of the four \(k\) values checked, purely because its own balance, 0.141, is the lowest of the four, a concrete limit in a scoring rule that takes the minimum of balance and stability: it has no way to tell a genuinely rare, resampling-stable archetype from a fake one, and vetoes both alike.
Plotted in two dimensions, the 40-household minority is not a statistical abstraction; it is a visibly separate cluster of points in the household factor space below.
On London, peak-normalising does not rescue the representation at all. Only \(k{=}2\) clears Tibshirani and Walther’s 0.8 prediction-strength floor (0.812), and even there, cluster-wise stability sits at just 0.384, well under Hennig’s 0.5 floor; every other \(k\) checked, 3 through 5, fails both thresholds outright, and stability never exceeds 0.411 anywhere in the range. Part of the reason is structural, not procedural: London’s half-hourly households compress far less cleanly than GoiEner’s hourly ones, reaching only 51% explained variance at the largest household rank tried, against GoiEner’s 99% at a fifth of that rank. A finer sampling rate carries finer behavioural variation a low-rank household factor struggles to summarise.
The same 2D projection on London makes the failure visible rather than just numerical: no clean separation between the two colours below, only a diffuse cloud a k-means boundary cuts through arbitrarily.
A related, cautionary finding belongs here too, even though it sits outside Chapter 1’s four named representations. A separate check on GoiEner, clustering directly on 850 raw multi-resolution feature columns with no dimensionality reduction at all, produced a silhouette of 0.990, higher than anything else in this whole comparison, and a split of 1,999 households against 1. Cluster-wise stability read 0.829, inside Hennig’s stable band, not below it: the single extreme household sits so far from everyone else in that high-dimensional space that a bootstrap resample reliably rediscovers it as its own cluster regardless of composition, extremeness itself fooling a check built to catch instability. Only the balance score, 0.006, close to the floor, caught what stability could not.
Tucker’s lesson, then, is not “Tucker fails” or “Tucker succeeds.” Its by-construction dimensionality reduction avoids the multi-resolution notebook’s curse-of-dimensionality trap, but a magnitude correction it does not apply automatically is doing most of the work, and even once applied, whether the result is trustworthy depends on the population, decisively yes on GoiEner, decisively no on London, at the ranks checked here. Chapter 2 built shape by hand, peak-normalising each household’s day before averaging it; Tucker’s household-mode factor arrives at something similar by decomposing the raw tensor directly, with no designer choosing which axis of variation to keep. Where the two converge, on GoiEner, that is independent confirmation of the same underlying structure by a completely different route.
Chronos-2 zero-shot embeddings
Every other representation in this chapter needs some hand-designed correction, a peak normalisation, a rank choice, a season split, before it clusters cleanly. Chronos-2 needs none of it. Fed each household’s own unaveraged summer sequence directly, with no fitting step on this data at all, its own embedding is naturally low-rank on both populations: PCA needs only 28 of 768 raw dimensions for 90% explained variance on GoiEner, 34 on London. Chapter 2 reaches PaCMAP only after first hand-building a shape vector; Chronos-2 skips the hand-building step entirely and still lands somewhere a validity curve can read cleanly, the two curves below.
On GoiEner, the resulting \(k{=}2\) splits 591 households against 1,409, and clears the trust gate decisively: prediction strength 0.849, minimum cluster stability 0.928. On London, \(k{=}2\) splits a near-even 654 against 630, balance 1.000, and clears the same gate more decisively still, prediction strength 0.891, stability 0.955. Every other \(k\) checked on either population, 3 through 5 or 9, fails the resampling floor outright.
A 2D projection of each embedding shows the two populations’ own different geometry directly: GoiEner’s split falls along one axis with a minority visibly apart from the rest, London’s falls into two similarly sized clouds, the near-even split the numbers above already report.
This is the one representation in the whole comparison that clears the same bar independently on both populations, the headline finding of this chapter. It is not, however, the same finding twice. GoiEner’s own 591-household group is a minority; London’s 654-630 split is close to even, the most balanced result any representation in this comparison finds on London, including the settled recipe’s 982-302 split. Chronos-2’s zero-shot embedding is not rediscovering the settled recipe’s archetypes through a different lens. It is finding a genuinely different axis of behaviour, coarser on GoiEner, markedly more even on London, that a hand-engineered peak-normalised shape does not surface.
Clearing the same bar on a second population is not the same claim as clearing it on a second season. Chapter 3’s own cross-quarter check, the same one applied to the settled recipe, applies here too: re-embed a different calendar quarter with the same pretrained encoder, re-cluster independently, and compare against the summer split by Adjusted Rand Index (ARI).
The two populations diverge sharply here too, in a way the cross-population check alone could not have shown. On GoiEner, summer and winter agree moderately, ARI 0.458, roughly the same order as CROCS’s own moderate cross-resample agreement earlier in this chapter, real but far from perfect persistence. On London, summer and winter agree at essentially chance level, ARI 0.001. The decisively balanced, decisively trust-gated 654-630 split London’s summer data produces is not a stable year-round axis of behaviour; it is what this embedding finds in one season, and a different season finds something close to uncorrelated with it. Chronos-2’s cross-population win and its within-population seasonal stability are two different claims, and only the first one holds on London.
Diffusion maps
Diffusion maps ask a sharper question than a silhouette curve can: read the eigenvalue spectrum of a random-walk kernel directly, and check whether a genuine gap separates a small number of clusters from a continuum, before ever fitting \(k\). Run here on top of both the Tucker and Chronos-2 embeddings above, on both populations, the answer is not a clean yes or no, and reporting that honestly matters more than forcing one. Chapter 2 also reduces dimensionality before clustering, PaCMAP on a hand-built shape vector, but never reads an eigenvalue spectrum first; a diffusion map is what that same reduction step looks like when the geometry itself, not a validity curve after the fact, is asked whether a gap exists at all.
On GoiEner, Tucker diffusion’s validity curve is suspiciously high and nearly flat (0.978-0.982) across most \(k\), the same magnitude-driven warning sign raw Tucker itself showed, isolating a single household at the curve’s favoured \(k\). But its own \(k{=}2\), a different operating point, actually clears the trust gate: prediction strength 0.833, stability 0.805. GoiEner’s Chronos-2 diffusion runs the opposite way, degrading an already-trustworthy raw embedding into one that fails cluster-wise stability at every \(k\) checked (a best of 0.467 at \(k{=}3\), still inside the dissolved band).
Plotted directly, the two diffusion coordinate spaces below look nothing alike: Tucker diffusion’s single isolated household sits far from a dense main body, while Chronos-2 diffusion’s two groups sit closer together, consistent with a weaker signal.
London sharpens both halves of that same story. Tucker diffusion never reaches a stable point at any \(k\) from 2 to 4, stability topping out at 0.61, well short of GoiEner’s rescued 0.805. Chronos-2 diffusion degrades London’s decisively balanced raw embedding even more sharply than it degraded GoiEner’s, cluster-wise stability collapsing from 0.955 in the raw embedding to 0.211 once diffused.
London’s coordinate plots make the degradation visually obvious: Tucker diffusion isolates a lone point far from the main body, and Chronos-2 diffusion’s once-clean two-group split now overlaps enough that a resampled fit struggles to reproduce it.
The consistent, cross-population lesson is narrower than “diffusion maps do not work here.” A spectral-gap check is only as good as the geometry it runs on, and applying one on top of an embedding that is already good is not free: on both populations checked, Chronos-2’s already-trust worthy structure got worse, never better, under a diffusion transform. Whether a diffusion map helps or actively destroys real structure has to be checked case by case, not assumed either way, a genuinely different, more cautionary lesson than the one this chapter’s other three representations teach.
CROCS-inspired two-stage clustering
CROCS keeps each household’s distinct daily behaviours separate, comparing households only at the level of how much their own typical days resemble each other’s, rather than collapsing a whole season into one blended average first. Chapter 2’s shape feature makes exactly that collapse, one peak-normalised mean day per household; CROCS is what the same population looks like when that averaging step is deferred, kept only at the point of comparison rather than at the point of feature-building. On both populations, most households show genuine within-season diversity: on GoiEner, only 25 of 2,000 collapse to a single representative day-type; on London, only 1 of 1,284 does, with a mean of 2.53 day-types per household. A handful of GoiEner households below make that diversity concrete: a household whose own days collapse to one representative shape next to one that splits into four, each shape weighted by how often it actually occurs across the season.
Where the two populations diverge is in how confidently that structure replicates. On GoiEner, the chosen \(k{=}2\) isolates a tight minority of 9 households against 1,991, but repeated 80% subsampling finds only moderate agreement, mean ARI 0.527 across 105 independent pairs, with the weakest pair essentially at chance, \(-0.022\). On London, the chosen \(k{=}2\) isolates an even more extreme minority, 3 households against 1,281, and the same resampling check finds it reproduces almost perfectly, mean ARI 1.000 across every one of the same 105 pairs.
The qualitative finding, a small, tight minority genuinely separating from the bulk under a per-household representative-day distance, holds on both populations. How much confidence the resampling check itself warrants in that finding does not: rock-solid on London, but markedly more provisional on GoiEner. A representation that finds the same kind of structure everywhere is not automatically finding it with the same reliability everywhere, and this chapter’s trust gate is what makes that distinction visible instead of papering over it with one silhouette number.
Where this leaves the settled recipe
The prediction-strength and cluster-stability numbers behind every representation checked in this chapter, side by side, make the same point the sections above build one population at a time: Chronos-2 is the only bar that clears both thresholds, Tibshirani and Walther’s 0.8 prediction-strength floor and Hennig’s 0.5 stability floor, on both populations at once.
CROCS is not on that chart: its own WSMD distance has no vector centroid for prediction strength or cluster stability to fit on, so its trust check uses repeated-subsample ARI agreement instead, a different metric on a different scale, reported in the table below rather than forced onto the same bars.
| Representation | GoiEner | London |
|---|---|---|
| Tucker (peak-normalised) | Passes (\(k{=}2\), PS 0.976, stability 0.858), but ranked last on composite balance | Fails at every \(k\) (best PS 0.812 at \(k{=}2\), but stability only 0.384 there; best stability 0.411 at \(k{=}3\), still short of the 0.5 floor) |
| Chronos-2 (zero-shot) | Decisive pass (\(k{=}2\), PS 0.849, stability 0.928) | Decisive pass (\(k{=}2\), PS 0.891, stability 0.955) |
| Diffusion maps | Mixed: Tucker-diffusion rescues one point (0.805); Chronos-2-diffusion fails everywhere | Fails: Tucker-diffusion never stabilises (0.61); Chronos-2-diffusion collapses to 0.211 |
| CROCS (RLS + WSMD) | minority (9/2,000), moderate resampling agreement (mean ARI 0.527) | minority (3/1,284), near-perfect resampling agreement (mean ARI 1.000) |
| Chronos-2, summer vs. winter | Moderate cross-quarter agreement (ARI 0.458) | Chance-level cross-quarter agreement (ARI 0.001) |
Four questions opened this chapter. Checked honestly against both populations, the answers are not uniform, and that unevenness is itself the finding worth keeping. Every representation finds some real structure somewhere; only Chronos-2 finds structure that clears the same resampling bar independently on both populations checked. Tucker’s own success is population-specific. Diffusion maps, on both populations, more often destroy a good embedding’s trustworthiness than improve it. CROCS finds the same kind of structure on both populations but not with the same reliability.
Does the settled shape+PaCMAP+GMM recipe remain the strongest known option? Yes, and the cross-quarter check above is a second reason why, not just the cross-population one. It carries the longest, broadest validation in this book, three independent utilities, not two, a year-long window built into every one of those checks from the start, and the most consistent resampling record across all of them. Chronos-2’s zero-shot embedding earns a disclosed, narrower place alongside it: the one alternative representation in this whole comparison whose trustworthy structure survives contact with a second, independently collected population, but only checked one season at a time. On GoiEner that structure also carries over moderately well from summer to winter; on London it does not carry over at all, a chance-level ARI of 0.001 that the cross-population check alone would never have surfaced. A hand-engineered shape feature earns its year-long window by construction; a pretrained encoder’s own embedding does not automatically inherit that same stability just because it clears the trust gate once, on one season. Whoever reaches for Chronos-2 in place of the settled recipe should check cross-quarter stability on their own population before trusting a single-season split, the same discipline this book applies to shape-based clustering throughout. Tucker, diffusion maps, and CROCS remain useful findings on their own narrower terms, not yet alternatives to the recipe itself.
Why bother
It would have been faster to skip this chapter entirely. The settled recipe already works, Chapter 2 built it, Chapter 3 already applied it to a second utility, and nothing forced a check of whether some other representation might work better or reveal something the shape feature cannot see. Skipping that check is exactly how a field ends up with four competing representations, each published against its own author’s favourite dataset, none checked against anyone else’s.
For a practitioner deciding whether to adopt Tucker, Chronos-2, diffusion maps, or CROCS, this chapter’s answer is not “use whichever paper sounds most sophisticated.” It is: check it against a second population, and a second season, before trusting it anywhere near production. Tucker’s success on GoiEner would have looked like unqualified proof of concept if London had never been checked; instead London shows the same mechanism failing outright, a result that would have gone unreported had this chapter stopped after one dataset, the same trap Chapter 1’s literature review already found waiting in most of this field’s published work. Chronos-2’s own cross-quarter check adds a second, sharper version of the same lesson: it cleared the cross-population bar on both populations checked, and would still have looked fully validated had this chapter stopped there, yet its decisive London split collapses to chance-level agreement one season later. Neither gap would have shown up from a single check alone.
For a researcher proposing a new representation, publish it, but also publish what happens when someone else runs it on their own population, not just the population it was designed against. Every representation checked here finds real structure of some kind; only one finds structure a second, independently collected population agrees is real too. That gap, not the first result, is the one worth reporting honestly.
For the next chapter, this one settles what does not need re-settling. Chapter 5 moves up one level of aggregation, from customers to whole feeders, and it does so with the settled shape+PaCMAP+GMM recipe, not a fresh representation search of its own. This chapter is why that choice does not need re-litigating: the alternatives have been checked, honestly, against two populations, and none of them unseats the recipe Chapter 5 is about to reuse.
Where this leads
Chapter 1 promised four representations would be checked against the settled recipe, honestly, on households in London and Spain, not just AusNet. That promise is kept. None of the four alternatives replaces the recipe outright; one, Chronos-2’s zero-shot embedding, earns a disclosed, season-qualified place beside it, structure a hand-engineered shape feature does not surface, still worth knowing about even though it does not win outright and does not hold up across every quarter checked. Chapter 5 takes the settled recipe one level up, from individual customers to whole feeders, on a population of 33 Low Voltage (LV) feeders this book has not yet had the chance to check.