Cluster#

milia.cluster groups series that behave alike, decides how many groups the data supports, and reports how far to trust them.

from milia import available, make
from milia.data import profiles, synthetic

population = synthetic.generate(meters=60, days=90, steps=48)
shapes = profiles.shape(population.pivot)
available("clusterer")
['agglomerative',
 'birch',
 'copkmeans',
 'dbscan',
 'gmm',
 'hdbscan',
 'kmeans',
 'kshape',
 'minibatch',
 'spectral',
 'tskmeans']
Four families of clustering objective: partition, hierarchical, density-connectivity, and graph and spectral.

Clusterers form groups in four ways: around centroids, by cutting a tree, by density, and by cutting a graph.#

Choose the number of groups#

validation.sweep scores every candidate number of groups by internal validity, and with resampling=True by prediction strength, the smallest-cluster bootstrap and subsample agreement. selection.recommend applies a strategy to the table. The trust strategy only accepts a number of groups that passes the resampling checks.

Prediction strength and cluster-wise stability.

The two resampling checks. Prediction strength asks whether the centroids of one half predict the other half. Stability asks whether each cluster reappears when the data are resampled.#

What the silhouette, the Davies-Bouldin index and the Calinski-Harabasz index compare.

The internal indices of the sweep compare how compact each group is with how far apart the groups are.#

from milia.cluster import selection, validation

scores = validation.sweep(shapes, range(2, 8), resampling=True)
k = selection.recommend(scores, strategy="trust")
scores[["k", "silhouette", "davies", "strength", "stability"]].round(2)
k silhouette davies strength stability
0 2 0.74 0.41 1.00 1.00
1 3 0.97 0.03 1.00 1.00
2 4 0.80 0.41 0.69 0.42
3 5 0.57 0.76 0.44 0.59
4 6 0.35 0.98 0.44 0.67
5 7 0.30 1.03 0.31 0.62
k
3

Fit and check#

model = make("clusterer", "kmeans", n_clusters=k)
labels = model.fit_predict(shapes)
validation.internal(shapes, labels), validation.external(labels, population.labels)
({'silhouette': 0.9749082728449507,
  'calinski': 36096.83062214639,
  'davies': 0.03406303662333405,
  'balance': 0.9999999999999998},
 {'ari': 1.0, 'nmi': 1.0, 'purity': 1.0})

When the true grouping is known, as it is here, validation.external measures agreement with it. On real data there is no truth, which is why the resampling checks matter.

Combine clusterers#

Consensus clusters the same data with several clusterers, or with one clusterer over several seeds, and groups the series that are most often put together.

from milia.cluster.consensus import Consensus

consensus = Consensus(n_clusters=k).fit_predict(shapes)
[int((consensus == group).sum()) for group in range(k)]
[20, 20, 20]

Say when a series is ambiguous#

conformal.sets calibrates a distance threshold on held-out series, so that a new series gets a set of groups: one when it clearly belongs, several when it is between groups, none when it is far from all of them.

from milia.cluster import conformal

sets = conformal.sets(shapes, make("clusterer", "kmeans", n_clusters=k), alpha=0.1)
sets.tau, sets.alpha
(0.06547433293370267, 0.1)

Limits#

k-means assumes compact, similarly sized groups. The resampling checks tell you when a split is not stable, not whether the groups are useful for your decision. Read the clusters, not only the indices.