milia.cluster.agreement#

Agreement between clusterings: across algorithms, seeds, time windows and known groups.

The key protocol is the noise floor: drift between two time windows is only real if their agreement is lower than the agreement between two seeds of the same algorithm on the same data (seeds()).

Module Attributes

FAMILIES

One clusterer per objective family, for families().

Functions

assign(similarity)

Match each cluster to a distinct reference class by the largest total similarity.

drift(periods, index, estimator, k)

Track one entity across periods: distance to its own centroid and to the nearest other.

families(X, k[, estimators])

Labels of the same data from one clusterer per objective family.

history(windows, reference, estimator, k)

How much history a clustering needs: each window's labels against a reference labelling.

jaccard(labels, mask)

Best Jaccard overlap between any one cluster and a known group.

overlap(labels, ids, reference)

How many entities of a reference set sit in the smallest cluster.

pairwise(labels[, metric])

Square table of agreement between every pair of labellings of the same rows.

recovery(labels, injected)

Whether injected entities land together: their dominant cluster, its recall and its purity.

seeds(X, estimator, k[, values])

Pairwise ARI between fits that differ only in random_state: the noise floor.

temporal(windows, estimator, k)

Recluster the same entities in each time window.

transitions(before, after[, normalize])

Cross-tabulation of cluster membership in two periods (e.g. first and last quarter).

milia.cluster.agreement.FAMILIES: dict[str, tuple[str, dict]] = {'agglomerative': ('agglomerative', {}), 'dbscan': ('dbscan', {'min_samples': 20}), 'gmm': ('gmm', {}), 'kmeans': ('kmeans', {}), 'spectral': ('spectral', {})}#

One clusterer per objective family, for families().

milia.cluster.agreement.pairwise(labels, metric='ari')#

Square table of agreement between every pair of labellings of the same rows.

Parameters:
  • labels (Mapping[Hashable, ArrayLike]) – Name to labels, all over the same rows.

  • metric (str) – "ari" (adjusted Rand index) or "nmi" (normalised mutual information).

Return type:

DataFrame

Returns:

Symmetric table with ones on the diagonal.

Examples

>>> pairwise({"a": [0, 0, 1, 1], "b": [1, 1, 0, 0], "c": [0, 1, 0, 1]}).round(2).loc["a"].tolist()
[1.0, 1.0, -0.5]
milia.cluster.agreement.families(X, k, estimators=None)#

Labels of the same data from one clusterer per objective family.

Partition (k-means), probabilistic (GMM), hierarchy (agglomerative), graph (spectral) and density (DBSCAN, which ignores k and may label noise -1). High agreement across families means the structure does not depend on the algorithm’s assumptions.

Parameters:
Return type:

dict[str, ndarray]

Returns:

Name to labels.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> centres = np.repeat(np.array([[0.0, 0.0], [6.0, 6.0], [0.0, 6.0]]), 20, axis=0)
>>> X, truth = centres + rng.normal(scale=0.5, size=(60, 2)), np.repeat([0, 1, 2], 20)
>>> out = families(X, 3)
>>> sorted(out), round(float(pairwise(out).loc["kmeans", "gmm"]), 2)
(['agglomerative', 'dbscan', 'gmm', 'kmeans', 'spectral'], 1.0)
milia.cluster.agreement.seeds(X, estimator, k, values=range(0, 5))#

Pairwise ARI between fits that differ only in random_state: the noise floor.

Drift between two time windows is only real if their agreement falls below this floor.

Parameters:
Return type:

ndarray

Returns:

One ARI per pair of seeds.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> centres = np.repeat(np.array([[0.0, 0.0], [6.0, 6.0], [0.0, 6.0]]), 20, axis=0)
>>> X, truth = centres + rng.normal(scale=0.5, size=(60, 2)), np.repeat([0, 1, 2], 20)
>>> from milia import make
>>> kmeans = make("clusterer", "kmeans", n_init=5)
>>> seeds(X, kmeans, 3, values=range(3)).round(2).tolist()
[1.0, 1.0, 1.0]
milia.cluster.agreement.temporal(windows, estimator, k)#

Recluster the same entities in each time window.

Parameters:
  • windows (Mapping[Hashable, ArrayLike]) – Window name to the entities’ features in that window (same rows in every window).

  • estimator (BaseEstimator) – Clusterer.

  • k (int) – Number of clusters.

Return type:

tuple[DataFrame, DataFrame]

Returns:

(summary, agreement) – per-window silhouette, and the pairwise ARI table between windows.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> centres = np.repeat(np.array([[0.0, 0.0], [6.0, 6.0], [0.0, 6.0]]), 20, axis=0)
>>> X, truth = centres + rng.normal(scale=0.5, size=(60, 2)), np.repeat([0, 1, 2], 20)
>>> from milia import make
>>> kmeans = make("clusterer", "kmeans", n_init=5)
>>> summary, table = temporal({"q1": X, "q2": X + 0.01}, kmeans, 3)
>>> len(summary), float(table.loc["q1", "q2"])
(2, 1.0)
milia.cluster.agreement.history(windows, reference, estimator, k)#

How much history a clustering needs: each window’s labels against a reference labelling.

For example labels from one day, one week and one month of data, each compared with the full-season labels.

Parameters:
Return type:

DataFrame

Returns:

One row per window with ari (against reference) and silhouette.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> centres = np.repeat(np.array([[0.0, 0.0], [6.0, 6.0], [0.0, 6.0]]), 20, axis=0)
>>> X, truth = centres + rng.normal(scale=0.5, size=(60, 2)), np.repeat([0, 1, 2], 20)
>>> from milia import make
>>> kmeans = make("clusterer", "kmeans", n_init=5)
>>> history({"day": X + rng.normal(scale=0.3, size=X.shape), "season": X}, truth, kmeans, 3)["ari"].round(
...     1
... ).tolist()
[1.0, 1.0]
milia.cluster.agreement.transitions(before, after, normalize=False)#

Cross-tabulation of cluster membership in two periods (e.g. first and last quarter).

Parameters:
  • before (ArrayLike) – Labels in the first period.

  • after (ArrayLike) – Labels of the same entities in the second period.

  • normalize (bool) – Divide each row by its total, giving transition shares.

Return type:

DataFrame

Returns:

Table with before labels as rows and after labels as columns.

Examples

>>> transitions([0, 0, 1, 1], [0, 1, 1, 1]).values.tolist()
[[1, 1], [0, 2]]
milia.cluster.agreement.drift(periods, index, estimator, k)#

Track one entity across periods: distance to its own centroid and to the nearest other.

When the second falls below the first, the entity has moved to another cluster, e.g. after adopting an EV.

Parameters:
  • periods (Mapping[Hashable, ArrayLike]) – Period name to all entities’ features in that period.

  • index (int) – Row of the tracked entity.

  • estimator (BaseEstimator) – Clusterer.

  • k (int) – Number of clusters.

Return type:

DataFrame

Returns:

One row per period with cluster, own and other.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> centres = np.repeat(np.array([[0.0, 0.0], [6.0, 6.0], [0.0, 6.0]]), 20, axis=0)
>>> X, truth = centres + rng.normal(scale=0.5, size=(60, 2)), np.repeat([0, 1, 2], 20)
>>> from milia import make
>>> kmeans = make("clusterer", "kmeans", n_init=5)
>>> table = drift({"q1": X}, 0, kmeans, 3)
>>> bool(table.loc[0, "own"] < table.loc[0, "other"])
True
milia.cluster.agreement.jaccard(labels, mask)#

Best Jaccard overlap between any one cluster and a known group.

A general check of whether a clustering recovers a group found independently.

Parameters:
  • labels (ArrayLike) – Cluster labels.

  • mask (ArrayLike) – Boolean membership of the known group.

Return type:

float

Returns:

The best overlap, from 0 to 1.

Examples

>>> jaccard([0, 0, 1, 1], [True, True, False, False])
1.0
milia.cluster.agreement.overlap(labels, ids, reference)#

How many entities of a reference set sit in the smallest cluster.

Parameters:
Return type:

dict[str, float]

Returns:

minority (its label), size, shared and reference (set size).

Examples

>>> overlap([0, 0, 0, 1], ["a", "b", "c", "d"], ["d", "x"])
{'minority': 1, 'size': 1, 'shared': 1, 'reference': 2}
milia.cluster.agreement.recovery(labels, injected)#

Whether injected entities land together: their dominant cluster, its recall and its purity.

Parameters:
  • labels (ArrayLike) – Cluster labels after the injection.

  • injected (ArrayLike) – Row indices of the injected entities.

Return type:

dict[str, float]

Returns:

cluster, recall (share of injected entities in it) and purity (share of its members that were injected).

Examples

>>> recovery([0, 0, 1, 1, 1], [2, 3])
{'cluster': 1, 'recall': 1.0, 'purity': 0.6666666666666666}
milia.cluster.agreement.assign(similarity)#

Match each cluster to a distinct reference class by the largest total similarity.

A clustering names its groups arbitrarily. When a few members of known class are available, each cluster is compared with the mean of every known class and the one-to-one assignment with the largest summed similarity (the Hungarian method) gives the cluster its class. Undefined similarities count as the lowest, so a class that is absent never wins.

Parameters:

similarity (ArrayLike) – Array of shape (clusters, classes).

Return type:

ndarray

Returns:

For each cluster the index of its class, or -1 when there are more clusters than classes.

Examples

>>> assign([[0.1, 0.9], [0.8, 0.2]]).tolist()
[1, 0]
>>> assign([[0.9, 0.1], [0.8, 0.2], [0.1, 0.0]]).tolist()
[0, 1, -1]