milia.cluster.selection#

Choosing the number of clusters.

recommend() reads a sweep() table under one strategy. composite() is the trust gate: separation quality multiplied by the weakest trust metric, so that perfect separation of an isolated outlier cannot buy back a missing trust signal.

Module Attributes

BOOKKEEPING

bookkeeping that the ranking never reads.

MONOTONE

Columns that improve as k grows by construction (inertia falls, BIC tends to follow), so ranking them rewards a large k whatever the data are.

Functions

best(scores, metric)

The k with the best value of one metric, in the metric's own direction.

borda(scores[, indices, minimum, diversity])

Diversity-weighted Borda ranking of the candidates k over the validity indices of a sweep.

composite(candidates[, separation, trust, ...])

Rank candidates by separation quality gated by trust.

gates(candidates, options[, key])

Rank of every candidate under several choices of trust metrics, side by side.

recommend(scores[, strategy, tiebreak, ...])

Recommended number of clusters from a sweep table.

sized(scores, minimum)

Keep the candidates whose smallest cluster has at least minimum members.

twostage(X, ks[, estimator, top, by, trust, ...])

Cheap validity sweep over every k, then the expensive resampling audit on the best few only.

utility(X, ks, outcomes, estimator[, ...])

Choose k by what the clusters explain: the spread of downstream outcomes across clusters.

milia.cluster.selection.best(scores, metric)#

The k with the best value of one metric, in the metric’s own direction.

Parameters:
Return type:

int

Returns:

The chosen k.

Raises:

ValueError – If the metric is missing or entirely NaN.

Examples

>>> import pandas as pd
>>> best(pd.DataFrame({"k": [2, 3, 4], "davies": [0.9, 0.4, 0.6]}), "davies")
3
milia.cluster.selection.sized(scores, minimum)#

Keep the candidates whose smallest cluster has at least minimum members.

No validity index detects a small, compact, well-separated cluster, because isolating a few distant points is rewarded by every separation index and is often reproducible. A minimum size is a constraint of the decision the clusters serve, for example the (1 - alpha) / alpha calibration points a conformal threshold needs in each group (nine at alpha = 0.1), and is applied before the candidates are ranked.

Parameters:
  • scores (DataFrame) – Sweep table with k and smallest columns.

  • minimum (int) – Smallest admissible cluster size.

Return type:

DataFrame

Returns:

The admissible rows, in their original order.

Raises:

ValueError – If the table has no smallest column, or no candidate is admissible.

Examples

>>> import pandas as pd
>>> table = pd.DataFrame({"k": [3, 4, 5], "smallest": [30, 30, 4]})
>>> sized(table, 9)["k"].tolist()
[3, 4]
milia.cluster.selection.BOOKKEEPING = ('error', 'smallest', 'strengthstd')#

bookkeeping that the ranking never reads.

Type:

Columns of a sweep that are not validity indices

milia.cluster.selection.MONOTONE = ('inertia', 'bic')#

Columns that improve as k grows by construction (inertia falls, BIC tends to follow), so ranking them rewards a large k whatever the data are. They stay in the sweep for the elbow and for reporting.

milia.cluster.selection.borda(scores, indices=None, minimum=None, diversity=True)#

Diversity-weighted Borda ranking of the candidates k over the validity indices of a sweep.

Each index is oriented so that higher is better and ranked across the candidates. Weights from weights() give less say to indices that mostly repeat others. Three choices make the ranking stable:

  • indices that improve with k by construction (MONOTONE) are not ranked, since they would reward a large k whatever the data are;

  • the weights are computed over every candidate of the sweep, also when a minimum cluster size removes some candidates, so that a gate does not change how the indices are weighted from a handful of remaining rows;

  • equal totals go to the smaller k, so the result does not depend on the order of the rows.

Parameters:
  • scores (DataFrame) – Sweep table with a k column.

  • indices (Sequence[str] | None) – Indices to rank; every validity index present in the sweep when omitted.

  • minimum (int | None) – Smallest admissible cluster size (see sized()); every candidate when omitted.

  • diversity (bool) – Weight the indices by their independence; equal weights (the plain Borda count) when false.

Return type:

DataFrame

Returns:

The admissible candidates indexed by k with a weighted column, best first; the weights are in attrs["weights"].

Raises:

ValueError – If no index has any value, or no candidate is admissible.

Examples

>>> import pandas as pd
>>> table = pd.DataFrame(
...     {"k": [2, 3, 4], "silhouette": [0.4, 0.7, 0.6], "davies": [0.9, 0.5, 0.6], "inertia": [9.0, 5.0, 3.0]}
... )
>>> int(borda(table).index[0]), list(borda(table).attrs["weights"])
(3, ['silhouette', 'davies'])
milia.cluster.selection.composite(candidates, separation=('silhouette', 'calinski', 'davies'), trust=('balance', 'stability'), key='k', extra=())#

Rank candidates by separation quality gated by trust.

Separation-only metrics cannot tell a real archetype from one outlier isolated in its own cluster, since an isolated point is perfectly separated. Here the separation metrics are combined by diversity-weighted Borda rank and scaled to [0, 1], the trust factor is the minimum of the trust metrics (NaN if any is missing), and the composite is their product, so no amount of separation buys back a missing trust signal.

Parameters:
  • candidates (DataFrame) – One row per candidate, e.g. a sweep table.

  • separation (Sequence[str]) – Columns combined into the separation score.

  • trust (Sequence[str]) – Columns, each roughly in [0, 1] and higher-is-better, whose minimum is the trust factor.

  • key (str) – Column identifying each candidate ("k" for a sweep, or a trial id).

  • extra (Sequence[str]) – Columns carried through to the result.

Return type:

DataFrame

Returns:

key, separation, trust, composite and extra columns, best first.

Examples

>>> import pandas as pd
>>> table = pd.DataFrame(
...     {
...         "k": [2, 3],
...         "silhouette": [0.9, 0.6],
...         "calinski": [9.0, 6.0],
...         "davies": [0.2, 0.5],
...         "balance": [0.1, 0.9],
...         "stability": [0.95, 0.9],
...     }
... )
>>> int(composite(table).iloc[0]["k"])  # k=2 separates best but isolates an outlier
3
milia.cluster.selection.recommend(scores, strategy='davies', tiebreak='silhouette', ranking=False, separation=('silhouette', 'calinski', 'davies'), trust=('strength', 'stability'), minimum=None)#

Recommended number of clusters from a sweep table.

Strategies:

  • a metric name ("silhouette", "davies", "bic", …): best by that metric;

  • "elbow": largest bend of the inertia curve (exploratory only: inertia always falls with k);

  • "consensus": majority vote of the separation, DBCV and resampling metrics present, ties broken by tiebreak;

  • "rank": diversity-weighted Borda over the validity indices (borda());

  • "trust": composite(), needing a sweep run with resampling=True. Preferred whenever resampling metrics exist.

Parameters:
  • scores (DataFrame) – Output of sweep().

  • strategy (str) – See above.

  • tiebreak (str) – Metric breaking a consensus tie.

  • ranking (bool) – Also return the ranking table ("rank" and "trust").

  • separation (Sequence[str]) – Separation metrics for "trust".

  • trust (Sequence[str]) – Trust metrics for "trust".

  • minimum (int | None) – Smallest admissible cluster size: candidates with a smaller cluster are removed first (sized()), and the weights of "rank" still come from every candidate.

Return type:

int | tuple[int, DataFrame]

Returns:

The recommended k, or (k, table) when ranking is set.

Raises:

ValueError – If the strategy is unknown or its metrics are missing.

Examples

>>> import pandas as pd
>>> scores = pd.DataFrame(
...     {
...         "k": [2, 3, 4],
...         "silhouette": [0.5, 0.8, 0.6],
...         "davies": [0.7, 0.4, 0.6],
...         "calinski": [50.0, 90.0, 70.0],
...     }
... )
>>> recommend(scores, "silhouette"), recommend(scores, "consensus"), recommend(scores, "rank")
(3, 3, 3)
milia.cluster.selection.gates(candidates, options, key='k', **kwargs)#

Rank of every candidate under several choices of trust metrics, side by side.

Shows whether a conclusion depends on the gate, e.g. with and without balance.

Parameters:
Return type:

DataFrame

Returns:

One row per candidate, one rank column per gate, and moves (largest rank change).

Examples

>>> import pandas as pd
>>> table = pd.DataFrame(
...     {
...         "k": [2, 3],
...         "silhouette": [0.9, 0.6],
...         "calinski": [9.0, 6.0],
...         "davies": [0.2, 0.5],
...         "balance": [0.1, 0.9],
...         "stability": [0.95, 0.9],
...     }
... )
>>> gates(table, {"both": ("balance", "stability"), "stability": ("stability",)})["moves"].tolist()
[1, 1]
milia.cluster.selection.twostage(X, ks, estimator=None, top=4, by='silhouette', trust=('balance', 'stability'), random_state=0, **kwargs)#

Cheap validity sweep over every k, then the expensive resampling audit on the best few only.

Parameters:
Return type:

DataFrame

Returns:

The audited candidates ranked by composite(), with their sweep metrics.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> centres = np.repeat(np.array([[0.0, 0.0], [6.0, 6.0], [0.0, 6.0]]), 20, axis=0)
>>> X, truth = centres + rng.normal(scale=0.5, size=(60, 2)), np.repeat([0, 1, 2], 20)
>>> int(twostage(X, range(2, 6), top=2, splits=2, bootstrap=3, jobs=1).iloc[0]["k"])
3
milia.cluster.selection.utility(X, ks, outcomes, estimator, others=None, rows=None)#

Choose k by what the clusters explain: the spread of downstream outcomes across clusters.

For each k: the spread (maximum minus minimum of cluster means) of every outcome column, the number of clusters represented among the scored entities, and the smallest agreement with the other clusterers.

Parameters:
  • X (ArrayLike) – Data.

  • ks (Sequence[int]) – Candidate numbers of clusters.

  • outcomes (DataFrame) – One row per scored entity, one column per outcome (e.g. peak voltage).

  • estimator (BaseEstimator) – Main clusterer.

  • others (Mapping[str, BaseEstimator] | None) – Other clusterers to measure agreement against.

  • rows (ArrayLike | None) – Rows of X the outcome rows belong to; the first len(outcomes) rows by default.

Return type:

DataFrame

Returns:

One row per k with <outcome>spread, represented and minari.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> centres = np.repeat(np.array([[0.0, 0.0], [6.0, 6.0], [0.0, 6.0]]), 20, axis=0)
>>> X, truth = centres + rng.normal(scale=0.5, size=(60, 2)), np.repeat([0, 1, 2], 20)
>>> from milia import make
>>> kmeans = make("clusterer", "kmeans", n_init=5)
>>> import pandas as pd
>>> table = utility(X, [2, 3], pd.DataFrame({"risk": truth * 1.0}), kmeans)
>>> float(table.loc[table["k"] == 3, "riskspread"].item())  # three clusters separate risk fully
2.0