milia.cluster.selection#
Choosing the number of clusters.
recommend() reads a sweep() table under
one strategy. composite() is the trust gate: separation quality
multiplied by the weakest trust metric, so that perfect separation of an
isolated outlier cannot buy back a missing trust signal.
Module Attributes
bookkeeping that the ranking never reads. |
|
Columns that improve as k grows by construction (inertia falls, BIC tends to follow), so ranking them rewards a large k whatever the data are. |
Functions
|
The k with the best value of one metric, in the metric's own direction. |
|
Diversity-weighted Borda ranking of the candidates k over the validity indices of a sweep. |
|
Rank candidates by separation quality gated by trust. |
|
Rank of every candidate under several choices of trust metrics, side by side. |
|
Recommended number of clusters from a sweep table. |
|
Keep the candidates whose smallest cluster has at least |
|
Cheap validity sweep over every k, then the expensive resampling audit on the best few only. |
|
Choose k by what the clusters explain: the spread of downstream outcomes across clusters. |
- milia.cluster.selection.best(scores, metric)#
The k with the best value of one metric, in the metric’s own direction.
- Parameters:
- Return type:
- Returns:
The chosen k.
- Raises:
ValueError – If the metric is missing or entirely NaN.
Examples
>>> import pandas as pd >>> best(pd.DataFrame({"k": [2, 3, 4], "davies": [0.9, 0.4, 0.6]}), "davies") 3
- milia.cluster.selection.sized(scores, minimum)#
Keep the candidates whose smallest cluster has at least
minimummembers.No validity index detects a small, compact, well-separated cluster, because isolating a few distant points is rewarded by every separation index and is often reproducible. A minimum size is a constraint of the decision the clusters serve, for example the
(1 - alpha) / alphacalibration points a conformal threshold needs in each group (nine atalpha = 0.1), and is applied before the candidates are ranked.- Parameters:
- Return type:
- Returns:
The admissible rows, in their original order.
- Raises:
ValueError – If the table has no
smallestcolumn, or no candidate is admissible.
Examples
>>> import pandas as pd >>> table = pd.DataFrame({"k": [3, 4, 5], "smallest": [30, 30, 4]}) >>> sized(table, 9)["k"].tolist() [3, 4]
- milia.cluster.selection.BOOKKEEPING = ('error', 'smallest', 'strengthstd')#
bookkeeping that the ranking never reads.
- Type:
Columns of a sweep that are not validity indices
- milia.cluster.selection.MONOTONE = ('inertia', 'bic')#
Columns that improve as k grows by construction (inertia falls, BIC tends to follow), so ranking them rewards a large k whatever the data are. They stay in the sweep for the elbow and for reporting.
- milia.cluster.selection.borda(scores, indices=None, minimum=None, diversity=True)#
Diversity-weighted Borda ranking of the candidates k over the validity indices of a sweep.
Each index is oriented so that higher is better and ranked across the candidates. Weights from
weights()give less say to indices that mostly repeat others. Three choices make the ranking stable:indices that improve with k by construction (
MONOTONE) are not ranked, since they would reward a large k whatever the data are;the weights are computed over every candidate of the sweep, also when a minimum cluster size removes some candidates, so that a gate does not change how the indices are weighted from a handful of remaining rows;
equal totals go to the smaller k, so the result does not depend on the order of the rows.
- Parameters:
scores (
DataFrame) – Sweep table with akcolumn.indices (
Sequence[str] |None) – Indices to rank; every validity index present in the sweep when omitted.minimum (
int|None) – Smallest admissible cluster size (seesized()); every candidate when omitted.diversity (
bool) – Weight the indices by their independence; equal weights (the plain Borda count) when false.
- Return type:
- Returns:
The admissible candidates indexed by k with a
weightedcolumn, best first; the weights are inattrs["weights"].- Raises:
ValueError – If no index has any value, or no candidate is admissible.
Examples
>>> import pandas as pd >>> table = pd.DataFrame( ... {"k": [2, 3, 4], "silhouette": [0.4, 0.7, 0.6], "davies": [0.9, 0.5, 0.6], "inertia": [9.0, 5.0, 3.0]} ... ) >>> int(borda(table).index[0]), list(borda(table).attrs["weights"]) (3, ['silhouette', 'davies'])
- milia.cluster.selection.composite(candidates, separation=('silhouette', 'calinski', 'davies'), trust=('balance', 'stability'), key='k', extra=())#
Rank candidates by separation quality gated by trust.
Separation-only metrics cannot tell a real archetype from one outlier isolated in its own cluster, since an isolated point is perfectly separated. Here the separation metrics are combined by diversity-weighted Borda rank and scaled to
[0, 1], the trust factor is the minimum of the trust metrics (NaN if any is missing), and the composite is their product, so no amount of separation buys back a missing trust signal.- Parameters:
candidates (
DataFrame) – One row per candidate, e.g. a sweep table.separation (
Sequence[str]) – Columns combined into the separation score.trust (
Sequence[str]) – Columns, each roughly in[0, 1]and higher-is-better, whose minimum is the trust factor.key (
str) – Column identifying each candidate ("k"for a sweep, or a trial id).extra (
Sequence[str]) – Columns carried through to the result.
- Return type:
- Returns:
key,separation,trust,compositeandextracolumns, best first.
Examples
>>> import pandas as pd >>> table = pd.DataFrame( ... { ... "k": [2, 3], ... "silhouette": [0.9, 0.6], ... "calinski": [9.0, 6.0], ... "davies": [0.2, 0.5], ... "balance": [0.1, 0.9], ... "stability": [0.95, 0.9], ... } ... ) >>> int(composite(table).iloc[0]["k"]) # k=2 separates best but isolates an outlier 3
- milia.cluster.selection.recommend(scores, strategy='davies', tiebreak='silhouette', ranking=False, separation=('silhouette', 'calinski', 'davies'), trust=('strength', 'stability'), minimum=None)#
Recommended number of clusters from a sweep table.
Strategies:
a metric name (
"silhouette","davies","bic", …): best by that metric;"elbow": largest bend of the inertia curve (exploratory only: inertia always falls with k);"consensus": majority vote of the separation, DBCV and resampling metrics present, ties broken bytiebreak;"rank": diversity-weighted Borda over the validity indices (borda());"trust":composite(), needing a sweep run withresampling=True. Preferred whenever resampling metrics exist.
- Parameters:
strategy (
str) – See above.tiebreak (
str) – Metric breaking a consensus tie.ranking (
bool) – Also return the ranking table ("rank"and"trust").separation (
Sequence[str]) – Separation metrics for"trust".minimum (
int|None) – Smallest admissible cluster size: candidates with a smaller cluster are removed first (sized()), and the weights of"rank"still come from every candidate.
- Return type:
- Returns:
The recommended k, or
(k, table)whenrankingis set.- Raises:
ValueError – If the strategy is unknown or its metrics are missing.
Examples
>>> import pandas as pd >>> scores = pd.DataFrame( ... { ... "k": [2, 3, 4], ... "silhouette": [0.5, 0.8, 0.6], ... "davies": [0.7, 0.4, 0.6], ... "calinski": [50.0, 90.0, 70.0], ... } ... ) >>> recommend(scores, "silhouette"), recommend(scores, "consensus"), recommend(scores, "rank") (3, 3, 3)
- milia.cluster.selection.gates(candidates, options, key='k', **kwargs)#
Rank of every candidate under several choices of trust metrics, side by side.
Shows whether a conclusion depends on the gate, e.g. with and without balance.
- Parameters:
- Return type:
- Returns:
One row per candidate, one rank column per gate, and
moves(largest rank change).
Examples
>>> import pandas as pd >>> table = pd.DataFrame( ... { ... "k": [2, 3], ... "silhouette": [0.9, 0.6], ... "calinski": [9.0, 6.0], ... "davies": [0.2, 0.5], ... "balance": [0.1, 0.9], ... "stability": [0.95, 0.9], ... } ... ) >>> gates(table, {"both": ("balance", "stability"), "stability": ("stability",)})["moves"].tolist() [1, 1]
- milia.cluster.selection.twostage(X, ks, estimator=None, top=4, by='silhouette', trust=('balance', 'stability'), random_state=0, **kwargs)#
Cheap validity sweep over every k, then the expensive resampling audit on the best few only.
- Parameters:
X (
ArrayLike) – Data.estimator (
BaseEstimator|None) – Clusterer; k-means by default.top (
int) – Candidates audited.by (
str) – Metric that shortlists them.trust (
Sequence[str]) – Trust metrics of the finalcomposite().random_state (
int) – Seed.**kwargs – Passed to
sweep()(e.g.jobs,splits).
- Return type:
- Returns:
The audited candidates ranked by
composite(), with their sweep metrics.
Examples
>>> import numpy as np >>> rng = np.random.default_rng(0) >>> centres = np.repeat(np.array([[0.0, 0.0], [6.0, 6.0], [0.0, 6.0]]), 20, axis=0) >>> X, truth = centres + rng.normal(scale=0.5, size=(60, 2)), np.repeat([0, 1, 2], 20) >>> int(twostage(X, range(2, 6), top=2, splits=2, bootstrap=3, jobs=1).iloc[0]["k"]) 3
- milia.cluster.selection.utility(X, ks, outcomes, estimator, others=None, rows=None)#
Choose k by what the clusters explain: the spread of downstream outcomes across clusters.
For each k: the spread (maximum minus minimum of cluster means) of every outcome column, the number of clusters represented among the scored entities, and the smallest agreement with the other clusterers.
- Parameters:
X (
ArrayLike) – Data.outcomes (
DataFrame) – One row per scored entity, one column per outcome (e.g. peak voltage).estimator (
BaseEstimator) – Main clusterer.others (
Mapping[str,BaseEstimator] |None) – Other clusterers to measure agreement against.rows (
ArrayLike|None) – Rows ofXthe outcome rows belong to; the firstlen(outcomes)rows by default.
- Return type:
- Returns:
One row per k with
<outcome>spread,representedandminari.
Examples
>>> import numpy as np >>> rng = np.random.default_rng(0) >>> centres = np.repeat(np.array([[0.0, 0.0], [6.0, 6.0], [0.0, 6.0]]), 20, axis=0) >>> X, truth = centres + rng.normal(scale=0.5, size=(60, 2)), np.repeat([0, 1, 2], 20) >>> from milia import make >>> kmeans = make("clusterer", "kmeans", n_init=5) >>> import pandas as pd >>> table = utility(X, [2, 3], pd.DataFrame({"risk": truth * 1.0}), kmeans) >>> float(table.loc[table["k"] == 3, "riskspread"].item()) # three clusters separate risk fully 2.0