milia.cluster.search#
Multi-objective search over whole clustering pipelines (Optuna, NSGA-II).
Each trial picks a feature set, a scaler, a reducer (with its own
hyperparameters), a clusterer and k, and is scored on three objectives, all
maximised: separation, cluster balance, and a cheap prediction-strength proxy
for stability. With gated=True the search is single-objective: separation,
with trials whose weaker trust signal falls below threshold discarded.
The Pareto front is only a shortlist: audit the chosen trial with the full
resampling checks (audit()) before trusting it.
Module Attributes
The ten named feature sets compared by the search, as |
|
Reducers the search chooses between ( |
Functions
|
Rebuild one trial's input and run the full validity and resampling sweep over k. |
|
The clusterers the search chooses between. |
|
The clustering input a trial built: its feature set, scaled and reduced. |
|
The ten named feature sets of |
|
Hyperparameter importance for each objective under each evaluator. |
|
The Optuna objective for |
|
The Pareto-optimal trials of a multi-objective study. |
|
The reducer a finished trial used, rebuilt from its parameters. |
|
Null-calibrated trust of several trials at their own k. |
|
Run, or resume, the pipeline search on a meter pivot. |
|
A reducer with hyperparameters sampled from an Optuna trial. |
- milia.cluster.search.FEATURESETS: dict[str, dict[str, Any]] = {'autocorr': {'autocorr': True, 'profile': False}, 'calendar': {'calendar': True, 'profile': False}, 'distribution': {'distribution': True, 'profile': False}, 'magnitude': {'magnitude': True, 'profile': False}, 'profile': {}, 'seasonal': {'period': 'seasonal'}, 'shape': {'profile': False, 'shape': True}, 'trend': {'profile': False, 'trend': True}, 'volatility': {'profile': False, 'volatility': True}, 'weekly': {'period': 'weekly'}}#
The ten named feature sets compared by the search, as
ProfileConfigsettings.
- milia.cluster.search.REDUCERS = ('pca', 'kernelpca', 'tsne', 'umap', 'pacmap')#
Reducers the search chooses between (
"none"only whenraw=True).
- milia.cluster.search.audit(params, sets, choices=None, ks=range(2, 11), splits=10, bootstrap=100)#
Rebuild one trial’s input and run the full validity and resampling sweep over k.
The search’s stability proxy is cheap by design; this is the careful check to run on the trial you intend to use.
- Parameters:
sets (
Mapping[str,DataFrame]) – Named feature sets, fromfeaturesets().choices (
Mapping[str,BaseEstimator] |None) – Named clusterers;clusterers()withextended=Trueby default.splits (
int) – Prediction-strength splits.bootstrap (
int) – Bootstrap resamples.
- Return type:
- Returns:
(X, scores, k)– the rebuilt input, the sweep table, and the k the trust gate recommends.
Examples
>>> from milia.data import synthetic >>> sets = featuresets(synthetic.generate(meters=12, days=21, steps=24).pivot) >>> params = { ... "feature": "shape", ... "scaler": "standard", ... "reducer": "pca", ... "pcacomponents": 2, ... "clusterer": "kmeans", ... "k": 3, ... } >>> X, scores, k = audit(params, sets, ks=range(2, 4), splits=2, bootstrap=2) >>> X.shape, k in (2, 3) ((12, 2), True)
- milia.cluster.search.clusterers(extended=False)#
The clusterers the search chooses between.
- Parameters:
extended (
bool) – Add mini-batch k-means, BIRCH, spectral and average-linkage agglomerative to k-means, GMM and Ward.- Return type:
- Returns:
Name to unfitted clusterer.
Examples
>>> sorted(clusterers()), len(clusterers(extended=True)) (['agglomerative', 'gmm', 'kmeans'], 7)
- milia.cluster.search.embed(params, sets)#
The clustering input a trial built: its feature set, scaled and reduced.
- Parameters:
- Return type:
- Returns:
Array of shape
(meters, dimensions).
Examples
>>> from milia.data import synthetic >>> sets = featuresets(synthetic.generate(meters=12, days=21, steps=24).pivot) >>> params = { ... "feature": "shape", ... "scaler": "standard", ... "reducer": "pca", ... "pcacomponents": 2, ... "clusterer": "kmeans", ... "k": 3, ... } >>> embed(params, sets).shape (12, 2)
- milia.cluster.search.featuresets(pivot, hemisphere='south', normalized=False)#
The ten named feature sets of
FEATURESETS, extracted from a pivot.- Parameters:
- Return type:
- Returns:
Set name to a meters-by-features frame, NaN filled with zero.
Examples
>>> from milia.data import synthetic >>> sets = featuresets(synthetic.generate(meters=12, days=21, steps=24).pivot) >>> sorted(sets)[:3], sets["shape"].shape (['autocorr', 'calendar', 'distribution'], (12, 3))
- milia.cluster.search.importance(study, names=('separation', 'balance', 'stability'), evaluators=('fanova', 'pedanova'), random_state=0)#
Hyperparameter importance for each objective under each evaluator.
Comparing fANOVA with PED-ANOVA guards against reading an artefact of one evaluator as a finding.
- Parameters:
- Return type:
- Returns:
Long table of
evaluator,objective,parameterandimportance.
Examples
>>> table = importance(study, evaluators=("fanova",)) >>> sorted(table["objective"].unique()) ['balance', 'separation', 'stability']
- milia.cluster.search.objective(sets, choices, scalers=('standard', 'minmax', 'robust', 'quantile', 'power'), separation='silhouette', dbcv=False, embedding=False, gated=False, threshold=0.7, splits=3, raw=False, kmin=3, kmax=12)#
The Optuna objective for
search().- Parameters:
choices (
Mapping[str,BaseEstimator]) – Named clusterers.separation (
Literal['silhouette','densitysilhouette']) – Separation metric;"densitysilhouette"reproduces ark’s searches.dbcv (
bool) – Use the lower of separation and DBCV (kNN core distance) as separation.embedding (
bool) – Search the output dimension of t-SNE, UMAP and PaCMAP too.gated (
bool) – Single-objective search with a trust floor.threshold (
float) – Trust floor forgated.splits (
int) – Prediction-strength splits of the stability proxy.raw (
bool) – Allow no reduction.kmin (
int) – Smallest k.kmax (
int) – Largest k.
- Returns:
A function of an Optuna trial returning separation, balance and the stability proxy (or separation alone when
gated). Invalid configurations are pruned.
Examples
>>> from milia.data import synthetic >>> sets = featuresets(synthetic.generate(meters=12, days=21, steps=24).pivot) >>> trial = optuna.trial.FixedTrial( ... { ... "feature": "shape", ... "scaler": "standard", ... "reducer": "pca", ... "pcacomponents": 2, ... "clusterer": "kmeans", ... "k": 3, ... } ... ) >>> len(objective(sets, clusterers())(trial)) 3
- milia.cluster.search.pareto(study, names=('separation', 'balance', 'stability'))#
The Pareto-optimal trials of a multi-objective study.
- Parameters:
- Return type:
- Returns:
One row per Pareto trial –
trial, its parameters and its objective values.
Examples
>>> import tempfile >>> from pathlib import Path >>> from milia.data import synthetic >>> pivot = synthetic.generate(meters=20, days=14, steps=24).pivot >>> study = search(pivot, trials=4, name="front", storage=Path(tempfile.mkdtemp()), population=2, kmax=4) >>> {"trial", "separation", "balance", "stability"} <= set(pareto(study).columns) True
- milia.cluster.search.rebuild(params)#
The reducer a finished trial used, rebuilt from its parameters.
- Parameters:
params (
Mapping[str,Any]) – The trial’s parameters (a row ofpareto()ortrial.params).- Return type:
- Returns:
The unfitted reducer, or
Nonewhen the trial used no reduction.- Raises:
ValueError – If the reducer name is unknown.
Examples
>>> params = { ... "feature": "shape", ... "scaler": "standard", ... "reducer": "pca", ... "pcacomponents": 2, ... "clusterer": "kmeans", ... "k": 3, ... } >>> type(rebuild(params)).__name__ 'PCA'
- milia.cluster.search.recheck(candidates, sets, choices=None, **settings)#
Null-calibrated trust of several trials at their own k.
Raw stability favours small k; the excess over a structureless reference does not, so this is the fairer comparison between Pareto candidates (see
calibrated()).- Parameters:
- Return type:
- Returns:
One row per candidate – its pipeline,
kand the calibrated metrics.
Examples
>>> from milia.data import synthetic >>> sets = featuresets(synthetic.generate(meters=12, days=21, steps=24).pivot) >>> params = { ... "feature": "shape", ... "scaler": "standard", ... "reducer": "pca", ... "pcacomponents": 2, ... "clusterer": "kmeans", ... "k": 3, ... } >>> table = recheck([params], sets, draws=1, splits=2, resamples=2, reps=2) >>> "excess" in table.columns True
- milia.cluster.search.search(pivot, trials, name, storage, hemisphere='south', choices=None, normalized=False, population=60, random_state=0, every=20, **settings)#
Run, or resume, the pipeline search on a meter pivot.
The study is stored in
storage/<name>.dband resumed when it exists, so only the missing trials run.- Parameters:
pivot (
DataFrame) – Meter pivot.trials (
int) – Total trials wanted.name (
str) – Study name.storage (
Path) – Directory of the SQLite study.hemisphere (
Literal['north','south']) – Season convention for the seasonal feature set.choices (
Mapping[str,BaseEstimator] |None) – Named clusterers;clusterers()by default.normalized (
bool) – Daily-max normalise before the profile feature sets.population (
int) – NSGA-II population size.random_state (
int) – Sampler seed.every (
int|None) – Log progress every this many trials (Nonefor silence).**settings (
Any) – Passed toobjective().
- Return type:
Study- Returns:
The study;
study.trials_dataframe()gives the table.- Raises:
ValueError – If an existing study has a different number of objectives.
Examples
>>> import tempfile >>> from pathlib import Path >>> from milia.data import synthetic >>> pivot = synthetic.generate(meters=20, days=14, steps=24).pivot >>> study = search(pivot, trials=4, name="demo", storage=Path(tempfile.mkdtemp()), population=2, kmax=4) >>> len(study.trials) 4
- milia.cluster.search.suggest(trial, name, cap, embedding=False)#
A reducer with hyperparameters sampled from an Optuna trial.
PCA and kernel PCA always search their dimension; t-SNE, UMAP and PaCMAP stay at two components unless
embeddingis set.- Parameters:
- Return type:
- Returns:
The unfitted reducer, or
Nonefor"none".- Raises:
ValueError – If the name is unknown.
Examples
>>> trial = optuna.trial.FixedTrial({"pcacomponents": 3}) >>> suggest(trial, "pca", cap=5).n_components 3