milia.cluster.search#

Multi-objective search over whole clustering pipelines (Optuna, NSGA-II).

Each trial picks a feature set, a scaler, a reducer (with its own hyperparameters), a clusterer and k, and is scored on three objectives, all maximised: separation, cluster balance, and a cheap prediction-strength proxy for stability. With gated=True the search is single-objective: separation, with trials whose weaker trust signal falls below threshold discarded.

The Pareto front is only a shortlist: audit the chosen trial with the full resampling checks (audit()) before trusting it.

Module Attributes

FEATURESETS

The ten named feature sets compared by the search, as ProfileConfig settings.

REDUCERS

Reducers the search chooses between ("none" only when raw=True).

Functions

audit(params, sets[, choices, ks, splits, ...])

Rebuild one trial's input and run the full validity and resampling sweep over k.

clusterers([extended])

The clusterers the search chooses between.

embed(params, sets)

The clustering input a trial built: its feature set, scaled and reduced.

featuresets(pivot[, hemisphere, normalized])

The ten named feature sets of FEATURESETS, extracted from a pivot.

importance(study[, names, evaluators, ...])

Hyperparameter importance for each objective under each evaluator.

objective(sets, choices[, scalers, ...])

The Optuna objective for search().

pareto(study[, names])

The Pareto-optimal trials of a multi-objective study.

rebuild(params)

The reducer a finished trial used, rebuilt from its parameters.

recheck(candidates, sets[, choices])

Null-calibrated trust of several trials at their own k.

search(pivot, trials, name, storage[, ...])

Run, or resume, the pipeline search on a meter pivot.

suggest(trial, name, cap[, embedding])

A reducer with hyperparameters sampled from an Optuna trial.

milia.cluster.search.FEATURESETS: dict[str, dict[str, Any]] = {'autocorr': {'autocorr': True, 'profile': False}, 'calendar': {'calendar': True, 'profile': False}, 'distribution': {'distribution': True, 'profile': False}, 'magnitude': {'magnitude': True, 'profile': False}, 'profile': {}, 'seasonal': {'period': 'seasonal'}, 'shape': {'profile': False, 'shape': True}, 'trend': {'profile': False, 'trend': True}, 'volatility': {'profile': False, 'volatility': True}, 'weekly': {'period': 'weekly'}}#

The ten named feature sets compared by the search, as ProfileConfig settings.

milia.cluster.search.REDUCERS = ('pca', 'kernelpca', 'tsne', 'umap', 'pacmap')#

Reducers the search chooses between ("none" only when raw=True).

milia.cluster.search.audit(params, sets, choices=None, ks=range(2, 11), splits=10, bootstrap=100)#

Rebuild one trial’s input and run the full validity and resampling sweep over k.

The search’s stability proxy is cheap by design; this is the careful check to run on the trial you intend to use.

Parameters:
Return type:

tuple[ndarray, DataFrame, int]

Returns:

(X, scores, k) – the rebuilt input, the sweep table, and the k the trust gate recommends.

Examples

>>> from milia.data import synthetic
>>> sets = featuresets(synthetic.generate(meters=12, days=21, steps=24).pivot)
>>> params = {
...     "feature": "shape",
...     "scaler": "standard",
...     "reducer": "pca",
...     "pcacomponents": 2,
...     "clusterer": "kmeans",
...     "k": 3,
... }
>>> X, scores, k = audit(params, sets, ks=range(2, 4), splits=2, bootstrap=2)
>>> X.shape, k in (2, 3)
((12, 2), True)
milia.cluster.search.clusterers(extended=False)#

The clusterers the search chooses between.

Parameters:

extended (bool) – Add mini-batch k-means, BIRCH, spectral and average-linkage agglomerative to k-means, GMM and Ward.

Return type:

dict[str, BaseEstimator]

Returns:

Name to unfitted clusterer.

Examples

>>> sorted(clusterers()), len(clusterers(extended=True))
(['agglomerative', 'gmm', 'kmeans'], 7)
milia.cluster.search.embed(params, sets)#

The clustering input a trial built: its feature set, scaled and reduced.

Parameters:
Return type:

ndarray

Returns:

Array of shape (meters, dimensions).

Examples

>>> from milia.data import synthetic
>>> sets = featuresets(synthetic.generate(meters=12, days=21, steps=24).pivot)
>>> params = {
...     "feature": "shape",
...     "scaler": "standard",
...     "reducer": "pca",
...     "pcacomponents": 2,
...     "clusterer": "kmeans",
...     "k": 3,
... }
>>> embed(params, sets).shape
(12, 2)
milia.cluster.search.featuresets(pivot, hemisphere='south', normalized=False)#

The ten named feature sets of FEATURESETS, extracted from a pivot.

Parameters:
  • pivot (DataFrame) – Meter pivot.

  • hemisphere (Literal['north', 'south']) – Season convention of the seasonal set.

  • normalized (bool) – Divide each meter by its daily maximum before the three profile sets, so they describe when energy is used rather than how much.

Return type:

dict[str, DataFrame]

Returns:

Set name to a meters-by-features frame, NaN filled with zero.

Examples

>>> from milia.data import synthetic
>>> sets = featuresets(synthetic.generate(meters=12, days=21, steps=24).pivot)
>>> sorted(sets)[:3], sets["shape"].shape
(['autocorr', 'calendar', 'distribution'], (12, 3))
milia.cluster.search.importance(study, names=('separation', 'balance', 'stability'), evaluators=('fanova', 'pedanova'), random_state=0)#

Hyperparameter importance for each objective under each evaluator.

Comparing fANOVA with PED-ANOVA guards against reading an artefact of one evaluator as a finding.

Parameters:
  • study (Study) – A finished study.

  • names (Sequence[str]) – Names of the objectives, in order.

  • evaluators (Sequence[str]) – "fanova" and/or "pedanova".

  • random_state (int) – Seed for fANOVA’s random forest, so the importances are reproducible.

Return type:

DataFrame

Returns:

Long table of evaluator, objective, parameter and importance.

Examples

>>> table = importance(study, evaluators=("fanova",))
>>> sorted(table["objective"].unique())
['balance', 'separation', 'stability']
milia.cluster.search.objective(sets, choices, scalers=('standard', 'minmax', 'robust', 'quantile', 'power'), separation='silhouette', dbcv=False, embedding=False, gated=False, threshold=0.7, splits=3, raw=False, kmin=3, kmax=12)#

The Optuna objective for search().

Parameters:
  • sets (Mapping[str, DataFrame]) – Named feature sets.

  • choices (Mapping[str, BaseEstimator]) – Named clusterers.

  • scalers (Sequence[str]) – Scaler names to choose between.

  • separation (Literal['silhouette', 'densitysilhouette']) – Separation metric; "densitysilhouette" reproduces ark’s searches.

  • dbcv (bool) – Use the lower of separation and DBCV (kNN core distance) as separation.

  • embedding (bool) – Search the output dimension of t-SNE, UMAP and PaCMAP too.

  • gated (bool) – Single-objective search with a trust floor.

  • threshold (float) – Trust floor for gated.

  • splits (int) – Prediction-strength splits of the stability proxy.

  • raw (bool) – Allow no reduction.

  • kmin (int) – Smallest k.

  • kmax (int) – Largest k.

Returns:

A function of an Optuna trial returning separation, balance and the stability proxy (or separation alone when gated). Invalid configurations are pruned.

Examples

>>> from milia.data import synthetic
>>> sets = featuresets(synthetic.generate(meters=12, days=21, steps=24).pivot)
>>> trial = optuna.trial.FixedTrial(
...     {
...         "feature": "shape",
...         "scaler": "standard",
...         "reducer": "pca",
...         "pcacomponents": 2,
...         "clusterer": "kmeans",
...         "k": 3,
...     }
... )
>>> len(objective(sets, clusterers())(trial))
3
milia.cluster.search.pareto(study, names=('separation', 'balance', 'stability'))#

The Pareto-optimal trials of a multi-objective study.

Parameters:
  • study (Study) – A finished study.

  • names (Sequence[str]) – Names of the objectives, in order.

Return type:

DataFrame

Returns:

One row per Pareto trial – trial, its parameters and its objective values.

Examples

>>> import tempfile
>>> from pathlib import Path
>>> from milia.data import synthetic
>>> pivot = synthetic.generate(meters=20, days=14, steps=24).pivot
>>> study = search(pivot, trials=4, name="front", storage=Path(tempfile.mkdtemp()), population=2, kmax=4)
>>> {"trial", "separation", "balance", "stability"} <= set(pareto(study).columns)
True
milia.cluster.search.rebuild(params)#

The reducer a finished trial used, rebuilt from its parameters.

Parameters:

params (Mapping[str, Any]) – The trial’s parameters (a row of pareto() or trial.params).

Return type:

object | None

Returns:

The unfitted reducer, or None when the trial used no reduction.

Raises:

ValueError – If the reducer name is unknown.

Examples

>>> params = {
...     "feature": "shape",
...     "scaler": "standard",
...     "reducer": "pca",
...     "pcacomponents": 2,
...     "clusterer": "kmeans",
...     "k": 3,
... }
>>> type(rebuild(params)).__name__
'PCA'
milia.cluster.search.recheck(candidates, sets, choices=None, **settings)#

Null-calibrated trust of several trials at their own k.

Raw stability favours small k; the excess over a structureless reference does not, so this is the fairer comparison between Pareto candidates (see calibrated()).

Parameters:
Return type:

DataFrame

Returns:

One row per candidate – its pipeline, k and the calibrated metrics.

Examples

>>> from milia.data import synthetic
>>> sets = featuresets(synthetic.generate(meters=12, days=21, steps=24).pivot)
>>> params = {
...     "feature": "shape",
...     "scaler": "standard",
...     "reducer": "pca",
...     "pcacomponents": 2,
...     "clusterer": "kmeans",
...     "k": 3,
... }
>>> table = recheck([params], sets, draws=1, splits=2, resamples=2, reps=2)
>>> "excess" in table.columns
True
milia.cluster.search.search(pivot, trials, name, storage, hemisphere='south', choices=None, normalized=False, population=60, random_state=0, every=20, **settings)#

Run, or resume, the pipeline search on a meter pivot.

The study is stored in storage/<name>.db and resumed when it exists, so only the missing trials run.

Parameters:
  • pivot (DataFrame) – Meter pivot.

  • trials (int) – Total trials wanted.

  • name (str) – Study name.

  • storage (Path) – Directory of the SQLite study.

  • hemisphere (Literal['north', 'south']) – Season convention for the seasonal feature set.

  • choices (Mapping[str, BaseEstimator] | None) – Named clusterers; clusterers() by default.

  • normalized (bool) – Daily-max normalise before the profile feature sets.

  • population (int) – NSGA-II population size.

  • random_state (int) – Sampler seed.

  • every (int | None) – Log progress every this many trials (None for silence).

  • **settings (Any) – Passed to objective().

Return type:

Study

Returns:

The study; study.trials_dataframe() gives the table.

Raises:

ValueError – If an existing study has a different number of objectives.

Examples

>>> import tempfile
>>> from pathlib import Path
>>> from milia.data import synthetic
>>> pivot = synthetic.generate(meters=20, days=14, steps=24).pivot
>>> study = search(pivot, trials=4, name="demo", storage=Path(tempfile.mkdtemp()), population=2, kmax=4)
>>> len(study.trials)
4
milia.cluster.search.suggest(trial, name, cap, embedding=False)#

A reducer with hyperparameters sampled from an Optuna trial.

PCA and kernel PCA always search their dimension; t-SNE, UMAP and PaCMAP stay at two components unless embedding is set.

Parameters:
  • trial (Trial) – The active Optuna trial.

  • name (str) – Reducer name from REDUCERS, or "none".

  • cap (int) – Largest dimension allowed, min(n_samples, n_features) of the input.

  • embedding (bool) – Also search the output dimension of t-SNE, UMAP and PaCMAP.

Return type:

object | None

Returns:

The unfitted reducer, or None for "none".

Raises:

ValueError – If the name is unknown.

Examples

>>> trial = optuna.trial.FixedTrial({"pcacomponents": 3})
>>> suggest(trial, "pca", cap=5).n_components
3