milia.cluster.describe#

Describing clusters: when they peak, what share of energy falls in a window, what they explain.

Functions

association(mask, attribute[, test])

Association between group membership and a categorical attribute (e.g. tariff).

contrast(means, peak, floor)

Mean in a peak window minus mean in a floor window, per cluster mean profile.

describe(profiles, groups[, window])

Per group: size, peak step, share of daily energy in a time window, and mean daily energy.

explained(values, groups[, permutations, ...])

Share of the variance of an outcome that lies between groups, against its chance level.

outcomes(labels, frame[, agg])

Downstream outcomes summarised by cluster: does the archetype explain risk?

milia.cluster.describe.describe(profiles, groups, window=(0, 12))#

Per group: size, peak step, share of daily energy in a time window, and mean daily energy.

Parameters:
  • profiles (DataFrame) – Mean daily profiles, one row per entity, one column per step.

  • groups (ArrayLike) – Group label of each entity.

  • window (tuple[int, int]) – [start, end) steps, e.g. midnight to 06:00 on half-hourly data is (0, 12).

Return type:

DataFrame

Returns:

One row per group with size, peak, share and energy.

Examples

>>> import pandas as pd
>>> profiles = pd.DataFrame([[4.0, 1.0, 1.0, 2.0], [1.0, 1.0, 4.0, 2.0]])
>>> describe(profiles, [0, 1], window=(0, 1))[["peak", "share"]].values.tolist()
[[0.0, 0.5], [2.0, 0.125]]
milia.cluster.describe.contrast(means, peak, floor)#

Mean in a peak window minus mean in a floor window, per cluster mean profile.

For example evening peak minus midday floor separates evening-peaking archetypes from daytime ones.

Parameters:
  • means (ArrayLike) – Cluster mean profiles, shape (clusters, steps) or (steps,).

  • peak (tuple[int, int]) – [start, end) steps of the peak window.

  • floor (tuple[int, int]) – [start, end) steps of the floor window.

Return type:

ndarray

Returns:

One contrast per profile.

Examples

>>> contrast([[0, 1, 5, 5], [5, 5, 1, 0]], (2, 4), (0, 2)).tolist()
[4.5, -4.5]
milia.cluster.describe.outcomes(labels, frame, agg=('mean', 'count'))#

Downstream outcomes summarised by cluster: does the archetype explain risk?

Parameters:
  • labels (ArrayLike) – Cluster label of each row.

  • frame (DataFrame) – One row per entity, one column per outcome.

  • agg (tuple[str, ...]) – Aggregations per outcome.

Return type:

DataFrame

Returns:

Table indexed by cluster with an (outcome, aggregation) column per pair.

Examples

>>> import pandas as pd
>>> float(outcomes([0, 0, 1], pd.DataFrame({"v": [1.0, 3.0, 5.0]})).loc[0, ("v", "mean")])
2.0
milia.cluster.describe.association(mask, attribute, test='fisher')#

Association between group membership and a categorical attribute (e.g. tariff).

Parameters:
  • mask (ArrayLike) – Boolean group membership.

  • attribute (ArrayLike) – Attribute of each entity.

  • test (Literal['fisher', 'chi2']) – "fisher" (exact, binary attribute only) or "chi2".

Return type:

dict[str, float]

Returns:

statistic and p.

Raises:

ValueError – If Fisher’s test is asked for a non-binary attribute.

Examples

>>> mask = [True] * 10 + [False] * 10
>>> association(mask, ["tou"] * 9 + ["std"] * 11)["p"] < 0.01
True
milia.cluster.describe.explained(values, groups, permutations=1000, random_state=0)#

Share of the variance of an outcome that lies between groups, against its chance level.

The correlation ratio, eta squared, is the between-group sum of squares over the total. It is positive for any grouping, and under no association its expectation is about (G - 1) / (N - 1) for G groups and N units, so a share is an effect only against that level (Kelley 1935; Olejnik and Algina 2003). This function returns the ratio, its bias-corrected form epsilon squared [SS_B - (G - 1) MS_W] / SS_T, the mean and 95th percentile of the ratio when the units are assigned to the groups at random, and the permutation p-value (1 + #{eta_b >= eta}) / (B + 1) (Phipson and Smyth 2010).

Parameters:
  • values (ArrayLike) – Outcome of each unit.

  • groups (ArrayLike) – Group of each unit.

  • permutations (int) – Number of random assignments of the units, B.

  • random_state (int | None) – Seed of the permutations.

Return type:

dict[str, float]

Returns:

eta2, epsilon2, expected (the analytic null expectation), null_mean, null_p95 and p.

Raises:

ValueError – If there are fewer than two groups or the outcome is constant.

Examples

>>> rng = np.random.default_rng(0)
>>> groups = np.repeat(np.arange(5), 40)
>>> noise = explained(rng.normal(size=200), groups, permutations=199)
>>> bool(noise["p"] > 0.05) and abs(noise["eta2"] - noise["expected"]) < 0.05
True
>>> signal = explained(groups + rng.normal(size=200), groups, permutations=199)
>>> bool(signal["p"] < 0.01) and signal["epsilon2"] > 0.5
True