milia.data.profiles#

Daily profiles and day tensors built from a meter pivot.

These replace the per-notebook copies of “hour-of-day mean, then peak normalise” and of season_tensor.

Functions

daily(pivot[, dates])

Mean daily profile of each meter.

diversity(values, sizes[, draws, random_state])

Peak demand per household of a group of n households, as a share of the mean individual peak.

groupmean(values, groups)

Mean of the rows of an array in each group, such as the mean daily profile of each cluster.

medoids(values, groups)

The row of each group with the smallest total Euclidean distance to the others of its group.

pivot(cube[, start, freq])

Lay a cube of daily readings out as a meter pivot, the inverse of tensor().

select(pivot[, dates])

The rows of a pivot that fall on the given days.

shape(pivot[, dates, how])

Normalised mean daily profile of each meter, peak-normalised by default.

slot(index, count)

Position of each timestamp within its day.

steps(index)

Readings per day implied by the index spacing.

tensor(pivot[, dates, count])

Meter-by-day-by-step tensor, the input of Tucker and CROCS.

weekly(cube[, days])

Each meter's mean daily profile of every week, normalised by its largest absolute value.

milia.data.profiles.steps(index)#

Readings per day implied by the index spacing.

Parameters:

index (DatetimeIndex) – A regular DatetimeIndex.

Return type:

int

Returns:

Readings per day, e.g. 48 for half-hourly data.

Raises:

ValueError – If the spacing does not divide a day.

Examples

>>> import pandas as pd
>>> steps(pd.date_range("2024-01-01", periods=96, freq="30min"))
48
milia.data.profiles.select(pivot, dates=None)#

The rows of a pivot that fall on the given days.

Parameters:
Return type:

DataFrame

Returns:

The selected rows.

Examples

>>> import pandas as pd, numpy as np
>>> pivot = pd.DataFrame({"a": np.arange(48.0)}, index=pd.date_range("2024-01-01", periods=48, freq="h"))
>>> len(select(pivot, pd.to_datetime(["2024-01-02"])))
24
milia.data.profiles.slot(index, count)#

Position of each timestamp within its day.

Parameters:
Return type:

ndarray

Returns:

Integers from 0 to count - 1.

Examples

>>> import pandas as pd
>>> slot(pd.to_datetime(["2024-01-01 00:30", "2024-01-01 23:30"]), 48).tolist()
[1, 47]
milia.data.profiles.daily(pivot, dates=None)#

Mean daily profile of each meter.

Parameters:
  • pivot (DataFrame) – Meter readings with a regular DatetimeIndex.

  • dates (DatetimeIndex | None) – Restrict to these days, e.g. one season; all days when omitted.

Return type:

DataFrame

Returns:

One row per meter, one column per step of the day.

Examples

>>> import pandas as pd, numpy as np
>>> index = pd.date_range("2024-01-01", periods=48, freq="h")
>>> pivot = pd.DataFrame({"a": np.tile(np.arange(24.0), 2)}, index=index)
>>> daily(pivot).shape, float(daily(pivot).loc["a", 23])
((1, 24), 23.0)
milia.data.profiles.shape(pivot, dates=None, how='peak')#

Normalised mean daily profile of each meter, peak-normalised by default.

The representation most of the clustering chapters use: when in the day a meter consumes, with its size removed.

Parameters:
Return type:

ndarray

Returns:

Array of shape (meters, steps); with how="peak" each row peaks at 1.

Examples

>>> import pandas as pd, numpy as np
>>> index = pd.date_range("2024-01-01", periods=48, freq="h")
>>> pivot = pd.DataFrame({"a": np.tile(np.arange(24.0), 2)}, index=index)
>>> float(shape(pivot).max())
1.0
milia.data.profiles.tensor(pivot, dates=None, count=None)#

Meter-by-day-by-step tensor, the input of Tucker and CROCS.

Parameters:
  • pivot (DataFrame) – Meter readings over whole days.

  • dates (DatetimeIndex | None) – Restrict to these days; all days when omitted.

  • count (int | None) – Readings per day; inferred from the index when omitted.

Return type:

ndarray

Returns:

Array of shape (meters, days, steps), meters in the pivot’s column order.

Raises:

ValueError – If the rows do not form whole days.

Examples

>>> import pandas as pd, numpy as np
>>> index = pd.date_range("2024-01-01", periods=48, freq="h")
>>> pivot = pd.DataFrame({"a": np.arange(48.0), "b": np.ones(48)}, index=index)
>>> tensor(pivot).shape
(2, 2, 24)
milia.data.profiles.weekly(cube, days=7)#

Each meter’s mean daily profile of every week, normalised by its largest absolute value.

The normalisation keeps the sign, so a net load that exports keeps its negative midday.

Parameters:
  • cube (ndarray) – Readings, shape (meters, days, steps); trailing days of a partial week are dropped.

  • days (int) – Days per week.

Return type:

ndarray

Returns:

Shape (meters, weeks, steps); a week of zero readings stays zero.

Examples

>>> cube = np.ones((1, 15, 4)) * np.array([1.0, 2.0, -4.0, 0.0])
>>> weekly(cube).shape, weekly(cube)[0, 0].tolist()
((1, 2, 4), [0.25, 0.5, -1.0, 0.0])
milia.data.profiles.pivot(cube, start='2020-01-01', freq='30min')#

Lay a cube of daily readings out as a meter pivot, the inverse of tensor().

Parameters:
  • cube (ndarray) – Readings, shape (meters, days, steps).

  • start (str) – Timestamp of the first reading.

  • freq (str) – Spacing of the readings.

Return type:

DataFrame

Returns:

One row per reading and one column per meter (0 to meters - 1).

Examples

>>> pivot(np.zeros((2, 1, 48))).shape
(48, 2)
milia.data.profiles.groupmean(values, groups)#

Mean of the rows of an array in each group, such as the mean daily profile of each cluster.

Parameters:
  • values (ndarray) – Array of shape (rows, ...).

  • groups (ndarray) – Group of each row, as a vector of length rows.

Return type:

dict[object, ndarray]

Returns:

A dictionary from each group, in sorted order, to the mean of its rows, of shape values.shape[1:].

Examples

>>> means = groupmean(np.array([[1.0, 2.0], [3.0, 4.0], [10.0, 20.0]]), np.array(["a", "a", "b"]))
>>> means["a"].tolist(), means["b"].tolist()
([2.0, 3.0], [10.0, 20.0])
milia.data.profiles.medoids(values, groups)#

The row of each group with the smallest total Euclidean distance to the others of its group.

The medoid summarises a group by one of its own members, so the summary keeps the shape and the peak of a real household that a mean smooths away. The mean minimises the sum of squared distances and the medoid the sum of distances, which makes it less sensitive to outliers.

Parameters:
  • values (ndarray) – Array of shape (rows, ...); every row is flattened for the distance.

  • groups (ndarray) – Group of each row, as a vector of length rows.

Return type:

dict[object, int]

Returns:

A dictionary from each group, in sorted order, to the index of its medoid among all rows.

Examples

>>> rows = np.array([[0.0], [1.0], [1.2], [10.0]])
>>> medoids(rows, np.array([0, 0, 0, 1]))
{0: 1, 1: 3}
milia.data.profiles.diversity(values, sizes, draws=100, random_state=0)#

Peak demand per household of a group of n households, as a share of the mean individual peak.

The demands of households do not peak at the same time, so the peak of the sum of n of them is less than n times the mean peak of one. The ratio of the first to the second, the coincidence factor, falls from one for a single household towards the ratio of the peak of the mean profile to the mean peak as n grows. It is the quantity that a representative profile built from a mean omits.

Parameters:
  • values (ndarray) – Array of shape (households, steps), or (households, days, steps), in kW.

  • sizes (Sequence[int]) – Numbers of households n to draw.

  • draws (int) – Random groups of each size.

  • random_state (int) – Seed of the draws.

Return type:

DataFrame

Returns:

A frame with one row per size and the columns size, mean (coincidence factor averaged over the draws), low and high (5th and 95th percentiles).

Examples

>>> flat = np.array([[1.0, 0.0], [0.0, 1.0]])
>>> diversity(flat, [1, 2], draws=5)["mean"].round(2).tolist()
[1.0, 0.5]