milia.data.prep#

Cleaning, normalisation and season labelling for meter time series.

Inputs are pivots: a pandas.DataFrame with a DatetimeIndex and one column per meter. Nothing here extracts features; that is milia.features.

Module Attributes

RESOLUTION

Resolution of a typical smart-meter energy register, in kWh.

Functions

clean(pivot[, missing])

Drop meters with too many missing readings, then fill the remaining gaps.

coverage(pivot[, minimum, freq, start, end])

Reindex onto a complete time grid and keep meters with enough real readings.

dailymax(pivot)

Divide each reading by its own meter's maximum on that day.

normalize(profiles[, axis, how])

Remove the magnitude of profiles, keeping their shape.

prune(pivot[, tol])

Drop meters whose readings barely vary, such as stuck or empty meters.

repeatratio(readings[, resolution, interval])

Ratio of repeated readings to changes of one register step, a statistic of a stuck register.

screen(cube[, limit, gap, gapdays])

Pass the meters whose readings show neither a stuck register nor communication dropouts.

seasons(index[, hemisphere])

Meteorological season of each timestamp.

validate(pivot)

Check the pivot contract every function in this module assumes.

milia.data.prep.validate(pivot)#

Check the pivot contract every function in this module assumes.

A pivot has a DatetimeIndex, at least one meter column, and no column that is entirely missing. Checking once at the boundary turns an orientation or loading bug into a clear error instead of a confusing pandas failure later.

Parameters:

pivot (DataFrame) – Meter readings.

Return type:

DataFrame

Returns:

pivot unchanged.

Raises:
  • TypeError – If the index is not a DatetimeIndex.

  • ValueError – If there are no meter columns, or a column is entirely missing.

Examples

>>> import pandas as pd, numpy as np
>>> index = pd.date_range("2024-01-01", periods=48, freq="h")
>>> pivot = pd.DataFrame({"a": np.arange(48.0), "b": np.ones(48)}, index=index)
>>> validate(pivot).shape
(48, 2)
milia.data.prep.clean(pivot, missing=0.5)#

Drop meters with too many missing readings, then fill the remaining gaps.

Interior gaps are interpolated in time; anything still missing (leading or trailing) takes the meter’s median. Dropped meters are logged.

Parameters:
  • pivot (DataFrame) – Meter readings.

  • missing (float) – Largest share of missing readings a meter may have.

Return type:

DataFrame

Returns:

The cleaned pivot.

Examples

>>> import pandas as pd, numpy as np
>>> index = pd.date_range("2024-01-01", periods=48, freq="h")
>>> pivot = pd.DataFrame({"a": np.arange(48.0), "b": np.ones(48)}, index=index)
>>> pivot.iloc[5, 0] = np.nan
>>> float(clean(pivot).iloc[5, 0])
5.0
milia.data.prep.coverage(pivot, minimum=0.99, freq=None, start=None, end=None)#

Reindex onto a complete time grid and keep meters with enough real readings.

The generic part of every dataset loader: a meter is kept only when at least minimum of the grid’s timestamps carry a real reading; the few gaps left are interpolated.

Parameters:
  • pivot (DataFrame) – Meter readings, possibly with missing timestamps.

  • minimum (float) – Smallest share of real readings a meter must have.

  • freq (str | None) – Grid frequency; inferred from the index when omitted.

  • start (str | Timestamp | None) – Grid start (inclusive); the first timestamp when omitted.

  • end (str | Timestamp | None) – Grid end (exclusive); one step past the last timestamp when omitted.

Return type:

DataFrame

Returns:

The complete-grid pivot, restricted to well-covered meters.

Examples

>>> import pandas as pd, numpy as np
>>> index = pd.date_range("2024-01-01", periods=48, freq="h")
>>> pivot = pd.DataFrame({"a": np.arange(48.0), "b": np.ones(48)}, index=index)
>>> gappy = pivot.drop(index=pivot.index[10:20])
>>> coverage(gappy, minimum=0.7, freq="h").shape
(48, 2)
>>> coverage(gappy, minimum=0.9, freq="h").shape
(48, 0)
milia.data.prep.prune(pivot, tol=1e-08)#

Drop meters whose readings barely vary, such as stuck or empty meters.

Parameters:
  • pivot (DataFrame) – Meter readings.

  • tol (float) – Smallest standard deviation a meter must have.

Return type:

DataFrame

Returns:

The pivot without the flat meters.

Examples

>>> import pandas as pd, numpy as np
>>> index = pd.date_range("2024-01-01", periods=48, freq="h")
>>> pivot = pd.DataFrame({"a": np.arange(48.0), "b": np.ones(48)}, index=index)
>>> list(prune(pivot).columns)
['a']
milia.data.prep.normalize(profiles, axis=-1, how='peak')#

Remove the magnitude of profiles, keeping their shape.

The forms differ in what they keep. peak divides by the maximum, so the ratio of demand to the peak is kept (what sizes a transformer). energy divides by the total, so each value is a share of the day’s energy (what a time-of-use tariff acts on). zscore subtracts the mean and divides by the standard deviation, keeping the form of the curve only. minmax rescales to the unit interval. A profile whose divisor is zero stays at zero rather than becoming NaN.

Parameters:
  • profiles (ArrayLike) – Array whose statistic is taken along axis: (n, steps) shapes use the default; an (n, days, steps) tensor normalised per meter uses axis=(1, 2).

  • axis (int | tuple[int, ...]) – Axis or axes over which each statistic is taken.

  • how (Literal['peak', 'energy', 'zscore', 'minmax']) – "peak", "energy", "zscore" or "minmax".

Return type:

ndarray

Returns:

The normalised array, same shape as the input.

Raises:

ValueError – If how is not one of the four forms.

Examples

>>> normalize(np.array([[1.0, 2.0], [0.0, 0.0]]))
array([[0.5, 1. ],
       [0. , 0. ]])
>>> normalize(np.array([[1.0, 3.0]]), how="energy")
array([[0.25, 0.75]])
>>> normalize(np.array([[1.0, 3.0]]), how="minmax")
array([[0., 1.]])
milia.data.prep.dailymax(pivot)#

Divide each reading by its own meter’s maximum on that day.

Removes day-to-day magnitude while keeping each day’s shape, so profiles compare when energy is used rather than how much.

Parameters:

pivot (DataFrame) – Meter readings.

Return type:

DataFrame

Returns:

The normalised pivot, each meter-day peaking at 1 (0 where the day’s maximum is 0).

Examples

>>> import pandas as pd, numpy as np
>>> index = pd.date_range("2024-01-01", periods=48, freq="h")
>>> pivot = pd.DataFrame({"a": np.arange(48.0), "b": np.ones(48)}, index=index)
>>> dailymax(pivot)["a"].resample("D").max().tolist()
[1.0, 1.0]
milia.data.prep.seasons(index, hemisphere='south')#

Meteorological season of each timestamp.

Parameters:
  • index (DatetimeIndex | Series) – Timestamps, as an index or a datetime series.

  • hemisphere (Literal['north', 'south']) – "north" (e.g. London, GoiEner) or "south" (e.g. AusNet).

Return type:

ndarray

Returns:

"spring", "summer", "autumn" or "winter" per timestamp.

Raises:

ValueError – If hemisphere is neither "north" nor "south".

Examples

>>> import pandas as pd
>>> index = pd.to_datetime(["2024-01-15", "2024-07-15"])
>>> seasons(index, "north").tolist(), seasons(index, "south").tolist()
(['winter', 'summer'], ['summer', 'winter'])
milia.data.prep.RESOLUTION = 0.001#

Resolution of a typical smart-meter energy register, in kWh.

milia.data.prep.repeatratio(readings, resolution=0.001, interval=0.5)#

Ratio of repeated readings to changes of one register step, a statistic of a stuck register.

Readings are converted to the energy each interval records, rounded to the register’s resolution, and only consecutive pairs of positive readings count. For a load that varies, a repeat is about as likely as a change of one step, so the ratio is near one; a stuck register repeats far more often. The computation is vectorised over any leading axes.

Parameters:
  • readings (ArrayLike) – Power in kW with days and intervals on the last two axes, e.g. (meters, days, 48).

  • resolution (float) – The register’s resolution in kWh.

  • interval (float) – Length of an interval in hours.

Return type:

ndarray

Returns:

The ratio for each series on the leading axes (a 0-d array for one meter); the count of repeats where no change of one step occurs.

Examples

>>> day = np.array([[0.4, 0.4, 0.402, 0.404, 0.404, 0.406]])  # two repeats, three one-step changes
>>> round(float(repeatratio(day)), 3)
1.333
milia.data.prep.screen(cube, limit=1.5, gap=4, gapdays=3)#

Pass the meters whose readings show neither a stuck register nor communication dropouts.

Parameters:
  • cube (ArrayLike) – Power in kW, shape (meters, days, steps).

  • limit (float) – Largest repeat ratio (repeatratio()) accepted.

  • gap (int) – Shortest run of zero readings counted as a dropout, in steps.

  • gapdays (int) – Largest number of days with such a run accepted.

Return type:

ndarray

Returns:

A boolean per meter, True where the meter passes.

Examples

>>> rng = np.random.default_rng(0)
>>> cube = rng.uniform(0.2, 2.0, (2, 10, 48))
>>> cube[1, :5, 10:20] = 0.0  # five days of five-hour dropouts
>>> screen(cube).tolist()
[True, False]