Experiment Engine#

ExperimentEngine runs structured rolling-origin experiments from an ExperimentSpec. It validates the selected spec, tunes or loads a fingerprint-matched backbone HPO cache, evaluates conditions across datasets and CV folds, aggregates results, and records tracking and run status. Experiment scripts construct the engine with a spec and pass a PipelineConfig to run().

Architecture#

        graph TD
    A[ExperimentSpec] --> B[ExperimentEngine.run]
    B --> C{Phase 1: Backbone HPO}
    C --> D[run_backbone_hpo per dataset and model]
    D --> E{matching fingerprinted cache?}
    E -->|no| F[Optuna tuning]
    E -->|yes| G[reuse best params]
    F --> H[atomic versioned JSON cache]
    B --> G{Phase 2: Ablation}
    G --> H[For each condition]
    H --> I[For each dataset]
    I --> J[For each CV fold]
    J --> K[load_backbone_params]
    K --> L[Apply condition overrides]
    L --> M[fit + evaluate]
    M --> N[log to MLflow]
    G --> O[aggregate per group]
    O --> P[CSV full + summary]

Phase 1 - Backbone HPO searches the backbone model’s hyperparameter space with Optuna. A JSON cache is reused only when its fingerprint matches the model, search space, fixed parameters, input data, and HPO protocol. Cache policy is selectable as reuse, refresh, or error-if-missing.

Phase 2 - Conditions evaluates each selected condition across datasets and CV folds, starting from the backbone parameters and applying condition overrides. Results are aggregated to CSV. Each run also writes a per-condition run_status.json; failed conditions are visible there, and an all-failed run raises instead of being promoted to latest.

Defining an Experiment#

ExperimentSpec#

ExperimentSpec declares the full experiment: which backbone to tune, which datasets to run on, and which conditions to compare.

from twiga.experiment import ExperimentSpec, Condition
from twiga.models.nn.mlpf_model import MLPFConfig
from twiga.models.nn.mlpffpqr_model import MLPFFPQRConfig
from twiga.utils.run_forecast import PipelineConfig

spec = ExperimentSpec(
    name="Chapter 5 FPQR example",
    output_prefix="fpqr_example",
    condition_cls=MLPFFPQRConfig,
    backbone_cls=MLPFConfig,
    datasets={"MLVS-PT": {"dataset_name": "MLVS-PT", "data_path": "examples/data"}},
    conditions=[
        Condition(name="fpqr", group="benchmark", overrides={}),
    ],
    cv_train_size=12,
    cv_test_size=4,
    cv_val_size=2,
    cv_stride=2,
    cv_folds=10,
)

base_cfg = PipelineConfig(
    dataset_name="MLVS-PT",
    data_path="examples/data",
    output_dir="results",
    train_start="2019-03",
    train_end="2021-12",
    test_start="2021-07",
    test_end="2021-12",
    input_window_size=96,
    forecast_horizon=48,
    window_stride=48,
    period="30min",
    epochs=500,
    num_trials=50,
)

engine = ExperimentEngine(spec)
summary = engine.run(
    base_cfg,
    groups=["benchmark"],
    dataset_keys=["MLVS-PT"],
    hpo_cache_policy="reuse",
)

This is the same spec shape and model pair used by experiment/chapter-5-fpqr/fpqr_experiment.py, reduced to one condition. It performs real training when run; use the chapter script’s CLI for an experiment and --help to inspect available controls.

The required fields are name, output_prefix, condition_cls, backbone_cls, conditions, and datasets. Each dataset value is a mapping of PipelineConfig field overrides, not pre-split dataframes.

Condition#

Each Condition names one ablation arm and specifies field overrides applied on top of the backbone config.

Condition(
    name="latent_gate_2g",
    group="design",
    overrides={
        "group_scheme": "2group",
        "gate_position": "latent",
        "fusion_type": "gated",
    },
    hpo_variant="2group",  # separate fingerprinted cache for this architecture
)

Field	Type	Description
`group`	`str`	Condition group selected with `--group`
`name`	`str`	Unique condition identifier within its group
`overrides`	`dict[str, Any]`	Config fields to override. Applied after backbone HPO params are loaded
`hpo_variant`	`str`	Selects a separate cache identity when a variant changes the search/model configuration

When to set hpo_variant

Two conditions that differ only in a regularisation weight can share backbone HPO results. Two conditions that differ in group_scheme or gate_position produce different network shapes and should use separate HPO caches. Set hpo_variant to a short descriptive string (e.g. "3group") to create an isolated cache for that arm.

Running an Experiment#

From Python#

from twiga.experiment import ExperimentEngine

engine = ExperimentEngine(spec)
summary = engine.run(
    base_cfg,
    groups=["benchmark"],
    dataset_keys=["MLVS-PT"],
    hpo_cache_policy="reuse",
)

run() returns a summary DataFrame. Outputs are written below <base_cfg.output_dir>/<slugified spec.name>/runs/<run_id>/. The run includes meta.json, run_status.json, result CSVs, and per-condition artifacts. latest points to the latest successful run by default. Partial runs are not promoted unless accept_partial=True; all-failed runs always raise.

From the command line#

ExperimentEngine.cli_main provides a minimal CLI entry point. Experiment scripts expose it directly:

uv run python experiment/chapter-5-fpqr/fpqr_experiment.py --help
uv run python experiment/chapter-5-fpqr/fpqr_experiment.py \
    --group benchmark --dataset MLVS-PT --hpo-cache-policy reuse

The chapter script constructs ExperimentEngine(SPEC) and calls ENGINE.cli_main(BASE_CFG). The CLI supports --group, --dataset, --skip-hpo, --hpo-cache-policy, --accept-partial, --tracking-uri, --epochs, --num-trials, and --folds. --skip-hpo requires a matching fingerprinted cache; it does not allow an unverified legacy cache.

Config Precedence#

When ExperimentEngine builds the model config for a condition it merges four layers in order (lowest to highest priority):

Config class defaults
ExperimentSpec-level overrides (applied to every condition)
Backbone HPO best parameters loaded from cache
Condition.overrides

This means a condition override always wins over HPO-tuned params, which in turn always win over spec-level settings.

HPO Cache#

Backbone HPO results are cached as versioned JSON under <experiment_root>/backbone_hpo/<dataset>/<model>_best_params.json. The envelope records a SHA-256 fingerprint of the model/search configuration, fixed search parameters, dataset contents and feature protocol, HPO window, trial budget, epoch budget, objective, and metrics. Existing flat/unversioned JSON files are treated as unverified and are not reused by reuse.

Choose a policy explicitly when invoking run_backbone_hpo() or ExperimentEngine.run():

reuse (default): use a matching cache; tune on a miss or stale fingerprint.
refresh: always run Optuna and atomically replace the cache.
error-if-missing: use a matching cache or raise FileNotFoundError; never tune.

--skip-hpo is a compatibility shortcut for error-if-missing. These files contain best parameters only; they are not persistent Optuna studies, so increasing num_trials does not resume an earlier study. Use refresh to rerun the search.

Load cached params directly with:

from twiga.experiment import load_backbone_params

params = load_backbone_params(
    dataset_key="MLVS-PT",
    hpo_cache_dir=Path("results/chapter_5_fpqr/backbone_hpo"),
    model_name_str="mlpf",
    controlled_fields=frozenset(),
)

MLflow Tracking#

Set MLFLOW_TRACKING_URI before running to enable automatic experiment tracking:

export MLFLOW_TRACKING_URI=http://localhost:5000
python experiment/chapter-7-conformal/crc_experiment.py

The run hierarchy mirrors the experiment structure:

Experiment: <spec.name>
  Parent run: <dataset>
    Child run: hpo/<dataset>/<model>      run_backbone_hpo()
    Child run: <condition>/<dataset>/fold_N

All condition overrides, fold metrics, and HPO parameters are logged as MLflow params and metrics automatically. No additional instrumentation is needed in the experiment script.

detect_tracking_uri is a safe helper that returns the configured URI or None if MLflow is absent or unconfigured:

from twiga.experiment import detect_tracking_uri

uri = detect_tracking_uri()  # None if not configured

Aggregating Results#

aggregate computes mean and standard deviation per (group, condition, metric_type) across folds and writes two CSV files: a full long-form table and a summary table.

from twiga.experiment import aggregate
from pathlib import Path

summary = aggregate(
    combined=full_df,
    prefix="ganf_ablation",
    root=Path("results/ganf_ablation"),
    suffix="_val",
    reference_conditions={"ETTh1": "baseline", "ETTm1": "baseline"},
)

ExperimentEngine.run calls aggregate automatically at the end of Phase 2. Call it manually only when post-processing results outside an engine run.

Visualising Results#

Use twiga.core.plot.evaluation to plot aggregated results. See Visualization for the full plotting API.

from pathlib import Path
from twiga.core.plot import load_ablation, plot_summary_heatmap, save_ablation_plots

result = load_ablation(
    root=Path("results/ganf_ablation"),
    prefix="ganf_ablation",
    suffix="_val",
)

p = plot_summary_heatmap(result.summary, metric="mae", reference="baseline")
p

save_ablation_plots(result, output_dir=Path("figures/ganf_ablation"), reference="baseline")

API Reference#

class twiga.experiment.ExperimentEngine(spec)#

Bases: object

Runs a ExperimentSpec end to end.

Usage:

engine = ExperimentEngine(SPEC)
engine.cli_main(base_cfg=PipelineConfig(...))

Or programmatically:

summary = engine.run(base_cfg, groups=["gating"], dataset_keys=["MLVS-PT"])

cli_main(base_cfg, argv=None)#

Parse CLI args then call run().

Recognised flags: --group, --dataset, --skip-hpo, --hpo-cache-policy, --accept-partial, --tracking-uri, --epochs, --num-trials, --folds.

Parameters:

base_cfg (PipelineConfig) – Root pipeline config (output_dir, epochs, etc.).
argv (list[str] | None) – Argument list; defaults to sys.argv[1:].

Return type:

None

run(base_cfg, groups=None, dataset_keys=None, skip_hpo=False, tracking_uri=None, accept_partial=False, hpo_cache_policy='reuse')#

Run all conditions × datasets and return a cross-condition summary.

Parameters:

base_cfg (PipelineConfig) – Root PipelineConfig. Dataset-specific keys are applied on top via dataclasses.replace.
groups (list[str] | None) – Condition groups to run. None runs all groups.
dataset_keys (list[str] | None) – Dataset keys from spec.datasets. None runs all datasets.
skip_hpo (bool) – Skip Phase 1 backbone HPO (reuse saved params).
tracking_uri (str | None) – MLflow tracking URI. Falls back to the MLFLOW_TRACKING_URI / TWIGA_MLFLOW_TRACKING_URI env vars. Pass None to disable tracking entirely.
accept_partial (bool) – Promote a run with both successful and failed conditions to latest. All-failed runs are never promoted.
hpo_cache_policy (Literal['reuse', 'refresh', 'error-if-missing']) – reuse matching cached params, refresh by always tuning, or error-if-missing to require a match.

Return type:

DataFrame

Returns:

Summary DataFrame with mean ± std per condition.

class twiga.experiment.ExperimentSpec(name, output_prefix, condition_cls, backbone_cls, conditions, datasets, controlled_fields=<factory>, fixed_overrides=<factory>, cv_split_freq='months', cv_train_size=12, cv_test_size=4, cv_val_size=2, cv_calib_size=0, cv_stride=1, cv_folds=10, cv_window='expanding', cv_train_size_mode='derived', hpo_train_months=14, hpo_val_months=2, hpo_enabled=True, hemisphere='NH', reference_conditions=<factory>, plot_figures=True, save_condition_plots=True, plot_split_scheme=True, sample_plot_steps=336, save_predictions=True)#

Bases: object

Full declaration of a twiga ablation / benchmark experiment.

Pass an instance to ExperimentEngine to run the experiment.

Variables:

name – Human-readable experiment title (used in logs and plot titles).
output_prefix – Prefix for all CSV output files (e.g. "mlgaf_ablation" → mlgaf_ablation_summary.csv).
condition_cls – Model config class instantiated per condition (e.g. MLPGAFConfig).
backbone_cls – Model config class used for Phase 1 backbone HPO. Must be specified explicitly — typically the plain backbone without a probabilistic head (e.g. MLPGAMConfig for a CRC experiment).
conditions – List of Condition objects defining the experimental grid.
datasets – Registry of datasets. Keys are short names used with --dataset; values are dicts of PipelineConfig field overrides (dataset_name, train_start, window_stride, …).
hemisphere – Meteorological hemisphere used when annotating fold seasons in the summary. "NH" (default) uses Northern-Hemisphere conventions (Dec–Feb = Winter). Use "SH" for Southern Hemisphere sites where seasons are reversed.

CV protocol (all fields default to the standard 10-fold expanding window):

cv_split_freq: Time unit for cv_train_size/cv_test_size/cv_stride: (one of ExperimentConfig’s split_freq values – "days", "hours", "weeks", "months", "years"). Defaults to "months", matching every existing spec’s CV protocol. Set to "weeks" for a dataset whose record is too short for a months-based protocol to give enough folds (e.g. an ~11-12 month OpenDSS synthetic record) – this was previously hardcoded to "months" inside ExperimentEngine, forcing such datasets to be driven by a hand-rolled CV loop outside the engine entirely; it is now a per-spec choice instead.

cv_train_size: Initial training window in cv_split_freq units. cv_test_size: Test window per fold in cv_split_freq units. cv_val_size: Validation window carved from the training tail. cv_calib_size: Calibration window for conformal experiments (0 =

disabled).

cv_stride: Advance between folds. cv_folds: Maximum number of folds. cv_window: "expanding" (default) grows the training window with

each fold, so later folds differ from earlier ones in both season and sample size. "rolling" keeps a fixed-width training span that slides forward, so every fold is fitted on the same amount of data and a difference between folds is attributable to season alone, not training-set size. Passed straight through to TimeBasedCV.

Output:

fixed_overrides: Applied to every model config before backbone params: (e.g. {"use_revin": False, "value_embed_type": "ConvEmb"}).
controlled_fields: Stripped from backbone HPO params so ablation: overrides always win.
reference_conditions: Maps group name → reference condition name for: Δ-vs-reference columns in the summary.

plot_figures: Whether to call save_ablation_plots after the run. plot_split_scheme: Whether to render and save a CV split-scheme sanity

plot (train/val/calib/test windows per fold, via plot_split_scheme()) for each dataset before any condition runs. Catches a misconfigured protocol (wrong fold count, stride, or window size) before compute is spent, rather than after.

save_predictions: Whether to persist each condition’s full per-fold: metrics and predictions to Parquet, at <run_root>/conditions/<dataset>/<group>/<condition>/{metrics,predictions}.parquet. On by default so the raw record (mechanism diagnostics, custom re-scoring) is always available after a run, not only the aggregated summary CSVs; set to False for a very large sweep where the extra Parquet writes are not needed.

backbone_cls: type#

condition_cls: type#

conditions: list[Condition]#

controlled_fields: frozenset#

cv_calib_size: int = 0#

cv_folds: int = 10#

cv_split_freq: Literal['days', 'hours', 'weeks', 'months', 'years'] = 'months'#

cv_stride: int = 1#

cv_test_size: int = 4#

cv_train_size: int = 12#

cv_train_size_mode: Literal['derived', 'fixed'] = 'derived'#

cv_val_size: int = 2#

cv_window: Literal['expanding', 'rolling'] = 'expanding'#

datasets: dict[str, dict]#

fixed_overrides: dict#

hemisphere: Literal['NH', 'SH'] = 'NH'#

hpo_enabled: bool = True#

hpo_train_months: int = 14#

hpo_val_months: int = 2#

name: str#

output_prefix: str#

plot_figures: bool = True#

plot_split_scheme: bool = True#

reference_conditions: dict[str, str]#

sample_plot_steps: int = 336#

save_condition_plots: bool = True#

save_predictions: bool = True#

validate(groups=None)#

Raise one ValueError listing invalid experiment settings.

Pass the groups selected for a run to validate only their executable conditions. This permits a spec to contain optional calibrated groups that a particular dataset/run intentionally excludes. Call this again after mutating the spec’s condition or dataset collections.

Return type:: None

class twiga.experiment.Condition(name, group, description='', overrides=<factory>, data_overrides=<factory>, model_cls=None, hpo_variant='', hpo_fixed_params=<factory>, hpo_model_cls=None, hpo_parent_model_cls=None, hpo_parent_variant='', hpo_objective_metric='winkler_score', hpo_num_trials=None, hpo_initial_params=None, hpo_enabled=None, hpo_search_space=None, hpo_sampler='tpe', hpo_stop_patience=None, hpo_validation_windows=1, hpo_select_on_cwe=False, metric_types=<factory>, conformal_config=None, stage1_epochs_frac=None, calib_source='train_tail', cv_split_freq=None, cv_train_size=None, cv_test_size=None, cv_val_size=None, cv_calib_size=None, cv_stride=None, cv_window=None, cv_train_size_mode=None, interval_alpha=None, checkpoint_monitor=None, result_transform=None, custom_scorer=None, reuse_checkpoint_from=None, reserve_calib_window=False, save_raw_predictions=False, reuse_checkpoint_dir=None, keep_checkpoints=False)#

Bases: object

One experimental condition — what varies between backtesting runs.

Variables:

name – Short identifier used in filenames and summaries.
group – Experiment group this condition belongs to (e.g. "gating").
description – Human-readable note, shown in logs.
overrides – Key–value pairs applied to the model config after backbone HPO params and fixed overrides. These always win.
data_overrides – Key–value pairs applied directly to this condition’s own DataPipelineConfig (via setattr, same mechanism as overrides for the model config), after build_data_config() builds it and before the forecaster is constructed. overrides cannot reach DataPipelineConfig fields (e.g. target_differencing) since it is only ever applied to the model config object – a separate instance, built fresh per condition in ExperimentEngine._backtest_condition (not shared across conditions the way a naive reading of “one dataset, one data_config” might suggest), so varying this per condition (e.g. one condition differencing a target, a reference condition not) is safe and does not affect any other condition.
model_cls – Override the spec’s condition_cls for this condition. Use for multi-model experiments (e.g. MLPF vs MLPGAM vs MLPGAF).
metric_types – Which evaluation methods to call per fold. Each entry maps to one call: "point" → evaluate_point_forecast; "interval" → evaluate_interval_forecast; "quantile" → evaluate_quantile_forecast. Ignored when custom_scorer is set. Defaults to ["point"].
conformal_config – When set the forecaster is given these conformal params and calib_size from the spec is used for calibration within each backtesting fold.
custom_scorer – Escape hatch for a per-fold evaluation the four built-in metric_types methods cannot express — e.g. sampling a Monte-Carlo ensemble from a calibrated region and scoring a downstream decision metric (Brier score) on it, rather than a metric computable directly from predict/predict_interval. Called once per fold as custom_scorer(forecaster, test_df) (the already fit, and calibrated when conformal_config is set, forecaster for this fold), and must return the same (pred_df, metric_df) shape the built-in evaluators do, so downstream tagging and predictions.parquet/metrics.parquet saving need no changes. Takes priority over metric_types when both are set.
stage1_epochs_frac – See ExperimentEngine._build_model_config.
calib_source – Where TimeBasedCV draws the calibration window from for this condition.
hpo_fixed_params – Field–value pairs pinned inside the Optuna search itself (before any trial is sampled), rather than layered on top of best_params afterward the way overrides is. Use this when a fixed field controls which other fields get sampled at all (e.g. BaseFPQRConfig.get_optuna_params’s crossing-group conditionals, active only when qvn_type in ("standard", "attention")) — pinning post-hoc via overrides alone would leave those conditional fields at an untuned class default whenever Optuna’s own free draw of the pinned field landed on a different branch. Merged across every condition sharing the same (model_cls, hpo_variant) HPO cache key, so all conditions on one backbone must agree on the same fixed values.
hpo_parent_model_cls – Optional parent HPO config class whose cached parameters are applied before this condition’s own HPO cache. This supports a shared-backbone search followed by a focused condition-specific refinement.
hpo_parent_variant – Variant suffix used for the parent HPO cache key.
hpo_objective_metric – HPO objective for quantile-producing backbones (run_backbone_hpo’s quantile_objective_metric; ignored for non-quantile backbones, which are always point-accuracy). "winkler_score" (default) or any name get_quantile_metrics() computes (e.g. the diagnostic "cwe_stratified"), all derived from the outermost predicted quantile pair with no calibration needed. Shared per (model_cls, hpo_variant) HPO cache key the same way hpo_fixed_params is – the last condition sharing a key wins if they disagree, so keep them consistent.
hpo_num_trials – Optuna trial budget for this condition’s backbone HPO cache key, overriding PipelineConfig.num_trials for that key only. Use it to give backbones with different search-space sizes a comparable budget per searched parameter. Shared per (model_cls, hpo_variant) key: the largest value among the conditions sharing a key is used. The trial count is not part of the HPO cache fingerprint, so changing it requires hpo_cache_policy="refresh" to take effect on an existing cache.
hpo_initial_params – Optional fully-suggested parameter dict (raw Optuna trial.params keys/prefixes, e.g. a prior study’s own best trial) enqueued as the first trial of this condition’s HPO search via study.enqueue_trial(..., skip_if_exists=True) – a seed/starting point, not a pin: every key is still genuinely sampled by Optuna for that one trial (so later trials are free to move away from it), and any field present in the current search space but absent from this dict is sampled normally for the seeded trial too. Merged per (model_cls, hpo_variant) HPO cache key the same way hpo_fixed_params is (last condition sharing a key wins on a key collision). Not part of the HPO cache fingerprint, same as hpo_num_trials – a cached best_params.json match short-circuits the search entirely regardless of this field, so use hpo_cache_policy="refresh" to actually re-run the search and enqueue the seed.
hpo_enabled – Whether this condition is tuned and loads its tuned parameters. None (default) follows ExperimentSpec.hpo_enabled; True/False overrides it for this condition alone, so a fixed-parameter study can tune one condition beside untuned ones.
hpo_search_space – Per-field replacements for the HPO class’s own search_space (range tuple or categorical list; None removes a field), for this condition’s search only. Merged per (model_cls, hpo_variant) key like hpo_fixed_params.
hpo_sampler – "tpe" (default) or "gp" – Optuna’s Gaussian-process sampler, for a small, mostly continuous search space.
hpo_stop_patience – Stop this condition’s search once this many completed trials in a row fail to improve the best value; hpo_num_trials stays the ceiling. None runs every trial.
hpo_validation_windows – Validation windows per trial. K > 1 scores each trial as the mean over K consecutive chunks of the HPO validation span (expanding training origin) and prunes with a WilcoxonPruner across them in place of per-epoch Hyperband; each trial trains K times, and pruning needs K >= 4. These three settings follow the last condition sharing an HPO key, like hpo_objective_metric.
hpo_select_on_cwe – Among trials within 1% of the best objective, keep the one with the highest validation stratified CWE (quantile heads). The objective is still optimised; CWE only breaks near-ties.
reuse_checkpoint_from – Name of another condition in the same ExperimentSpec whose per-fold trained weights this condition loads instead of calling forecaster.fit() – e.g. a calibration-only condition that otherwise differs from an already-trained condition only by conformal_config. The named condition must (a) have already run, in the same ExperimentEngine.run() call, in a group ordered before this condition’s own group (--group lists groups in run order), and (b) use the identical CV layout (cv_train_size/cv_test_size/ cv_val_size/cv_calib_size/cv_stride/cv_window, falling back to the same ExperimentSpec defaults), the same effective calibration-window reservation (see reserve_calib_window), and the same data_overrides/ model_cls – otherwise fold boundaries or the model architecture itself disagree and a loaded checkpoint is invalid for this condition’s own folds. ExperimentEngine validates the CV-layout and calibration-reservation match at run time and raises if either disagrees; it cannot detect a data_overrides/ architecture mismatch (that surfaces as a RuntimeError from load_state_dict instead). Both this condition and the one it names automatically get per-fold checkpoint persistence (the source’s checkpoints are kept, not deleted by cleanup_training_artifacts, specifically because some other condition reuses them) – no separate flag needed. None (default) trains normally.
reserve_calib_window – When True, this condition’s CV split reserves a calibration window (cv_calib_size months/weeks/etc, same as a condition with conformal_config set) even though conformal_config is None and the forecaster is never actually calibrated. Exists purely so an uncalibrated condition can match a sibling calibrated condition’s CV layout exactly – e.g. a plain benchmark condition that a conformal variant of it wants to reuse a checkpoint from via reuse_checkpoint_from (identical fold boundaries are required for that, see its own docstring) without itself paying for calibration at evaluation time or losing the apples-to-apples native-interval comparison the benchmark condition exists for. Has no effect when conformal_config is already set (that already reserves the window on its own). Default False preserves the existing behaviour: no calibration window unless conformal_config is set.
save_raw_predictions – When True, each fold also saves the model’s raw forecast arrays in original units – loc, scale (for a parametric head), quantiles and quantile_levels (for a quantile head), the observations and the per-step timestamps – for the validation window, the calibration window (when one is reserved) and the test window, as raw/fold_XX_{val,calib,test}.npz in the condition’s output directory. Any conformal method can then be calibrated and scored afterwards from these arrays without retraining. Routes the condition through the per-fold loop (as custom_scorer does).
keep_checkpoints – Keep this condition’s per-fold checkpoints after the run (checkpoints/<condition>/<model_type>/fold_XX) even when no other condition reuses them, so a later run can load them through reuse_checkpoint_dir instead of training again.
reuse_checkpoint_dir – Directory of an earlier run’s per-fold checkpoints (.../checkpoints/<condition>/<model_type>, holding fold_XX/) to load instead of training, e.g. to export raw forecasts from weights a previous run kept. Unlike reuse_checkpoint_from it is not validated against a source condition: the folds and architecture must match that run’s.

calib_source: str = 'train_tail'#

checkpoint_monitor: str | None = None#

conformal_config: ConformalConfig | None = None#

custom_scorer: Callable[[TwigaForecaster, DataFrame], tuple[DataFrame, DataFrame]] | None = None#

cv_calib_size: int | None = None#

cv_split_freq: Literal['days', 'hours', 'weeks', 'months', 'years'] | None = None#

cv_stride: int | None = None#

cv_test_size: int | None = None#

cv_train_size: int | None = None#

cv_train_size_mode: Literal['derived', 'fixed'] | None = None#

cv_val_size: int | None = None#

cv_window: Literal['expanding', 'rolling'] | None = None#

data_overrides: dict#

description: str = ''#

group: str#

hpo_enabled: bool | None = None#

hpo_fixed_params: dict#

hpo_initial_params: dict | None = None#

hpo_model_cls: type | None = None#

hpo_num_trials: int | None = None#

hpo_objective_metric: str = 'winkler_score'#

hpo_parent_model_cls: type | None = None#

hpo_parent_variant: str = ''#

hpo_sampler: Literal['tpe', 'gp'] = 'tpe'#

hpo_search_space: dict | None = None#

hpo_select_on_cwe: bool = False#

hpo_stop_patience: int | None = None#

hpo_validation_windows: int = 1#

hpo_variant: str = ''#

interval_alpha: float | None = None#

keep_checkpoints: bool = False#

metric_types: list[str]#

model_cls: type | None = None#

name: str#

overrides: dict#

reserve_calib_window: bool = False#

result_transform: Callable[[DataFrame, DataFrame], tuple[DataFrame, DataFrame]] | None = None#

reuse_checkpoint_dir: str | None = None#

reuse_checkpoint_from: str | None = None#

save_raw_predictions: bool = False#

stage1_epochs_frac: float | None = None#

twiga.experiment.run_backbone_hpo(backbone_cls, cfg, data, target_series, calendar_variables, exogenous_features, lags, latitude, longitude, dataset_key, hpo_cache_dir, hpo_variant='', lookback_window_size=None, hpo_train_months=14, hpo_val_months=2, fixed_search_params=None, data_overrides=None, cache_policy='reuse', quantile_objective_metric='winkler_score', initial_params=None, search_space_overrides=None, sampler='tpe', stop_patience=None, validation_windows=1, select_on_cwe=False)#

Run Optuna HPO for backbone_cls on a fixed train/val split.

Saves best params via backbone_params_path() (dataset_key slugified there for filesystem safety) and returns the param dict. The file is shared across runs and reused only when its fingerprint matches.

The Optuna objective is a point-accuracy metric (cfg.metrics[0], typically MAE) for most backbones, EXCEPT quantile-producing configs (FPQR/QR, _QUANTILE_CONFIG_BASES), where it is Winkler score instead – point accuracy alone cannot distinguish a well-calibrated quantile grid from a degenerate/collapsed one at equal median accuracy, and Winkler (unlike plain pinball loss) penalises both interval width and coverage misses jointly.

Parameters:

backbone_cls (type) – Model config class with a from_data_config factory.
cfg (PipelineConfig) – Pipeline config providing train_start, epochs, num_trials.
data (DataFrame) – Full dataset DataFrame (must have a timestamp column).
target_series (str) – Target variable name.
calendar_variables (list) – Calendar feature names.
exogenous_features (list) – Exogenous feature names.
lags (list) – Lag indices.
latitude (float) – Site latitude (used by some feature builders).
longitude (float) – Site longitude.
dataset_key (str) – Short dataset identifier used in the cache path.
hpo_cache_dir (Path) – Root directory for cached HPO params.
hpo_variant (str) – Optional suffix appended to the model name when computing the cache key (e.g. "3group" → mlpgaf_3group_best_params.json). Allows a single config class to have per-variant HPO files.
lookback_window_size (int | None) – ACF-derived recommendation from load_and_preprocess()’s eighth return value, passed through to build_data_config(). None (the default) keeps cfg.input_window_size exactly as before.
hpo_train_months (int) – Width of the HPO training span, in months from cfg.train_start. Default 14 (unchanged from the original hardcoded value) assumes at least 16 months of history; shrink for a shorter record (e.g. the ~11-12 month OpenDSS synthetic records) via hpo_train_months.
hpo_val_months (int) – Width of the HPO validation span immediately after the training span. Default 2 (unchanged).
fixed_search_params (dict | None) – Field–value pairs pinned on model_config before Optuna samples anything, and stripped from its own search_space so they are never resampled. Unlike a post-hoc override applied to best_params after HPO finishes, this keeps any other field whose sampling is conditional on the pinned one (e.g. BaseFPQRConfig’s crossing-group params, only sampled when qvn_type in ("standard", "attention")) correctly explored under the branch that will actually be used, rather than left at an untuned class default whenever a free Optuna draw of the pinned field happened to land on a different branch.
data_overrides (dict | None) – Field–value pairs applied to data_config (the DataPipelineConfig built from target_series etc. above) before the model config is derived from it via backbone_cls.from_data_config. For a field like target_differencing that changes what the target column actually is (e.g. a target’s first difference instead of its raw level), HPO must tune against the same transformed target the final backtest condition trains against, not the raw one – applying it only after HPO (the way overrides applies to a model config) would tune hyperparameters for a different problem than the one they are then used to solve. Included in _hpo_cache_fingerprint’s own data_config field already, so a condition-specific data_overrides value automatically gets its own cache entry rather than colliding with an unrelated condition sharing the same (backbone_cls, hpo_variant) key.
cache_policy (Literal['reuse', 'refresh', 'error-if-missing']) – "reuse" accepts only a matching cache and tunes on a miss; "refresh" always tunes; "error-if-missing" requires a matching cache and never starts Optuna.
quantile_objective_metric (str) – HPO objective for quantile-producing backbones only (_QUANTILE_CONFIG_BASES; ignored otherwise). Any name get_quantile_metrics computes – "winkler_score" (default), "pinball", or a diagnostic like "cwe_stratified" – all derived from the outermost predicted quantile pair, no calibration needed.
initial_params (dict | None) – Optional fully-suggested parameter dict (raw Optuna trial.params keys/prefixes, e.g. a prior study’s own best trial) enqueued as this study’s first trial via study.enqueue_trial(initial_params, skip_if_exists=True) – a seed, not a pin: this trial’s parameters are still genuinely sampled (any key missing from this dict is sampled normally for it), and later trials remain completely free. Not part of the HPO cache fingerprint, so a matching cached best_params.json still short-circuits the search before this is ever used – pass cache_policy="refresh" to force a real search that actually enqueues the seed.
search_space_overrides (dict | None) – Per-field replacements for the config’s own search_space (a range tuple or categorical list; None removes the field), applied before fixed_search_params are stripped. Lets one condition search a different range than the class default without changing it for every other experiment. Part of the cache fingerprint, through the resulting search space.
sampler (Literal['tpe', 'gp']) – "tpe" (default, multivariate TPE) or "gp" (Gaussian process; suits a small, mostly continuous search space).
stop_patience (int | None) – Stop the search once this many completed trials in a row fail to improve the best value; cfg.num_trials stays the ceiling. None runs every trial.
validation_windows (int) – Number of validation windows. 1 (default) scores each trial on the single HPO split. K > 1 cuts the validation span into K consecutive chunks; window k trains on everything before chunk k and scores on it, the trial’s objective is the mean over windows, and a WilcoxonPruner stops a trial that is significantly worse than the best one. Each trial then trains K times, and the signed-rank test cannot reach p < 0.1 before four windows, so K < 4 averages without pruning.
select_on_cwe (bool) – Keep the trial with the highest validation stratified CWE among those within 1% of the best objective (quantile heads only, as only they report CWE; other heads fall back to the usual choice). Non-default sampler/stop_patience/validation_windows/ select_on_cwe enter the cache fingerprint.

Return type:

dict

Returns:

Best hyperparameter dict (same format as load_backbone_params()). PED-ANOVA importances of the tuned parameters are written beside it as <model>_importance.json.

Raises:

ValueError – If the record is too short for the requested hpo_train_months + hpo_val_months – raised here, before forecaster.tune()’s own “‘val_df’ is empty” error, so the message names the actual cause (an HPO split too wide for this record) rather than an empty-dataframe symptom several calls downstream.

twiga.experiment.load_backbone_params(dataset_key, hpo_cache_dir, model_name_str, controlled_fields, fallback_paths=None)#

Load saved backbone HPO params, strip controlled fields and model prefix.

Searches fallback_paths first (in order), then the canonical engine path. Returns an empty dict — with a warning — when no file is found.

Parameters:

dataset_key (str) – Short dataset identifier (e.g. "MLVS-PT").
hpo_cache_dir (Path) – Root directory for cached HPO params (typically <experiment_root>/backbone_hpo).
model_name_str (str) – Model name string (e.g. "mlpgaf").
controlled_fields (frozenset) – Keys to strip from the loaded params so ablation condition overrides always take precedence.
fallback_paths (list[Path] | None) – Additional JSON files to try before the canonical path.

Return type:

dict

Returns:

Dict of hyperparameter names → values, ready to setattr onto a model config object.

twiga.experiment.aggregate(combined, prefix, root, suffix, reference_conditions)#

Compute mean ± std per (group, condition, metric_type) and save CSVs.

Parameters:

combined (DataFrame) – Long-form DataFrame with one row per fold/horizon, tagged with group, condition, dataset, and optionally metric_type columns.
prefix (str) – Filename prefix for output CSVs.
root (Path) – Directory to write <prefix>_full<suffix>.csv and <prefix>_summary<suffix>.csv.
suffix (str) – Optional tag appended to filenames (e.g. "_val").
reference_conditions (dict[str, str]) – Maps group → reference condition name for Δ-vs-reference columns.

Return type:

DataFrame

Returns:

Summary DataFrame with MultiIndex (group, condition[, metric_type]) and one column per metric, plus _std variants, n_runs, and Δ columns. Empty DataFrame if no recognised metric columns are present.