milia.benchmark.provenance#
Provenance sidecars: which settings, package versions and input files produced an output.
Functions
|
The provenance record of an output. |
|
The provenance record of a run: creation time, package versions, settings and input hashes. |
|
SHA-256 digest of a file, so an output records exactly which inputs it came from. |
|
The sidecar file of an output: |
|
Installed version of a distribution, with |
|
Write the provenance sidecar of an output. |
- milia.benchmark.provenance.sha256(path)#
SHA-256 digest of a file, so an output records exactly which inputs it came from.
Examples
>>> import tempfile >>> from pathlib import Path >>> folder = Path(tempfile.mkdtemp()) >>> (folder / "x.txt").write_text("a") 1 >>> sha256(folder / "x.txt")[:12] 'ca978112ca1b'
- milia.benchmark.provenance.version(name)#
Installed version of a distribution, with
+git.<commit>for a VCS install.- Parameters:
name (
str) – Distribution name, e.g."twiga".- Return type:
- Returns:
The version, or
Nonewhen the distribution is not installed.
Examples
>>> version("definitely-not-installed") is None True
- milia.benchmark.provenance.record(settings, inputs=None, packages=('pundamilia', 'scikit-learn', 'numpy', 'pandas', 'twiga'))#
The provenance record of a run: creation time, package versions, settings and input hashes.
- Parameters:
- Return type:
- Returns:
The record as a dictionary.
Examples
>>> sorted(record({"seed": 1})) ['created', 'inputs', 'packages', 'settings']
- milia.benchmark.provenance.sidecar(path)#
The sidecar file of an output:
<name>.provenance.jsonbeside it.Examples
>>> from pathlib import Path >>> sidecar(Path("runs/scores.parquet")).name 'scores.parquet.provenance.json'
- milia.benchmark.provenance.write(path, settings, inputs=None)#
Write the provenance sidecar of an output.
- Parameters:
- Return type:
- Returns:
The sidecar path.
Examples
>>> import tempfile >>> from pathlib import Path >>> folder = Path(tempfile.mkdtemp()) >>> write(folder / "scores.parquet", {"seed": 1}).name 'scores.parquet.provenance.json'
- milia.benchmark.provenance.read(path)#
The provenance record of an output.
- Parameters:
path (
Path) – The output file.- Return type:
- Returns:
The record, or an empty dictionary when the output has none.
Examples
>>> import tempfile >>> from pathlib import Path >>> folder = Path(tempfile.mkdtemp()) >>> _ = write(folder / "scores.parquet", {"seed": 1}) >>> read(folder / "scores.parquet")["settings"], read(folder / "missing") ({'seed': 1}, {})