Skip to content

Tracking code provenance

When an experiment needs explaining after the fact, the question is always exactly what code produced this? The version number cannot answer it. Because private package registries reject PEP 440 local version segments, Stained Glass Core strips the commit hash from its version, so two feature branches forked from the same commit produce identical version strings. And by the time a wheel is installed — especially a bytecode wheel inside a Docker image — there is no .git directory left to inspect.

stainedglass_core.provenance answers it instead.

Quick start

from stainedglass_core import provenance

provenance.build_info()
# {'version': '3.65.0.post1.dev12',
#  'commit': 'd718a6a8d5841f16b4a0c...',   # the full 40-character SHA
#  'branch': 'feat/new-loss',
#  'dirty': True,
#  'remote_url': 'git@github.com:protopia-ai/stained-glass-core.git',
#  'build_kind': 'bytecode-wheel',
#  'content_digest': 'sha256:0afbb16caed3a760...',
#  'source': 'build-stamp',
#  ...}

This works identically for an editable feature-branch checkout, a plain wheel, a bytecode-protected wheel, and a bytecode wheel inside a Docker image.

Recording a training run

Log the short tag so a run is findable, and store the full record so it is reconstructable:

import wandb

from stainedglass_core import provenance

info = provenance.build_info()

wandb.init(
    project="my-experiment",
    tags=[f"stainedglass_core@{provenance.short_commit(info['commit'])}"],
    config={"stainedglass_core": info},
)

build_info() is a plain JSON-serializable dictionary, so it drops straight into a run config, an MLflow parameter set, or a job manifest. Format the tag to suit your tracker -- Weights and Biases, for instance, caps tags at 64 characters. The runnable examples under docs/examples/ use this same pattern.

What each layer needs installed

The build stamp, direct_url.json and metadata layers need nothing beyond the package itself. The working-tree layer reads live git state through GitPython, which is an optional extra:

pip install 'stainedglass_core[git]'

Without it, an editable checkout skips that layer and falls through to whichever one can answer. The repository's own make install-requirements installs the extra, so this only affects consumers who install stainedglass_core directly without it.

Knowing how much to trust the answer

Every record carries a source field naming the layer that produced it:

source Applies to Trust
working-tree Editable or source install Live git state, read at call time.
build-stamp Any wheel, including bytecode and Docker Recorded at build time; accurate unless the tree was dirty.
direct-url pip install git+ssh://...@branch Commit recorded by pip. No branch dirtiness information.
metadata-only Anything else Version only. Treat the code as unidentified.

If you gate on provenance quality, metadata-only is the value to alert on.

Reconstructing the exact code

build_info() carries everything needed to recover the code, but only a record that confirms all three of commit, clean tree and pushed commit can be trusted to do so:

info = provenance.build_info()

# pip fetched a `direct-url` commit from `remote_url`, which shows it was pushed there, unless that URL is a local path.
fetched_by_pip = info["source"] == "direct-url" and not (
    info["remote_url"] or ""
).startswith("file:")
pushed = info["on_remote"] is True or fetched_by_pip

if info["commit"] is None or info["dirty"] is not False or not pushed:
    raise RuntimeError(
        f"cannot confirm this build can be recreated (source={info['source']})"
    )

print(f"git clone {info['remote_url']} && git checkout {info['commit']}")

Test each field for the confirming value rather than the failing one, because None means unknown, never fine:

  • commit is None — a metadata-only record, or git could not be read. There is nothing to check out.
  • dirty is True — the build included uncommitted changes, so it never existed as a commit. None means git could not tell.
  • on_remote is False — the commit was never pushed, so nobody else can fetch it. None means nobody checked, which is always the case outside a build stamp (see below). A direct-url record needs no check, since pip fetched the commit from remote_url, except from a file: URL, which is a local path. For an editable run, push before you launch and confirm it yourself, for example with git branch --remotes --contains HEAD.

remote_url is recorded with any embedded credentials stripped, so a record built from an authenticated HTTPS checkout names the repository without carrying its access token into every wheel, image and tracked run. Cloning a private repository from that URL therefore needs your own credentials.

on_remote is recorded only by the build stamp: resolving it needs a network round-trip, which build_info() does not perform. It is None from any other source.

Push before you build

Building a wheel from an unpushed commit produces an artifact whose recorded provenance nobody can act on. make wheel and make bytecode-wheel therefore fail when the commit is absent from origin:

ERROR: Commit 0064d8b8... is not present on remote 'origin'. A build stamped with an unpushed
commit records a state nobody else can fetch, so the artifact cannot be reproduced.
Push the commit, or pass --allow-unpushed for a deliberately local build.

Push the branch and rebuild. For a genuinely throwaway local build, forward the escape hatch through make:

make wheel BUILD_INFO_FLAGS=--allow-unpushed

BUILD_INFO_FLAGS is accepted by make wheel, make bytecode-wheel and make build-info. Anything stamped that way should not reach a shared training cluster or a customer.

A commit counts as present when it is reachable from a ref that origin advertises, not merely when it is one — a pushed commit stops being a branch tip as soon as the next commit lands, and it is still fetchable. If the remote cannot be reached, or any branch tip origin advertises is not in the local object store, on_remote is recorded as None and the build only warns — neither is evidence that the commit is missing. A missing branch tip usually means someone pushed after your last fetch; run git fetch origin and rebuild to settle it.

A pull-request CI run builds the synthetic merge of the branch into its base, which no ref keeps. GitHub regenerates that commit under a new SHA whenever it rechecks mergeability, so it can disappear from origin mid-build. In that run only, if the commit has exactly two pushed parents and re-merging them with git merge-tree reproduces its tree, its code can still be recreated: on_remote is recorded as None and the build warns. The stamp records both parents in parents, so merging them recreates the code after GitHub has discarded the merge's SHA. A merge holding a hand-resolved conflict or any other local edit fails that comparison and still fails the build. Outside a pull-request run, an unpushed merge is treated like any other unpushed commit, however cleanly it re-merges: push it before you build.

Confirming two deployments match

content_digest() folds the per-file SHA-256 entries from the installed wheel's RECORD into a single digest. It identifies the code even when git state is missing or misleading, which makes it the way to confirm that a training job and an inference proxy were installed from the same wheel:

from stainedglass_core import provenance

training_digest = provenance.content_digest()
# compare against the value reported by the proxy

It returns None for an editable install, whose RECORD lists only path-configuration shims. Use the working-tree git state in that case.

The digest reports what was installed, not what is on disk now: it re-reads no files, so an edit made after installation leaves it unchanged. Catching that needs a signature over the artifact — anyone able to rewrite an installed file can rewrite the RECORD beside it, so no digest the package computes about itself can answer it.

Relationship to utils.git

stainedglass_core.utils.git.local_repo_commit_wandb_tags is the older helper, and the two now have distinct jobs:

build_info() local_repo_commit_wandb_tags()
Editable checkout yes yes
Wheel, bytecode wheel, Docker yes no -- returns []
Reports on this package any number of locally checked-out repositories

Reach for build_info() unless you specifically need to describe several repositories at once -- your training repo alongside this one, say.

Both apply the same ownership check. Repository discovery searches parent directories, so without it any package inside a virtual environment created within a checkout is attributed to that checkout: asking the old helper about torch returned stained-glass-core. A discovered repository is now used only when git ls-files --error-unmatch confirms it tracks the file, so an unowned module yields [] rather than a confident wrong answer.