Part XXIII — Modern Architectures, Generative Models & LLM Engineering · Chapter 10

Experiment tracking & reproducibility

Hook

"Just rerun it with the same hyperparameters" sounds like it should be enough to reproduce a result. It isn't — because a hyperparameter file only ever records the settings someone remembered to write down, and a training run depends on more than that.

Intuition
stepgradientnoisew beforew after
12-1109
21-2910
330107

lr=1, gradients=2, 1, 3 — identical for every seed. Final weight: 7

Every seed gets the exact same learning rate, the exact same gradient schedule, the exact same starting weight. Only the seed changes — and the final weight changes with it.

Formalize

A training run's outcome is a function of everything that fed into it — not just the hyperparameters that happen to live in a config file:

wfinal=f(  lr,gradients,w0⏟usually logged, seed⏟often not  )w_{\text{final}} = f(\;\underbrace{\text{lr}, \text{gradients}, w_0}_{\text{usually logged}},\ \underbrace{\text{seed}}_{\text{often not}}\;)
  • wfinalw_{\text{final}} — the weight a training run actually ends up with.
  • ff — the training process itself, treated as a function of everything that feeds into it.
  • lr\text{lr} — the learning rate.
  • gradients\text{gradients} — the sequence of gradients computed during training.
  • w0w_0 — the starting weight, before any training steps.
  • seed\text{seed} — the random seed governing every source of stochasticity in the run.
  1. A reproducibility claim is only as strong as its inputs

    "Reproducible" means every argument to ff is pinned down, not just the ones that happen to live in a config file.

  2. Logging hyperparameters without the seed isn't reproducibility

    Log lr\text{lr}, gradients\text{gradients}, and w0w_0 but not seed\text{seed}, and "same config" silently means "one of many possible outcomes" instead of "this exact outcome."

Play

seed 1

lr = 1

gradients = [2, 1, 3]

final weight = 7

seed 2

lr = 1

gradients = [2, 1, 3]

final weight = 2

Same lr, same gradient schedule, same starting weight — printed on both cards, character for character. The only difference between the two runs is a single integer neither card would show you unless someone thought to log it.

Worked example
  1. Seed 1's noise sequence: -1, -2, 0
    1. 10−1×(2+(−1))=910 - 1 \times (2 + (-1)) = 9
    2. 9−1×(1+(−2))=109 - 1 \times (1 + (-2)) = 10
    3. 10−1×(3+0)=710 - 1 \times (3 + 0) = 7
  2. Seed 2's noise sequence: -1, 1, 2 — same lr, same gradients
    1. 10−1×(2+(−1))=910 - 1 \times (2 + (-1)) = 9 — identical so far
    2. 9−1×(1+1)=79 - 1 \times (1 + 1) = 7 — already diverged
    3. 7−1×(3+2)=27 - 1 \times (3 + 2) = 2
  3. Two final weights, one logged config

    7 versus 2. A config file listing only lr=1 and the gradient schedule describes both runs equally well — and neither one exactly.

Checkpoint

A logged experiment reports lr=1, the same gradient schedule, and a final weight of 8 — but the seed wasn't recorded. Find the seed that reproduces it.

Pick a seed to try it
Summary
wfinal=f(lr,gradients,w0,seed)w_{\text{final}} = f(\text{lr}, \text{gradients}, w_0, \text{seed})

Nothing here is exotic randomness — it's a plain linear congruential generator, the same kind of determinism a real training run's data shuffling, dropout masks, and weight initialization all rely on. The lesson isn't "randomness is bad," it's that reproducibility is only as complete as the list of things an experiment tracker actually captures. Seeds are the obvious member of that list; real pipelines also need the library versions, hardware, and data snapshot pinned down, for exactly the same reason. The next chapter turns to what happens when the data a deployed model sees drifts away from what it was trained on.