Skip to main content

ST-005: Accumulator / Optimizer-State Precision

Status:Preliminary
Tags:Ablation TestOptimizerAdam
Last modified:

Purpose

This stress test applies rail and floor analysis to training internals that are not part of the forward model but still determine the update.

Mechanism​

For Adam, vulnerable objects include:

mt,vt,vt+Ο΅,mtvt+Ο΅.m_t, \qquad v_t, \qquad \sqrt{v_t}+\epsilon, \qquad \frac{m_t}{\sqrt{v_t}+\epsilon}.

For batch training, vulnerable objects include:

βˆ‘b=1BGb,1Bβˆ‘b=1BGb.\sum_{b=1}^{B}G_b, \qquad \frac{1}{B}\sum_{b=1}^{B}G_b.

How This Differs From Rail/Floor Tests​

The failure mechanism is still saturation, underflow, or quantization error. The difference is scope: the vulnerable tensors are optimizer state and accumulation registers rather than only activations, weights, gradients, and updates.

Main Plots​

  • Adam m norm,
  • Adam v norm,
  • denominator statistics,
  • raw step norm,
  • quantized step norm,
  • accumulator rail pressure,
  • final update norm.

Priority​

This test should come after the simpler fixed-point path is clear.