Skip to main content

ST-004: Quantization Floor / Update Dead-Zone

Status:Preliminary
Tags:Ablation TestQuantizationUnderflow
Last modified:

Purpose

This stress test targets the opposite of explosion. Learning can be numerically stable but useless because the update is too small to be represented.

Mechanism​

The dead-zone condition is:

αtη∥Gt∥≲qΔ.\alpha_t\eta\|G_t\| \lesssim q_\Delta.

Expected symptom:

Δθtfloat≠0butQΔ(Δθt)=0.\Delta\theta_t^{\mathrm{float}} \neq 0 \qquad \text{but} \qquad Q_\Delta(\Delta\theta_t)=0.

Why This Matters​

A conservative throttle can accidentally kill learning. A useful controller must avoid both sides:

too large⇒unstable\text{too large} \Rightarrow \text{unstable} too small⇒dead update.\text{too small} \Rightarrow \text{dead update}.

Main Plots​

  • update norm,
  • update underflow rate,
  • αt\alpha_t,
  • effective update quantum threshold,
  • loss plateau behavior.
Controller limitation

If the stable-useful interval is empty, no scalar throttle can save the selected precision. The dtype, learning rate, or update format must change.