Skip to main content

Ablation Tests

Status:PreliminaryIn progress
Tags:Ablation RegistryGlobal Throttle
Last modified:

Purpose

This section defines the stress-test families used to evaluate fixed-point online learning. Each experiment should instantiate one family with a concrete dataset, model, precision map, optimizer, controller, and stress configuration.

The goal is not to make training fail by choosing arbitrary bad hyperparameters. The goal is to create controlled situations where:

floatĀ trainingĀ isĀ stablebuthardware-likeĀ fixed-pointĀ trainingĀ becomesĀ unsafe.\text{float training is stable} \quad \text{but} \quad \text{hardware-like fixed-point training becomes unsafe}.

Then we test whether the global throttle can preserve useful learning by adapting:

Ī·teff=αtĪ·,0<αt≤1.\eta_t^{\mathrm{eff}}=\alpha_t\eta, \qquad 0<\alpha_t\leq 1.

Experiment Trace​

Experiment pages should reference ablation tests by stable ID:

dataset: DS-...
model: MDL-...
precision: PREC-...
stress_test: ST-...
optimizer: ...
controller: CTRL-...
metrics: [...]

Stress-Test Table​

Stress IDFamilyMain questionPriority
ST-000High learning-rate sanityCan the controller stabilize a deliberately unstable loop?Low
ST-001Distribution / operating-point shiftDoes a fixed Ī·\eta become unsafe after deployment drift?High
ST-002Transient spike / corrupted batchCan one abnormal batch corrupt the learning trajectory?Medium
ST-003Rail saturation / overflowDo fixed-point tensors hit representable rails?High
ST-004Quantization floor / update dead-zoneDoes learning die because updates are too small to represent?High
ST-005Accumulator / optimizer-state precisionDo Adam state or gradient accumulators fail even when weights look safe?Medium
ST-006Naive safety mechanismsDo clipping methods stabilize numerics while distorting learning?Medium
ST-007Legacy Īŗ\kappa projection mismatchDoes old row/column projection preserve rails by distorting the update trajectory?Low

Quantization And Throttling Are Coupled​

The precision choice defines the representable range and resolution. The throttle controls the training trajectory inside that representable envelope.

For fixed-point updates, stable and useful learning requires:

qĪ”Ī·āˆ„Gt∄+ϵ≲αt≤χηCtctrl+ϵ.\boxed{ \frac{q_\Delta}{\eta\|G_t\|+\epsilon} \lesssim \alpha_t \leq \frac{\chi}{\eta C_t^{\mathrm{ctrl}}+\epsilon}. }

The left side is a usefulness constraint: if αt\alpha_t is too small, the update falls below the fixed-point quantum qĪ”q_\Delta and learning silently dies. The right side is a stability constraint: if αt\alpha_t is too large, the update may overshoot or saturate.

Throttle is not a substitute for valid precision

The global throttle cannot make an impossible fixed-point design possible. Good dtype selection and input capping are still part of a robust hardware design. The controller should be evaluated inside a plausible representational envelope, not as a replacement for one.

Shared Metrics​

Every ablation should log enough information to identify the failure mechanism.

MetricWhy it matters
losstask-level stability
output errortask-level recovery
∄θt∄\|\theta_t\|weight growth
∄Gt∄\|G_t\|gradient growth
āˆ„Ī”Īøt∄\|\Delta\theta_t\|update magnitude
αt\alpha_tcontroller action
ηteff=αtη\eta_t^{\mathrm{eff}}=\alpha_t\etaeffective learning rate
CtctrlC_t^{\mathrm{ctrl}}online curvature / sensitivity proxy
αtηCtctrl\alpha_t\eta C_t^{\mathrm{ctrl}}effective stability margin
rail pressuresaturation risk
update underflow ratedead-zone risk
update cosine / phaseoffline geometry diagnostic

For tiny models, also log:

Ī»max⁔(Ht),ρ(Iāˆ’Ī·Ht),ρ(Iāˆ’Ī±tĪ·Ht).\lambda_{\max}(H_t), \qquad \rho(I-\eta H_t), \qquad \rho(I-\alpha_t\eta H_t).
Diagnostic caution

Update cosine and phase are useful offline diagnostics, especially when comparing against clipping or Īŗ\kappa projection. They should not be assumed to be practical hardware controller signals.

Success And Failure Criteria​

A throttled run is successful only if it remains stable and keeps learning:

finite loss∧bounded rail pressure∧nonzero useful updates∧improving error.\text{finite loss} \quad \land \quad \text{bounded rail pressure} \quad \land \quad \text{nonzero useful updates} \quad \land \quad \text{improving error}.
Do not confuse survival with learning

A controller that prevents divergence only by making every update vanish has not solved online training. It has converted instability into frozen adaptation.

OrderStress testReason
1ST-001matches the deployment story
2ST-003directly hardware-relevant
3ST-004defines when throttle cannot help
4ST-002tests runtime shock absorption
5ST-006compares against simple fixes

ST-005 and ST-007 should come after the basic fixed-point path is clear.