Ablation Tests
This section defines the stress-test families used to evaluate fixed-point online learning. Each experiment should instantiate one family with a concrete dataset, model, precision map, optimizer, controller, and stress configuration.
The goal is not to make training fail by choosing arbitrary bad hyperparameters. The goal is to create controlled situations where:
Then we test whether the global throttle can preserve useful learning by adapting:
Experiment Traceā
Experiment pages should reference ablation tests by stable ID:
dataset: DS-...
model: MDL-...
precision: PREC-...
stress_test: ST-...
optimizer: ...
controller: CTRL-...
metrics: [...]
Stress-Test Tableā
| Stress ID | Family | Main question | Priority |
|---|---|---|---|
ST-000 | High learning-rate sanity | Can the controller stabilize a deliberately unstable loop? | Low |
ST-001 | Distribution / operating-point shift | Does a fixed become unsafe after deployment drift? | High |
ST-002 | Transient spike / corrupted batch | Can one abnormal batch corrupt the learning trajectory? | Medium |
ST-003 | Rail saturation / overflow | Do fixed-point tensors hit representable rails? | High |
ST-004 | Quantization floor / update dead-zone | Does learning die because updates are too small to represent? | High |
ST-005 | Accumulator / optimizer-state precision | Do Adam state or gradient accumulators fail even when weights look safe? | Medium |
ST-006 | Naive safety mechanisms | Do clipping methods stabilize numerics while distorting learning? | Medium |
ST-007 | Legacy projection mismatch | Does old row/column projection preserve rails by distorting the update trajectory? | Low |
Quantization And Throttling Are Coupledā
The precision choice defines the representable range and resolution. The throttle controls the training trajectory inside that representable envelope.
For fixed-point updates, stable and useful learning requires:
The left side is a usefulness constraint: if is too small, the update falls below the fixed-point quantum and learning silently dies. The right side is a stability constraint: if is too large, the update may overshoot or saturate.
The global throttle cannot make an impossible fixed-point design possible. Good dtype selection and input capping are still part of a robust hardware design. The controller should be evaluated inside a plausible representational envelope, not as a replacement for one.
Shared Metricsā
Every ablation should log enough information to identify the failure mechanism.
| Metric | Why it matters |
|---|---|
| loss | task-level stability |
| output error | task-level recovery |
| weight growth | |
| gradient growth | |
| update magnitude | |
| controller action | |
| effective learning rate | |
| online curvature / sensitivity proxy | |
| effective stability margin | |
| rail pressure | saturation risk |
| update underflow rate | dead-zone risk |
| update cosine / phase | offline geometry diagnostic |
For tiny models, also log:
Update cosine and phase are useful offline diagnostics, especially when comparing against clipping or projection. They should not be assumed to be practical hardware controller signals.
Success And Failure Criteriaā
A throttled run is successful only if it remains stable and keeps learning:
A controller that prevents divergence only by making every update vanish has not solved online training. It has converted instability into frozen adaptation.
Recommended First Waveā
| Order | Stress test | Reason |
|---|---|---|
| 1 | ST-001 | matches the deployment story |
| 2 | ST-003 | directly hardware-relevant |
| 3 | ST-004 | defines when throttle cannot help |
| 4 | ST-002 | tests runtime shock absorption |
| 5 | ST-006 | compares against simple fixes |
ST-005 and ST-007 should come after the basic fixed-point path is clear.