Skip to main content

🗂️ How this fits into κ-budgeting

Status:Draft
Tags:UnassignedNeeds revisionFormulation
Last modified:

κ\kappa-budgeting is still useful as a static safety rail ∥Wℓ∥≤κℓ\|W_\ell\|\leq \kappa_\ell, since it bounds:

  • fixed-point dynamic range,
  • layer gain,
  • representational Lipschitzness,
  • overflow risk,
  • static hardware safety.

On the other hand, the throttle controls:

θt+1=θt−αtηGt.\theta_{t+1} = \theta_t-\alpha_t\eta G_t.

It bounds:

  • learning speed,
  • closed-loop adaptation stability,
  • overshoot under drift,
  • optimizer stiffness.

So:

κ-budgeting controls the admissible region.\boxed{ \text{$\kappa$-budgeting controls the admissible region.} } global throttling controls the trajectory inside that region.\boxed{ \text{global throttling controls the trajectory inside that region.} }

A very clean combined formulation is:

θt+1ctrl=θt−αtηGt.\theta_{t+1}^{\text{ctrl}} = \theta_t-\alpha_t\eta G_t.

Then:

θt+1=Πκ(θt+1ctrl).\theta_{t+1} = \Pi_{\kappa} \left( \theta_{t+1}^{\text{ctrl}} \right).

But the key is that Πκ\Pi_\kappa should be a loose guard rail, not the primary stabilizer applied aggressively every step.

Key insight

The throttle should prevent the system from hitting the κ\kappa rails too violently. The κ\kappa rails are still there as final protection. So if we want, we can actually disable the κ\kappa projection to spare hardware and computation and keep the global throttle as the main stabilizer.