Skip to main content

🎚️ The global throttle mechanism

Status:PreliminaryIn progress
Tags:UnassignedFormulationGlobal ThrottleNeeds revision
Last modified:

Motivation​

Instead of projecting each row/column independently (which as we saw in the kappa-budgeting section introduces rotation in the θ\theta plane), we introduce a single global scalar:

0<αt≤1.0<\alpha_t\leq 1.

Then the SGD update becomes:

θt+1=θt−αtηGtwithGt=∇θL(θt).\boxed{ \theta_{t+1} = \theta_t-\alpha_t\eta G_t } \qquad \text{with} \quad G_t=\nabla_\theta\mathcal{L}(\theta_t).

This is important because the update remains parallel to the original gradient update.

Raw update:

Δθtraw=−ηGt.\Delta\theta_t^{\text{raw}} = -\eta G_t.

Controlled update:

Δθtctrl=−αtηGt.\Delta\theta_t^{\text{ctrl}} = -\alpha_t\eta G_t.

Therefore:

Δθtctrl=αtΔθtraw.\Delta\theta_t^{\text{ctrl}} = \alpha_t\Delta\theta_t^{\text{raw}}.

Now, to verify whether direction is preserved or not, we can use the cosine similarity:

cos⁡(Δθtctrl,Δθtraw)=Δθtctrl Δθtraw∥Δθtctrl∥∥Δθtraw∥.\cos \left(\Delta\theta_t^{\text{ctrl}}, \Delta\theta_t^{\text{raw}} \right) = \frac{\Delta\theta_t^{\text{ctrl}}\,\Delta\theta_t^{\text{raw}}}{ \|\Delta\theta_t^{\text{ctrl}}\| \|\Delta\theta_t^{\text{raw}}\| }.

As mentioned before, we want this to be 11, which means the scaling used to ensure stability (on every layer) do not change the direction of the update. Now, if we replace the definition for our global throttle, we get:

cos⁡(Δθtctrl,Δθtraw)=αtΔθtraw Δθtraw∥αtΔθtraw∥∥Δθtraw∥=αt∥Δθtraw∥2αt∥Δθtraw∥∥Δθtraw∥=1.\cos \left(\Delta\theta_t^{\text{ctrl}}, \Delta\theta_t^{\text{raw}} \right) = \frac{\alpha_t\Delta\theta_t^{\text{raw}}\,\Delta\theta_t^{\text{raw}}}{ \|\alpha_t\Delta\theta_t^{\text{raw}}\| \|\Delta\theta_t^{\text{raw}}\| } = \frac{\alpha_t\|\Delta\theta_t^{\text{raw}}\|^2}{ \alpha_t\|\Delta\theta_t^{\text{raw}}\| \|\Delta\theta_t^{\text{raw}}\| } = 1.
Key insight

That means global throttling preserves the direction of learning. It only changes the speed.

row/column κ projection⇒may rotate the update\text{row/column $\kappa$ projection} \Rightarrow \text{may rotate the update} global throttle⇒preserves the update direction\text{global throttle} \Rightarrow \text{preserves the update direction}

Estimation of local curvature/sensitivity​

In section overview we showed that convergence of SGD is guaranteed if the learning rate η\eta satisfies:

η<2λmax⁡(H),\eta<\frac{2}{\lambda_{\max}(H)},

where HH is the Hessian of the loss. Therefore, to adapt αt\alpha_t to ensure stability, we need an estimate of λmax⁡(H)\lambda_{\max}(H). However, as also introduced in the overview section, finding the actual hessian is computationally expensive and sometimes even not feasible. Therefore, we need a cheap online estimate of the local curvature/sensitivity of the loss landscape.

Let's recall the formulas for the gradient and the Hessian.

G(θ)=∇θL(θ)H(θ)=∇θG(θ)=∇θ2L(θ)\begin{aligned} G(\theta) &= \nabla_\theta\mathcal{L}(\theta) \\ H(\theta) &= \nabla_\theta G(\theta) &= \nabla_\theta^2\mathcal{L}(\theta) \end{aligned}

Now, we can approximate the hessian using Taylor expansion around some operating point. For a small change in θ\theta, we have:

Δθt=θt−θt−1,\Delta\theta_t = \theta_t-\theta_{t-1},

Taylor expansion gives:

Gt−Gt−1≈Ht(θt−θt−1).G_t-G_{t-1} \approx H_t(\theta_t-\theta_{t-1}).

So:

ΔGt≈HtΔθt.\Delta G_t \approx H_t\Delta\theta_t.

Taking norms:

∥ΔGt∥≈∥HtΔθt∥.\|\Delta G_t\| \approx \|H_t\Delta\theta_t\|.

Divide by:

∥Δθt∥.\|\Delta\theta_t\|.

Then:

∥ΔGt∥∥Δθt∥≈∥HtΔθt∥∥Δθt∥.\frac{ \|\Delta G_t\| }{ \|\Delta\theta_t\| } \approx \frac{ \|H_t\Delta\theta_t\| }{ \|\Delta\theta_t\| }.

This is an observed directional curvature estimate. In other words, this is a first-order approximation of the curvature along the direction the optimizer just moved.

So we define:

C^t=∥Gt−Gt−1∥2∥θt−θt−1∥2+ϵ.\boxed{ \widehat{C}_t = \frac{ \|G_t-G_{t-1}\|_2 }{ \|\theta_t-\theta_{t-1}\|_2+\epsilon }. }

How does this relate to λmax⁡(Ht)\lambda_{\max}(H_t)?​

For a quadratic loss, we have:

L(θ)=12θ⊤Hθ,\mathcal{L}(\theta) = \frac{1}{2}\theta^\top H \theta,

then:

G(θ)=Hθ,G(\theta) = H\theta,

and:

ΔGt=HΔθt.\Delta G_t = H\Delta\theta_t.

Then:

∥ΔGt∥∥Δθt∥=∥HΔθt∥∥Δθt∥≤∥H∥2=λmax⁡(H).\frac{\|\Delta G_t\| }{\|\Delta\theta_t\| } = \frac{\|H\Delta\theta_t\| }{\|\Delta\theta_t\| } \leq \|H\|_2 = \lambda_{\max}(H).
Key insight

So as we can see, for a quadratic loss, this estimator is an upper bound on the true Hessian spectral norm. For general losses, it is a local directional curvature estimate that can be used as a proxy for the local Lipschitzness of the gradient field.

A caveat

The previous estimate is not perfect. It is very good for convex problems and things like a single-dense layer architecture, but it can (and will) be inaccurate for non-convex problems and more complex architectures. However, it is a simple and computationally cheap way to get a sense of the local curvature/sensitivity, which is what we need to adapt the learning rate and prevent divergence.

The global throttle mechanism preserves the geometry of the original gradient update, but it adaptively reduces the learning rate when the gradient field becomes stiff/sharp. This allows the model to keep learning without diverging, even under distribution shift.

What happens for nonlinear / nonconvex networks?​

For deeper networks, ReLU networks, CNNs, cross-entropy, etc., the loss is not globally quadratic.

For a general network:

y^=f(x;θ),\hat{y}=f(x;\theta),

with loss:

L(θ),\mathcal{L}(\theta),

we use a local Taylor expansion:

L(θ+δ)≈L(θ)+G(θ)⊤δ+12δ⊤H(θ)δ.\mathcal{L}(\theta+\delta) \approx \mathcal{L}(\theta) + G(\theta)^\top\delta + \frac{1}{2}\delta^\top H(\theta)\delta.

The gradient descent map is:

θt+1=θt−ηG(θt).\theta_{t+1} = \theta_t-\eta G(\theta_t).

Linearizing perturbations:

δt+1≈(I−ηHt)δt.\delta_{t+1} \approx (I-\eta H_t)\delta_t.

So the same stability idea applies locally.

However:

  • HtH_t changes over time,
  • HtH_t may be indefinite,
  • ReLU networks are piecewise smooth,
  • mini-batch gradients include sampling noise,
  • quantization introduces non-smooth perturbations.

Therefore, for general networks we should not claim C^t=λmax⁡(Ht)\widehat{C}_t=\lambda_{\max}(H_t) exactly. Instead, we claim: C^t\widehat{C}_t is an online estimate of local update-field sensitivity:

C^t≈∥HtΔθt∥∥Δθt∥.\widehat{C}_t \approx \frac{ \|H_t\Delta\theta_t\| }{ \|\Delta\theta_t\| }.

Then, for a general loss, we want to keep:

αtηLt<χ,\alpha_t\eta L_t<\chi,

where LtL_t is a local estimate of gradient-field Lipschitzness and chichi is some stability margin (normally χ≤2\chi \leq 2). Then, for instance, if the controller uses Lt≈CtctrlL_t\approx C_t^{\text{ctrl}}:

αtηCtctrl≤χ.\boxed{ \alpha_t\eta C_t^{\text{ctrl}}\leq\chi. }
Key insight

Even for nonlinear, non-convex networks, this global throttle mechanism can still be used as a stabilizer by using the local curvature/sensitivity estimate to adapt the learning rate and prevent divergence.

Dynamical system controller design​

The global throttle can be interpreted as a closed-loop controller on the effective learning rate. The controlled SGD update is

θt+1=θt−αtηGt,\theta_{t+1} = \theta_t-\alpha_t\eta G_t,

where Gt=∇θL(θt)G_t=\nabla_\theta \mathcal{L}(\theta_t). The role of the controller is to keep the effective stability margin

mt=ηαtCtctrlm_t = \eta\alpha_t C_t^{\mathrm{ctrl}}

near a chosen target χ<2\chi<2. We define the margin error

et=mt−χ=ηαtCtctrl−χ.e_t = m_t-\chi = \eta\alpha_t C_t^{\mathrm{ctrl}}-\chi.

If et>0e_t>0, the update is too aggressive and the controller should reduce αt\alpha_t. If αt<0\alpha_t<0, the update is conservative and the controller may increase αt\alpha_t.

Case 0: Algebraic safe-gain throttle​

The simplest controller directly sets αt\alpha_t to the estimated safe value:

αt=min⁡(1,χη(Ctctrl+ϵ)).\alpha_t = \min\left( 1, \frac{\chi} {\eta(C_t^{\mathrm{ctrl}}+\epsilon)} \right).

This is the controller used in the first global-throttle sanity experiment. It is the instantaneous solution of

ηαtCtctrl=χ.\eta\alpha_t C_t^{\mathrm{ctrl}}=\chi.

Therefore, it can be interpreted as the zero-time-constant limit of a dynamical controller.

Case 1: First-order gain controller​

Instead of setting αt\alpha_t instantaneously, we can treat it as a state variable. In continuous time:

α˙=kα(χ−ηαCctrl).\dot{\alpha} = k_\alpha(\chi-\eta\alpha C^{\mathrm{ctrl}}).

If ηαCctrl>χ\eta\alpha C^{\mathrm{ctrl}}>\chi, then α˙<0\dot{\alpha}<0, so the controller brakes. If ηαCctrl<χ\eta\alpha C^{\mathrm{ctrl}}<\chi, then α˙>0\dot{\alpha}>0, so the controller accelerates.

For locally constant CctrlC^{\mathrm{ctrl}}, the equilibrium is

α⋆=χηCctrl.\alpha^\star = \frac{\chi} {\eta C^{\mathrm{ctrl}}}.

A discrete-time implementation is

αt+1=αt+kα(χ−ηαtCtctrl).\alpha_{t+1} = \alpha_t + k_\alpha \left( \chi-\eta\alpha_t C_t^{\mathrm{ctrl}} \right).

To choose kαk_\alpha, assume Ctctrl=CC_t^{\mathrm{ctrl}}=C is locally constant. Then the error ut=αt−α⋆u_t=\alpha_t-\alpha^\star evolves as

ut+1=(1−kαηC)ut.u_{t+1} = (1-k_\alpha\eta C)u_t.

Thus, stable α\alpha-adaptation requires

0<kαηC<2.0<k_\alpha\eta C<2.

For non-oscillatory adaptation, choose

0<kαηC<1.0<k_\alpha\eta C<1.

Using an estimated maximum curvature Cmax⁡C_{\max}, a conservative choice is

kα=ckηCmax⁡,0<ck<1.k_\alpha = \frac{c_k}{\eta C_{\max}}, \qquad 0<c_k<1.
Key insight

A first-order controller can adapt the global throttle αt\alpha_t over time to keep the effective learning rate near the stability margin. The adaptation speed can be tuned via kαk_\alpha. A first choice can be:

kα=0.5ηCmax⁡⇝α˙=kα(χ−ηαCctrl).k_\alpha = \frac{0.5}{\eta C_{\max}} \quad \leadsto \quad \dot{\alpha}=k_\alpha(\chi-\eta\alpha C^{\mathrm{ctrl}}).

Case 2: Second-order damped gain controller​

A second-order controller gives αt\alpha_t its own velocity:

α¨+kdα˙+kp(ηαCctrl−χ)=0.\ddot{\alpha} + k_d\dot{\alpha} + k_p \left( \eta\alpha C^{\mathrm{ctrl}}-\chi \right) = 0.

For locally constant CctrlC^{\mathrm{ctrl}}, the equilibrium is again

α⋆=χηCctrl.\alpha^\star = \frac{\chi} {\eta C^{\mathrm{ctrl}}}.

Let

u=α−α⋆.u=\alpha-\alpha^\star.

Then

ηαCctrl−χ=ηCctrlu,\eta\alpha C^{\mathrm{ctrl}}-\chi = \eta C^{\mathrm{ctrl}}u,

and the local controller dynamics become

u¨+kdu˙+kpηCctrlu=0.\ddot{u} + k_d\dot{u} + k_p\eta C^{\mathrm{ctrl}}u = 0.

This is a damped second-order system with natural frequency

ωn=kpηCctrl\omega_n = \sqrt{k_p\eta C^{\mathrm{ctrl}}}

and damping ratio

ζ=kd2kpηCctrl.\zeta = \frac{k_d} {2\sqrt{k_p\eta C^{\mathrm{ctrl}}}}.

Critical damping occurs when

kd=2kpηCctrl.k_d = 2\sqrt{k_p\eta C^{\mathrm{ctrl}}}.

In practice, choose a reference curvature CrefC_{\mathrm{ref}}, a desired gain-settling time TαT_\alpha, and set

ωn=4Tα,\omega_n=\frac{4}{T_\alpha}, kp=ωn2ηCref,k_p = \frac{\omega_n^2} {\eta C_{\mathrm{ref}}}, kd=2ωn.k_d=2\omega_n.

If one wants conservative non-ringing behavior over all expected curvatures C≤Cmax⁡C\leq C_{\max}, choose

kd≥2kpηCmax⁡.k_d \geq 2\sqrt{k_p\eta C_{\max}}.

A hardware-friendly discrete version is

vα,t+1=βvα,t+kα(χ−ηαtCtctrl)αt+1=αt+vα,t+1\begin{aligned} v_{\alpha,t+1} &= \beta v_{\alpha,t} + k_\alpha \left( \chi-\eta\alpha_t C_t^{\mathrm{ctrl}} \right) \\[8pt] \alpha_{t+1} &= \alpha_t+v_{\alpha,t+1} \end{aligned}

which is basically a momentum update on α\alpha with a proportional term on the margin error.

In fact, a good choice could be to make this system be critically-damped at the reference curvature CrefC_{\mathrm{ref}}:

kα=16Tα2ηCref,β=2kαηCref−1.k_\alpha = \frac{16}{T_\alpha^2\eta C_{\mathrm{ref}}}, \qquad \beta = 2\sqrt{k_\alpha\eta C_{\mathrm{ref}}}-1.
But what is the best choice for $C_{\mathrm{ref}}$?

???

Case 3: Quantization-aware gain feasibility​

In fixed-point training, reducing αt\alpha_t too much can make the update vanish after quantization. If qΔq_\Delta is the update quantum, a useful update requires approximately

ηαt∥Gt∥2≳qΔ.\eta\alpha_t\|G_t\|_2 \gtrsim q_\Delta.

Therefore,

αt≳qΔη∥Gt∥2+ϵ.\alpha_t \gtrsim \frac{q_\Delta} {\eta\|G_t\|_2+\epsilon}.

At the same time, stability requires

αt≲χηCtctrl+ϵ.\alpha_t \lesssim \frac{\chi} {\eta C_t^{\mathrm{ctrl}}+\epsilon}.

Define

αmin⁡,t=qΔη∥Gt∥2+ϵ\alpha_{\min,t} = \frac{q_\Delta} {\eta\|G_t\|_2+\epsilon}

and

αmax⁡,t=χηCtctrl+ϵ.\alpha_{\max,t} = \frac{\chi} {\eta C_t^{\mathrm{ctrl}}+\epsilon}.

Useful stable fixed-point learning requires

αmin⁡,t≤αmax⁡,t.\alpha_{\min,t} \leq \alpha_{\max,t}.

If this interval collapses, the current fixed-point format cannot simultaneously provide stable and useful updates.