Instead of projecting each row/column independently (which as we saw in the kappa-budgeting section introduces rotation in the θ plane), we introduce a single global scalar:
0<αt≤1.
Then the SGD update becomes:
θt+1=θt−αtηGtwithGt=∇θL(θt).
This is important because the update remains parallel to the original gradient update.
Raw update:
Δθtraw=−ηGt.
Controlled update:
Δθtctrl=−αtηGt.
Therefore:
Δθtctrl=αtΔθtraw.
Now, to verify whether direction is preserved or not, we can use the cosine similarity:
As mentioned before, we want this to be 1, which means the scaling used to ensure stability (on every layer) do not change the direction of the update. Now, if we replace the definition for our global throttle, we get:
In section overview we showed that convergence of SGD is guaranteed if the learning rate η satisfies:
η<λmax(H)2,
where H is the Hessian of the loss. Therefore, to adapt αt to ensure stability, we need an estimate of λmax(H). However, as also introduced in the overview section, finding the actual hessian is computationally expensive and sometimes even not feasible. Therefore, we need a cheap online estimate of the local curvature/sensitivity of the loss landscape.
Let's recall the formulas for the gradient and the Hessian.
G(θ)H(θ)=∇θL(θ)=∇θG(θ)=∇θ2L(θ)
Now, we can approximate the hessian using Taylor expansion around some operating point. For a small change in θ, we have:
Δθt=θt−θt−1,
Taylor expansion gives:
Gt−Gt−1≈Ht(θt−θt−1).
So:
ΔGt≈HtΔθt.
Taking norms:
∥ΔGt∥≈∥HtΔθt∥.
Divide by:
∥Δθt∥.
Then:
∥Δθt∥∥ΔGt∥≈∥Δθt∥∥HtΔθt∥.
This is an observed directional curvature estimate. In other words, this is a first-order approximation of the curvature along the direction the optimizer just moved.
So as we can see, for a quadratic loss, this estimator is an upper bound on the true Hessian spectral norm. For general losses, it is a local directional curvature estimate that can be used as a proxy for the local Lipschitzness of the gradient field.
A caveat
The previous estimate is not perfect. It is very good for convex problems and things like a single-dense layer architecture, but it can (and will) be inaccurate for non-convex problems and more complex architectures. However, it is a simple and computationally cheap way to get a sense of the local curvature/sensitivity, which is what we need to adapt the learning rate and prevent divergence.
The global throttle mechanism preserves the geometry of the original gradient update, but it adaptively reduces the learning rate when the gradient field becomes stiff/sharp. This allows the model to keep learning without diverging, even under distribution shift.
For deeper networks, ReLU networks, CNNs, cross-entropy, etc., the loss is not globally quadratic.
For a general network:
y^=f(x;θ),
with loss:
L(θ),
we use a local Taylor expansion:
L(θ+δ)≈L(θ)+G(θ)⊤δ+21δ⊤H(θ)δ.
The gradient descent map is:
θt+1=θt−ηG(θt).
Linearizing perturbations:
δt+1≈(I−ηHt)δt.
So the same stability idea applies locally.
However:
Ht changes over time,
Ht may be indefinite,
ReLU networks are piecewise smooth,
mini-batch gradients include sampling noise,
quantization introduces non-smooth perturbations.
Therefore, for general networks we should not claim Ct=λmax(Ht) exactly. Instead, we claim: Ct is an online estimate of local update-field sensitivity:
Ct≈∥Δθt∥∥HtΔθt∥.
Then, for a general loss, we want to keep:
αtηLt<χ,
where Lt is a local estimate of gradient-field Lipschitzness and chi is some stability margin (normally χ≤2). Then, for instance, if the controller uses Lt≈Ctctrl:
αtηCtctrl≤χ.
Key insight
Even for nonlinear, non-convex networks, this global throttle mechanism can still be used as a stabilizer by using the local curvature/sensitivity estimate to adapt the learning rate and prevent divergence.
Instead of setting αt instantaneously, we can treat it as a state variable. In continuous time:
α˙=kα(χ−ηαCctrl).
If ηαCctrl>χ, then α˙<0, so the controller brakes. If ηαCctrl<χ, then α˙>0, so the controller accelerates.
For locally constant Cctrl, the equilibrium is
α⋆=ηCctrlχ.
A discrete-time implementation is
αt+1=αt+kα(χ−ηαtCtctrl).
To choose kα, assume Ctctrl=C is locally constant. Then the error ut=αt−α⋆ evolves as
ut+1=(1−kαηC)ut.
Thus, stable α-adaptation requires
0<kαηC<2.
For non-oscillatory adaptation, choose
0<kαηC<1.
Using an estimated maximum curvature Cmax, a conservative choice is
kα=ηCmaxck,0<ck<1.
Key insight
A first-order controller can adapt the global throttle αt over time to keep the effective learning rate near the stability margin. The adaptation speed can be tuned via kα. A first choice can be:
In fixed-point training, reducing αt too much can make the update vanish after quantization. If qΔ is the update quantum, a useful update requires approximately
ηαt∥Gt∥2≳qΔ.
Therefore,
αt≳η∥Gt∥2+ϵqΔ.
At the same time, stability requires
αt≲ηCtctrl+ϵχ.
Define
αmin,t=η∥Gt∥2+ϵqΔ
and
αmax,t=ηCtctrl+ϵχ.
Useful stable fixed-point learning requires
αmin,t≤αmax,t.
If this interval collapses, the current fixed-point format cannot simultaneously provide stable and useful updates.