EXP-001: Global Throttle Parameter Sweep
Experiment Trace
| Field | Value |
|---|---|
| Dataset | DS-AFFINE-LINEAR-000 |
| Model | MDL-DENSE1-LINEAR-NOBIAS-000 |
| Optimizer | SGD-style online loop |
| Controller | NONE GT GT |
| Precision mode | ap_fixed<16,8, AP_RND, AP_SAT> |
| Ablation focus | ST-000 |
| Drift | Parameter sweep |
| Learning rate range |
Summary and background
This experiment aims to investigate how the stability of the system depends on the learning rate and the drift parameter under different global throttle controllers. By sweeping over a range of and values, we can visualize the stability regions and understand the impact of the global throttle on the training dynamics. As we introduced in the Formulation Overview section, the condition of convergence depends on:
where we estimate the with the curvature proxy , and thus:
Because this depends on (the drift parameter), we expect to see a stability region in the space, for which the system diverges (final loss is bigger than some threshold) when no controller is used; but that it does converge (or at least that the divergence area is smaller) when the global throttle is used ( is used):
Setup and code
Here we will briefly describe the setup of the experiment, and then we will jump to the results. For more details on the code, please check the notebook where the code is fully documented and explained.
First, we define the global parameters for the simulation (sweep, thresholds, etc):
N = 1000
SEED = 42
MAX_STEPS = 1000
TARGET_LOSS = None
PATIENCE = 0
BLOWUP_LOSS = 1e2
controller_enabled = True
controller_name = [None, "gt-order-0", "gt-order-1", "gt-order-2"][2 if controller_enabled else 0]
# Gamma space
ETAS = np.logspace(-2, 0.3, 41) # 0.01 to about 2
GAMMAS = np.geomspace(1.0, 4.0, 41) # 1 to 16
CHI = 1.0
PRECISIONS = [[16, 6]] # [[WL, IWL]]
The creation of the dataset and the model is exactly as shown in EXP-000A and EXP-000B, so we skip the details here. Because this test does require a few utilities, we'll add them here explicitly.
Reinitialization utility
We need to make sure that all tests start from the same initial conditions (same initial weights, same data distribution, etc) to make sure that the results are comparable. For that, we create a utility to reset the model weights to their initial state before every run of the simulation.
# Utility to reset the model weights to their initial state (we want every point to start from the same initial condition)
initial_weights = model.model.get_weights()
def reset_model(model):
model.model.set_weights(initial_weights)
Make precision utility
In the results shown here we do not sweep over different quantization/precisions, but doing so would be straightforward by just adding more precisions to the PRECISIONS variable and then looping over them in the main loop. For that reason, we create a utility to convert the model weights to a given precision, so that we can easily apply it in case we want to add quantization sweeps in the future.
Note that in this case we assume some relations between the precisions of different tensors (e.g., the gradient WL is 4 bits wider than the weight WL, etc). This is, by no means, the optimal way to do it, and it should be investigated further in the future to answer the question what is the optimal precision stack for any layer, given the precision of the weights?. For now, we just assume some relations that seem reasonable and that are consistent with what we have seen in previous experiments.
from enabol import dtypes, PrecisionDict
def make_precision_dict(BASE_WL, BASE_IWL, QMODE="AP_RND", OMODE="AP_SAT"):
wide_weight = dtypes.ap_fixed(WL=BASE_WL, IWL=BASE_IWL, QMODE=QMODE, OMODE=OMODE)
wide_activation = dtypes.ap_fixed(WL=BASE_WL, IWL=BASE_IWL+2, QMODE=QMODE, OMODE=OMODE)
wide_gradient = dtypes.ap_fixed(WL=BASE_WL+4, IWL=BASE_IWL+2, QMODE=QMODE, OMODE=OMODE)
wide_update = dtypes.ap_fixed(WL=BASE_WL+4, IWL=BASE_IWL, QMODE=QMODE, OMODE=OMODE)
wide_accumulator = dtypes.ap_fixed(WL=BASE_WL+12, IWL=BASE_IWL+8, QMODE=QMODE, OMODE=OMODE)
wide_loss = dtypes.ap_fixed(WL=2*BASE_WL, IWL=BASE_IWL+10, QMODE=QMODE, OMODE=OMODE)
return PrecisionDict({
"input": {"value": wide_activation},
"dense0": {
"weight": wide_weight,
"activation": wide_activation,
"gradient": wide_gradient,
"update": wide_update,
"accumulator": wide_accumulator,
},
"loss": {"value": wide_loss},
})
Summarizing the results
Inside the loop we will get some results from the metrics and we'll have to summarize them in a way that can be useful later. For that, we create a small utility function:
def safe_max(h, key):
if key not in h:
return np.nan
x = np.asarray(h[key], dtype=float)
if x.size == 0:
return np.nan
return float(np.nanmax(x))
def safe_min(h, key):
if key not in h:
return np.nan
x = np.asarray(h[key], dtype=float)
if x.size == 0:
return np.nan
return float(np.nanmin(x))
def safe_final(h, key):
if key not in h:
return np.nan
x = np.asarray(h[key], dtype=float)
if x.size == 0:
return np.nan
return float(x[-1])
def summarize_run(h, eta, gamma, precision_dict, controller_name):
loss = h["loss"]
finite = np.all(np.isfinite(loss))
diverged = (not finite) or bool(np.any(h["diverged"] > 0)) or np.nanmax(loss) > BLOWUP_LOSS
converged = False
converged_step = np.nan
if not diverged:
if TARGET_LOSS is not None:
below = loss < TARGET_LOSS
for idx in range(len(loss) - PATIENCE):
if np.all(below[idx:idx + PATIENCE]):
converged = True
converged_step = idx
break
else:
converged = True
converged_step = len(loss)
return {
"eta": eta,
"gamma": gamma,
"controller": controller_name or "none",
"precision": str(precision_dict["dense0"]["weight"]),
"diverged": diverged,
"converged": converged,
"converged_step": converged_step,
"final_loss": float(loss[-1]),
"log_final_loss": float(np.log10(loss[-1] + 1e-12)) if not diverged else np.nan,
"final_alpha": float(h["alpha"][-1]),
"min_alpha": float(np.nanmin(h["alpha"])),
"min_update_cosine": float(np.nanmin(h["update_cosine"])),
"max_grad_sat": float(np.nanmax(h["gradient_saturation_fraction_max"])),
"max_curvature_proxy": safe_max(h, "curvature_proxy"),
"max_curvature_for_control": safe_max(h, "curvature_for_control"),
"max_hessian_lambda": safe_max(h, "hessian_lambda_max"),
"max_hessian_spectral_norm": safe_max(h, "hessian_spectral_norm"),
"max_margin_lambda_raw": safe_max(h, "stability_margin_lambda_raw"),
"max_margin_lambda_ctrl": safe_max(h, "stability_margin_lambda_ctrl"),
"max_margin_norm_raw": safe_max(h, "stability_margin_norm_raw"),
"max_margin_norm_ctrl": safe_max(h, "stability_margin_norm_ctrl"),
"max_spectral_radius_raw": safe_max(h, "spectral_radius_raw"),
"max_spectral_radius_ctrl": safe_max(h, "spectral_radius_ctrl"),
"final_margin_lambda_raw": safe_final(h, "stability_margin_lambda_raw"),
"final_margin_lambda_ctrl": safe_final(h, "stability_margin_lambda_ctrl"),
"min_controller_feasible": safe_min(h, "controller_feasible"),
"min_alpha_min_bound": safe_min(h, "alpha_min_bound"),
"max_alpha_min_bound": safe_max(h, "alpha_min_bound"),
"min_alpha_max_bound": safe_min(h, "alpha_max_bound"),
"max_alpha_max_bound": safe_max(h, "alpha_max_bound"),
}
Sweep loop
The main loop that performs the sweeps, trains, and summarizes the results is the following:
results = []
total_runs = len(PRECISIONS) * len(ETAS) * len(GAMMAS)
for i, (WL, IWL) in enumerate(PRECISIONS):
precision_dict = make_precision_dict(WL, IWL)
for j, GAMMA in enumerate(GAMMAS):
# Make drifted dataset
Xd = GAMMA * X
# Metrics
drift_metrics = MetricsConfig(
profiles=("core", "geometry", "stability", "teacher", "quantization", "controller_bounds"),
reference_A=dataset.reference_weight_matrix / GAMMA,
)
for k, ETA in enumerate(ETAS):
# Print
iteration = i * len(GAMMAS) * len(ETAS) + j * len(ETAS) + k
print(f"[{(iteration+1)/total_runs:.2%}] Running with ETA={ETA}, GAMMA={GAMMA}, {precision_dict['dense0']['weight']}")
# Reset model weights
reset_model(model)
# We need a new controller for every run, since it has states (not for 0th or None, but it's good practice anyways)
controller = enabol.Controller.from_str(controller_name, chi=CHI)
# Train
h_nom = model.train_instrumented(
Xd,
Y,
epochs=MAX_STEPS,
stop_loss=None, # auto means lower bound according to quantization.
diverge_loss=BLOWUP_LOSS, # auto means upper bound according to quantization.
stop_patience=1,
stop_min_steps=0,
batch_size=len(Xd),
learning_rate=ETA,
controller=controller,
metrics=drift_metrics,
precision_dict=precision_dict,
verbose = 'progressbar'
)
# Append result
results.append(summarize_run(h_nom, ETA, GAMMA, precision_dict, controller_name))
import pandas as pd
df = pd.DataFrame(results)
FILE = f"../results/parameter_sweep_controller_{controller_name}.csv"
df.to_csv(FILE, index=False)
Plotting the results
Finally, we can plot the results using the following utility function:
# Now we need to form the (eta,gamma) grid that's the same for all metrics.
import numpy as np
import matplotlib.pyplot as plt
def _parse_float(value: str) -> float:
if value == "" or value is None:
return np.nan
return float(value)
def get_mesh(df: pd.DataFrame) -> tuple[np.ndarray, np.ndarray]:
etas = np.array(sorted(df['eta'].unique()), dtype=float)
gammas = np.array(sorted(df['gamma'].unique()), dtype=float)
return etas, gammas
def pivot_grid(
df: pd.DataFrame,
etas: np.ndarray,
gammas: np.ndarray,
*,
value: str,
black_diverged: bool = True,
) -> np.ndarray:
grid = np.full((len(etas), len(gammas)), np.nan, dtype=float)
eta_index = {eta: i for i, eta in enumerate(etas)}
gamma_index = {gamma: j for j, gamma in enumerate(gammas)}
for _, row in df.iterrows():
eta = float(row["eta"])
gamma = float(row["gamma"])
if black_diverged and row.get("diverged"):
continue
grid[eta_index[eta], gamma_index[gamma]] = _parse_float(row.get(value, ""))
return grid
def plot_phase(
etas: np.ndarray,
gammas: np.ndarray,
*,
grids: dict[str, np.ndarray],
title: str | None = None,
cmaps: dict[str, str] = {"log_final_loss": "viridis"},
):
NPLOTS = len(grids)
NCOLS = np.sqrt(NPLOTS).round().astype(int)
NROWS = np.ceil((NPLOTS / NCOLS)).astype(int)
fig, axs = plt.subplots(nrows=NROWS, ncols=NCOLS, figsize=(7*NCOLS, 5*NROWS),
sharex=True, sharey=True)
axs = np.array(axs).flatten() if NPLOTS > 1 else np.array([axs])
if len(etas) >= 2 and len(gammas) >= 2:
for ax, (value, grid) in zip(axs, grids.items()):
im = ax.imshow(
np.ma.masked_invalid(grid),
origin="lower",
aspect="auto",
interpolation="nearest",
cmap=cmaps.get(value, "viridis"),
extent=[gammas.min(), gammas.max(), etas.min(), etas.max()],
)
ax.set_xlabel("γ")
ax.set_ylabel("η")
ax.set_yscale("log")
ax.set_xscale("log")
ax.set_title(title or value)
fig.colorbar(im, ax=ax, label=value)
else:
raise ValueError("Need at least 2 unique values for eta and gamma to plot the phase diagram.")
fig.tight_layout()
plt.show()
which is called with the following code after loading the results from the CSV:
import pandas as pd
# Reload (so I can re-run this code without re-running the whole sweep)
FILE = f"../results/parameter_sweep_controller_{controller_name}.csv"
#FILE = "../results/parameter_sweep_controller_gt0.csv"
#FILE = "../results/parameter_sweep_controller_gt-order-1.csv"
df = pd.read_csv(FILE)
from IPython.display import display
display(df)
etas, gammas = get_mesh(df)
grids = {key: pivot_grid(df, etas, gammas, value=key)
for key in ["log_final_loss", "final_margin_lambda_raw", "min_alpha", "min_update_cosine", "max_spectral_radius_raw", "max_curvature_proxy"]}
plot_phase(etas, gammas, grids=grids, cmaps={"log_final_loss": "viridis", "final_margin_lambda_raw": "cividis", "min_alpha": "bwr", "min_update_cosine": "plasma", "max_grad_sat": "inferno", "max_spectral_radius_raw": "Spectral", "max_curvature_proxy": "magma_r"})
# Compute the area of the divergence region (where log_final_loss is NaN)
divergence_mask = np.isnan(grids["log_final_loss"])
divergence_area = np.sum(divergence_mask) * (np.diff(gammas).mean() * np.diff(etas).mean())
total_area = (gammas.max() - gammas.min()) * (etas.max() - etas.min())
divergence_fraction = divergence_area / total_area
print(f"Estimated area of divergence region fraction: {divergence_fraction:.4%}")
Results
The final results are shown below. The results mainly consist of a set of plots for different metrics (final loss, stability margin, curvature proxy, etc) in the space, for different controllers. The main metric of interest is the final loss (or log final loss), which we use to determine whether the system diverged or not. The different tests are:
No controller
In this case no controller was used, which means that the result is just the plain SGD algorithm without any stability control. As mentioned at the top, we would expect a large divergence region for any condition over . Figure 3 shows the phase diagram for three metrics: log of the final loss value, minimum update cosine similarity, and minimum alpha value. For each plot, the dashed line represents precisely this theoretical boundary of divergence (note plots are in log-log scale, so the boundary is a straight line with slope -2).
The most outstanding elements shown in this plot are:
- The divergence area (estimated to be around of the total shown area) is quite large and it is almost exactly located to the NE of the theoretical boundary, which is consistent. The only reason why the boundary is not sharper is hypothesized to be that the simulation did not run for long enough.

Global throttle order 0
In this case, we activate the simplest throttle controller, which is of order . This means that at any time , the controller tries to apply a deterministic value computed according to:
where is just a small value to avoid division by zero. The expectation is that this control mechanism will reduce the divergence area. The results are shown in Figure 4. The most outstanding elements are:
- The divergence area is significantly reduced (estimated to be around of the total shown area), which is consistent with our expectations and shows that the global throttle controller is effective in improving stability.
- Note that in reality, the divergence area shown in this plot does not necessarily correlate with true network divergence, but rather with the fact that the final loss is above the
BLOWUP_LOSSthreshold. What this means is that, as long as those regions have an equivalent value of , the network will remain stable, since the updates will be effectively stopped.

Global throttle order 1
In this final case, we activate the global throttle controller of order 1. This means that at any time , the coefficient reacts to the curvature via the following dynamics:
Because the initial alpha is , this means that this system is inherently much more stable than the previous one, since it starts with a very conservative throttle and only after some iterations of stable curvature it starts to increase the throttle and allow for more aggressive updates. The results are shown in Figure 5. The most outstanding elements are:
- The divergence area is further reduced (estimated to be around of the total shown area), which is consistent with our expectations and shows that the global throttle controller of order 1 is even more effective in improving stability compared to the order 0 controller.
- Again, we want to highlight that the divergence area shown in this plot does not necessarily correlate with true network divergence, but rather with the fact that the final loss is above the
BLOWUP_LOSSthreshold. As long as those regions have an equivalent value of , the network will remain stable, since the updates will be effectively stopped.

Conclusions
The results of this experiment show a clear improvement in stability when using the global throttle controllers, especially the order 1 controller. The divergence area is significantly reduced compared to the no controller case, which is consistent with our theoretical analysis.
The main conclusion is that the global throttle controllers are effective in improving stability and guaranteeing convergence in a much larger region of the hyperparameter space, at least for the ablation test of drift and learning rate.