Skip to main content

🧰 Utils Module Reference

Status:ValidIn progress
Tags:ImplementationUtilsStability
Last modified:

What this page covers

This page is a coder-facing reference for enabol/utils.py: tensor flattening, stable norms, update-direction metrics, matrix norms, analytic Hessian helpers, local stability metrics, and the clean half-MSE loss used by the first ablations.

Module Map​

GroupFunctionsUsed by
Tensor geometryflatten_tensors, tensor_l2_norm, safe_cosineBaseModel.train_instrumented(...)
Matrix diagnosticsspectral_norm_np, matrix_norms_npgain/norm logging and future κ analysis
Hessian diagnosticsanalytic_single_dense_hessian, hessian_metrics_np, stability_metrics_from_hessianEXP-000A / one-layer sanity tests
Loss helpershalf_mse_batch_lossclean Hessian experiments
Utils TODOs
  • Keep pure NumPy diagnostics and TensorFlow training helpers clearly separated as this file grows.
  • Move plotting or experiment-specific metrics out of utils.py; this file should stay reusable.
  • Add tests for edge cases such as empty tensor lists, non-finite matrices, and singular Hessians.

Imports And Dependencies​

import numpy as np
import tensorflow as tf
from typing import Sequence, Optional, Dict, Any

utils.py mixes TensorFlow helpers and NumPy diagnostics. That is acceptable for the current small harness, but if the module grows, split candidates are:

Future fileContents
tensor_utils.pyTensorFlow flattening, norms, cosine helpers.
linear_diagnostics.pyMatrix norms, Hessian helpers, stability metrics.
losses.pySmall custom losses used by ablations.

Tensor Geometry Functions​

flatten_tensors(tensors)​

def flatten_tensors(tensors: Sequence[tf.Tensor]) -> tf.Tensor:
...

Flattens a sequence of tensors and concatenates them into one global vector. None entries are skipped.

This is central to the closed-loop controller because the controller treats all trainable parameters as one vector:

θ=flatten⁡(W1,b1,W2,b2,…)\theta = \operatorname{flatten}(W_1, b_1, W_2, b_2, \ldots)

and all gradients as one vector:

G=flatten⁡(∂L∂W1,∂L∂b1,…).G = \operatorname{flatten}\left(\frac{\partial L}{\partial W_1}, \frac{\partial L}{\partial b_1}, \ldots\right).

Usage:

theta = flatten_tensors(model.trainable_variables)
grad_vec = flatten_tensors(grads)

If all inputs are None or the list is empty, it returns an empty tf.float32 tensor.

tensor_l2_norm(x, eps=1e-12)​

def tensor_l2_norm(x: tf.Tensor, eps: float = 1e-12) -> tf.Tensor:
return tf.sqrt(tf.reduce_sum(tf.square(x)) + eps)

Computes a stable L2 norm. The small eps prevents exact zero inside the square root.

Usage:

grad_norm = tensor_l2_norm(grad_vec)
update_norm = tensor_l2_norm(delta_actual)

safe_cosine(a, b, eps=1e-12)​

def safe_cosine(a: tf.Tensor, b: tf.Tensor, eps: float = 1e-12) -> tf.Tensor:
denom = tensor_l2_norm(a, eps) * tensor_l2_norm(b, eps) + eps
return tf.reduce_sum(a * b) / denom

Computes cosine similarity between two flattened vectors.

Current use:

cos_actual = safe_cosine(delta_actual, -grad_vec)

Interpretation:

ValueMeaning
near 1actual update follows the intended descent direction.
near 0actual update is mostly orthogonal to the intended descent direction.
below 0actual update is partly anti-descent.
Cosine is a diagnostic, not necessarily a hardware controller signal

The update cosine is useful in software because we can compare the raw intended update against the post-quantization applied update. In real hardware, the same register quantization may make that comparison unavailable or expensive. Treat it as an analysis metric unless a hardware-feasible estimator is explicitly designed.

Matrix Diagnostic Functions​

spectral_norm_np(W)​

def spectral_norm_np(W: np.ndarray) -> float:
...

Computes the matrix spectral norm using SVD. If the input contains non-finite values or SVD fails, it returns np.nan rather than crashing the experiment.

Behavior:

InputOutput
finite 2D matrixlargest singular value.
finite non-2D arrayvector L2 norm after flattening.
non-finite arraynp.nan.
failed SVDnp.nan.

matrix_norms_np(W)​

def matrix_norms_np(W: np.ndarray) -> dict[str, float]:
...

Returns common matrix norms in one dictionary:

{
"fro": ...,
"l1": ...,
"linf": ...,
"spectral": ...,
}

These are the natural diagnostics for future κ/gain studies:

∥Wl∥F,∥Wl∥1,∥Wl∥∞,∥Wl∥2.\lVert W_l \rVert_F, \qquad \lVert W_l \rVert_1, \qquad \lVert W_l \rVert_\infty, \qquad \lVert W_l \rVert_2.

If any value in W is non-finite, every returned norm is np.nan.

Hessian And Stability Functions​

analytic_single_dense_hessian(...)​

def analytic_single_dense_hessian(
X_batch: np.ndarray,
d_out: int,
keras_mse_scaling: bool = True,
) -> np.ndarray:
...

Analytic Hessian for one Dense layer without bias:

Y^=XWT.\hat{Y} = X W^T.

The input covariance is:

ΣX=XTXN.\Sigma_X = \frac{X^T X}{N}.

For the clean half-MSE loss:

L=12N∥Y^−Y∥F2,L = \frac{1}{2N}\lVert \hat{Y} - Y \rVert_F^2,

this gives:

H=Idout⊗ΣX.H = I_{d_{out}} \otimes \Sigma_X.

If keras_mse_scaling=True, the helper uses the scaling associated with Keras-style mean squared error:

H=2doutIdout⊗ΣX.H = \frac{2}{d_{out}} I_{d_{out}} \otimes \Sigma_X.

Usage:

H = analytic_single_dense_hessian(X_batch, d_out=2, keras_mse_scaling=False)
metrics = hessian_metrics_np(H)

hessian_metrics_np(H)​

def hessian_metrics_np(H: np.ndarray) -> Dict[str, float]:
...

Computes core Hessian metrics after symmetrizing defensively:

{
"hessian_lambda_max": lam_max,
"hessian_lambda_min": lam_min,
"hessian_spectral_norm": h_norm_2,
}

These are used to compare the cheap online curvature proxy against true local curvature in toy experiments.

stability_metrics_from_hessian(H, eta, alpha=1.0)​

def stability_metrics_from_hessian(
H: np.ndarray,
eta: float,
alpha: float = 1.0,
) -> Dict[str, float]:
...

Computes local discrete-time stability metrics for the SGD-like update:

θt+1=θt−αη∇L(θt).\theta_{t+1} = \theta_t - \alpha \eta \nabla L(\theta_t).

The local linearized update map is:

M=I−αηH.M = I - \alpha \eta H.

The function returns:

{
"stability_margin_lambda": alpha * eta * lambda_max,
"stability_margin_norm": alpha * eta * hessian_spectral_norm,
"spectral_radius_update_map": rho(I - alpha * eta * H),
}

For a convex quadratic, the familiar gradient-descent stability condition is approximately:

0<αηλi(H)<2.0 < \alpha\eta\lambda_i(H) < 2.

So the scalar margin:

αηλmax(H)\alpha\eta\lambda_{max}(H)

should remain below 2.

Loss Helpers​

half_mse_batch_loss(y_true, y_pred)​

def half_mse_batch_loss(y_true: tf.Tensor, y_pred: tf.Tensor) -> tf.Tensor:
...

Computes:

L=12N∥Y^−Y∥F2.L = \frac{1}{2N}\lVert \hat{Y} - Y \rVert_F^2.

This loss is intentionally used in the first ablations because it makes the one-layer Hessian formula clean:

H=Idout⊗ΣX.H = I_{d_{out}} \otimes \Sigma_X.

Usage:

with tf.GradientTape() as tape:
y_pred = model(x_batch, training=True)
loss = half_mse_batch_loss(y_batch, y_pred)

Extension Notes​

Keep this module boring and reusable. If a helper starts depending on a specific experiment, plot layout, or documentation artifact, it probably belongs in a notebook, history.py, or a future experiment-specific module instead of utils.py.