🧰 Utils Module Reference
This page is a coder-facing reference for enabol/utils.py: tensor flattening, stable norms, update-direction metrics, matrix norms, analytic Hessian helpers, local stability metrics, and the clean half-MSE loss used by the first ablations.
Module Map
| Group | Functions | Used by |
|---|---|---|
| Tensor geometry | flatten_tensors, tensor_l2_norm, safe_cosine | BaseModel.train_instrumented(...) |
| Matrix diagnostics | spectral_norm_np, matrix_norms_np | gain/norm logging and future κ analysis |
| Hessian diagnostics | analytic_single_dense_hessian, hessian_metrics_np, stability_metrics_from_hessian | EXP-000A / one-layer sanity tests |
| Loss helpers | half_mse_batch_loss | clean Hessian experiments |
- Keep pure NumPy diagnostics and TensorFlow training helpers clearly separated as this file grows.
- Move plotting or experiment-specific metrics out of
utils.py; this file should stay reusable. - Add tests for edge cases such as empty tensor lists, non-finite matrices, and singular Hessians.
Imports And Dependencies
import numpy as np
import tensorflow as tf
from typing import Sequence, Optional, Dict, Any
utils.py mixes TensorFlow helpers and NumPy diagnostics. That is acceptable for the current small harness, but if the module grows, split candidates are:
| Future file | Contents |
|---|---|
tensor_utils.py | TensorFlow flattening, norms, cosine helpers. |
linear_diagnostics.py | Matrix norms, Hessian helpers, stability metrics. |
losses.py | Small custom losses used by ablations. |
Tensor Geometry Functions
flatten_tensors(tensors)
def flatten_tensors(tensors: Sequence[tf.Tensor]) -> tf.Tensor:
...
Flattens a sequence of tensors and concatenates them into one global vector. None entries are skipped.
This is central to the closed-loop controller because the controller treats all trainable parameters as one vector:
and all gradients as one vector:
Usage:
theta = flatten_tensors(model.trainable_variables)
grad_vec = flatten_tensors(grads)
If all inputs are None or the list is empty, it returns an empty tf.float32 tensor.
tensor_l2_norm(x, eps=1e-12)
def tensor_l2_norm(x: tf.Tensor, eps: float = 1e-12) -> tf.Tensor:
return tf.sqrt(tf.reduce_sum(tf.square(x)) + eps)
Computes a stable L2 norm. The small eps prevents exact zero inside the square root.
Usage:
grad_norm = tensor_l2_norm(grad_vec)
update_norm = tensor_l2_norm(delta_actual)
safe_cosine(a, b, eps=1e-12)
def safe_cosine(a: tf.Tensor, b: tf.Tensor, eps: float = 1e-12) -> tf.Tensor:
denom = tensor_l2_norm(a, eps) * tensor_l2_norm(b, eps) + eps
return tf.reduce_sum(a * b) / denom
Computes cosine similarity between two flattened vectors.
Current use:
cos_actual = safe_cosine(delta_actual, -grad_vec)
Interpretation:
| Value | Meaning |
|---|---|
near 1 | actual update follows the intended descent direction. |
near 0 | actual update is mostly orthogonal to the intended descent direction. |
below 0 | actual update is partly anti-descent. |
The update cosine is useful in software because we can compare the raw intended update against the post-quantization applied update. In real hardware, the same register quantization may make that comparison unavailable or expensive. Treat it as an analysis metric unless a hardware-feasible estimator is explicitly designed.
Matrix Diagnostic Functions
spectral_norm_np(W)
def spectral_norm_np(W: np.ndarray) -> float:
...
Computes the matrix spectral norm using SVD. If the input contains non-finite values or SVD fails, it returns np.nan rather than crashing the experiment.
Behavior:
| Input | Output |
|---|---|
| finite 2D matrix | largest singular value. |
| finite non-2D array | vector L2 norm after flattening. |
| non-finite array | np.nan. |
| failed SVD | np.nan. |
matrix_norms_np(W)
def matrix_norms_np(W: np.ndarray) -> dict[str, float]:
...
Returns common matrix norms in one dictionary:
{
"fro": ...,
"l1": ...,
"linf": ...,
"spectral": ...,
}
These are the natural diagnostics for future κ/gain studies:
If any value in W is non-finite, every returned norm is np.nan.
Hessian And Stability Functions
analytic_single_dense_hessian(...)
def analytic_single_dense_hessian(
X_batch: np.ndarray,
d_out: int,
keras_mse_scaling: bool = True,
) -> np.ndarray:
...
Analytic Hessian for one Dense layer without bias:
The input covariance is:
For the clean half-MSE loss:
this gives:
If keras_mse_scaling=True, the helper uses the scaling associated with Keras-style mean squared error:
Usage:
H = analytic_single_dense_hessian(X_batch, d_out=2, keras_mse_scaling=False)
metrics = hessian_metrics_np(H)
hessian_metrics_np(H)
def hessian_metrics_np(H: np.ndarray) -> Dict[str, float]:
...
Computes core Hessian metrics after symmetrizing defensively:
{
"hessian_lambda_max": lam_max,
"hessian_lambda_min": lam_min,
"hessian_spectral_norm": h_norm_2,
}
These are used to compare the cheap online curvature proxy against true local curvature in toy experiments.
stability_metrics_from_hessian(H, eta, alpha=1.0)
def stability_metrics_from_hessian(
H: np.ndarray,
eta: float,
alpha: float = 1.0,
) -> Dict[str, float]:
...
Computes local discrete-time stability metrics for the SGD-like update:
The local linearized update map is:
The function returns:
{
"stability_margin_lambda": alpha * eta * lambda_max,
"stability_margin_norm": alpha * eta * hessian_spectral_norm,
"spectral_radius_update_map": rho(I - alpha * eta * H),
}
For a convex quadratic, the familiar gradient-descent stability condition is approximately:
So the scalar margin:
should remain below 2.
Loss Helpers
half_mse_batch_loss(y_true, y_pred)
def half_mse_batch_loss(y_true: tf.Tensor, y_pred: tf.Tensor) -> tf.Tensor:
...
Computes:
This loss is intentionally used in the first ablations because it makes the one-layer Hessian formula clean:
Usage:
with tf.GradientTape() as tape:
y_pred = model(x_batch, training=True)
loss = half_mse_batch_loss(y_batch, y_pred)
Extension Notes
Keep this module boring and reusable. If a helper starts depending on a specific experiment, plot layout, or documentation artifact, it probably belongs in a notebook, history.py, or a future experiment-specific module instead of utils.py.