๐ฆ Models / NN Module Reference
This page is a coder-facing reference for the enabol.nn package: the shared BaseModel, the current dense-only LinearBlockModel, quantization hooks, controllers, optimizer-style update rules, telemetry, and the custom instrumented training loop. It also tracks model families from old_enabol/nn.py that have not been ported yet.
Module Mapโ
| Group | Objects | Purpose |
|---|---|---|
| Model classes | BaseModel, LinearBlockModel | Own dataset linkage, Keras model construction, simple training, and the public instrumented-training entry point. |
| BaseModel public methods | summary, reinitialize_weights, train, train_instrumented | User-facing model utilities and training entry points. |
| BaseModel precision helpers | _forward_with_precision, _quantize_gradients, _quantize_variable_storage, rail helpers | Internal fake-quantization and diagnostic hooks. |
| Controllers | BaseController, GlobalThrottleOrder*, QuantizationAwareOrder2Controller | Own controller parameters, alpha computation, state, and controller-specific log fields. |
| Update rules | BaseUpdateRule, SGDUpdateRule | Propose raw optimizer deltas before controller intervention. |
| Telemetry | CurvatureSensor, HistoryRecorder | Collect tensor-native signals and materialize NumPy histories. |
| Trainer/apply path | InstrumentedTrainer, UpdateApplier | Orchestrate the online loop and apply controller-approved updates. |
Model Coverageโ
This table tracks active model classes and legacy model families that are still missing from the current enabol module. Priority is intentionally blank for implemented models.
| Model family | Class / source | Status | Priority | Notes / Documentation |
|---|---|---|---|---|
| Base model and training harness | BaseModel | Valid | Internal parent class documented below. | |
| Dense linear blocks | LinearBlockModel (enabol.nn.models) | Valid | MDL-DENSE1-LINEAR-NOBIAS-000 | |
| Generic MLP | MLPModel | Missing | High | Needed for two-layer and deeper dense ablations. |
| Bounded activation layers | ClippedReLU, ClippedReLUAdaptive | Missing | Medium | Needed when activation rails become part of the controller study. |
| Fusion CNN family | FusionModel, TinyFusionModel, NanoFusionModel, NanoFusionModel16 | Missing | Medium | Useful for TinyML image-like regression once dense ablations are stable. |
| Generic CNN classifier | CNN | Missing | Medium | Useful once MNIST/Fashion-MNIST/CIFAR datasets are ported. |
| Jet tagging MLP | JetTaggingModel | Missing | Medium | Relevant to hls4ml/physics workflows. |
| Autoencoders | AEModel, cAEModel | Missing | Low | Port only if reconstruction experiments return. |
- Port a minimal
MLPModelor extendLinearBlockModelenough to support two-layer dense experiments. - Decide whether bounded activations belong in
enabol.nn.modelsor a separateenabol.nn.activationsmodule. - Add model registry pages only when a model is used by an experiment.
Imports And Dependenciesโ
from abc import ABC
from dataclasses import dataclass, field
from typing import Any, Dict, Mapping, Union, Optional, Tuple
import numpy as np
import tensorflow as tf
Local dependencies:
from ..dataset import BaseDataset
from ..history import FitHistory
from ..precision import PrecisionDict, ensure_precision_dict
from ..quantization import quantize_tensor, rail_stats
from ..utils import ...
from .controller import BaseController, make_controller
from .training import InstrumentationConfig, InstrumentedTrainer
Important utilities:
| Utility | Purpose |
|---|---|
flatten_tensors | Convert trainable variables or gradients into one global vector. |
half_mse_batch_loss | Loss used by analytic Hessian experiments. |
tensor_l2_norm | Stable tensor norm for logging. |
safe_cosine | Update direction diagnostic. |
analytic_single_dense_hessian | Exact Hessian for one-layer no-bias regression. |
stability_metrics_from_hessian | Computes eta * lambda_max and update-map spectral radius. |
Classesโ
BaseModelโ
@dataclass(eq=False)
class BaseModel(ABC):
dataset: BaseDataset
loss: Union[str, tf.keras.losses.Loss] = "mse"
optimizer: Union[str, tf.keras.optimizers.Optimizer] = "sgd"
metrics: list = field(default_factory=lambda: [])
model: Optional[tf.keras.Model] = field(init=False, default=None)
input_shape: Optional[Tuple[int, ...]] = field(init=False, default=None)
output_shape: Optional[Tuple[int, ...]] = field(init=False, default=None)
name: str = "BaseModel"
verbose: bool = False
seed: Optional[int] = None
BaseModel owns the dataset reference, the Keras model object, and the custom training loop.
Subclasses are responsible for constructing self.model.
__post_init__()โ
def __post_init__(self):
self.input_shape = self.dataset.input_shape[1:]
self.output_shape = self.dataset.output_shape[1:]
Removes the batch dimension from dataset shapes.
Example:
dataset.X.shape == (1000, 4)
model.input_shape == (4,)
_compile(...)โ
def _compile(self, optimizer=None, loss=None, metrics=None, **kwargs):
...
Thin wrapper over self.model.compile(...).
Used by the simple train() method. The instrumented ablation loop does not use Keras optimizer application because it needs direct control over every update.
summary()โ
def summary(self) -> None:
...
Prints Keras model summary if the model exists.
Usage:
model.summary()
reinitialize_weights()โ
def reinitialize_weights(self):
...
Loops over Keras layers and reassigns kernels/biases using the layer initializers.
Usage:
model.reinitialize_weights()
Used before comparing baseline and controller runs so they start from comparable initial conditions.
train(...)โ
def train(self, X, Y, epochs=10, batch_size=32) -> dict[str, np.ndarray]:
...
Simple Keras-style training loop:
- Builds a
tf.data.Dataset. - Calls
_compile(). - Uses the compiled optimizer and loss.
- Applies gradients with
optimizer.apply_gradients. - Returns only loss history.
This is mostly a convenience method. Ablation work should use train_instrumented().
train_instrumented(...)โ
def train_instrumented(
self,
X,
Y,
epochs=10,
batch_size=32,
learning_rate=0.05,
shuffle=True,
loss_mode="half_mse",
controller=None,
metrics=None,
precision_dict=None,
) -> FitHistory:
...
This is the main ablation loop.
Inputsโ
| Argument | Purpose |
|---|---|
X, Y | NumPy training arrays. |
learning_rate | Base SGD learning rate eta. |
loss_mode | "half_mse" for Hessian-clean experiments or "keras_mse". |
controller | Controller object or registry string. None uses NoController. Controller-specific fields such as chi, eps, and curvature_ema_rho live on the controller object. |
metrics | Optional MetricsConfig. None preserves the legacy full metric list; MetricsConfig() records the compact core profile. Metric-only settings such as reference_A and analytic Hessian logging live here. |
precision_dict | Optional PrecisionDict; None means full floating point. |
Training Loop Structureโ
The instrumented loop is easiest to understand as a top-level online training loop with clickable internal blocks. The explorer starts at train_instrumented(...); click a block to inspect its local flowchart and pseudo-code, then use the back arrow to return to the parent loop.
- input data , learning rate , controller settings, precision map
- for each epoch and batch do
- end for
- return
Metric Blockโ
The loop returns a FitHistory whose keys are selected by MetricsConfig. The exact set of populated metrics depends on the enabled profiles, reference_A, analytic Hessian logging, and precision_dict.
| Metric | Symbol / expression | What it measures | Notes |
|---|---|---|---|
loss | Current batch loss. | Usually half_mse in clean Hessian experiments. | |
rmse | Output error scale. | Easier to read than raw loss. | |
theta_norm | Global parameter magnitude. | Detects growth or collapse. | |
grad_norm | Global gradient magnitude. | Detects exploding or vanishing gradients. | |
raw_update_norm | Intended update magnitude before storage effects. | Uses the throttled raw update direction. | |
actual_update_norm | Applied update magnitude after quantization/storage. | Detects silent update underflow. | |
update_cosine | Direction preservation. | Near 1 means little update rotation. | |
update_angle_rad | Angular distortion in radians. | Used by phase-style plots. | |
update_radius_ratio | Applied-vs-intended update radius. | Values near 0 indicate update death. | |
curvature_proxy | Online curvature/update-field sensitivity proxy. | Controller input candidate. | |
curvature_ema | Smoothed curvature proxy. | Reduces step-to-step noise. | |
alpha | Applied global throttle. | 1 in baseline mode. | |
alpha_would | Controller value that would be applied. | Logged even when controller is disabled. | |
eta_eff | Effective learning rate. | Compare against stability bounds. | |
forward_gain_spectral | approximate | Forward gain proxy. | Static gain diagnostic, not the closed-loop condition. |
hessian_lambda_max | True local curvature when available. | Exact only for supported toy cases. | |
stability_margin_lambda_raw | Raw SGD stability margin. | Quadratic stability boundary is near 2. | |
stability_margin_lambda_ctrl | Throttled stability margin. | Should stay below the boundary in stable controller runs. | |
spectral_radius_raw | Raw local update-map spectral radius. | Stable if below 1 in the quadratic local model. | |
spectral_radius_ctrl | Throttled local update-map spectral radius. | Controller should reduce this when curvature rises. | |
weight_error_fro | Teacher-weight error. | Only meaningful when reference_A is supplied. | |
| finite/divergence flags | boolean indicators | Numerical run health. | Used to stop or label unstable trajectories. |
With quantization enabled, the loop also logs rail-pressure maxima:
| Metric | What it measures | Interpretation |
|---|---|---|
weight_saturation_fraction_max | Max fraction of weights at fixed-point rails. | High values mean storage rails are active. |
weight_near_rail_fraction_max | Max fraction of weights near rails. | Early warning before saturation. |
gradient_saturation_fraction_max | Max fraction of gradients at rails. | Indicates gradient clipping by dtype. |
gradient_near_rail_fraction_max | Max fraction of gradients near rails. | Early warning for gradient rail pressure. |
update_saturation_fraction_max | Max fraction of updates at rails. | Indicates update dtype is too tight or learning rate too high. |
update_underflow_fraction_max | Max fraction of nonzero updates below half a quantum. | Indicates learning may silently die. |
Update Geometry Diagnosticsโ
Two metrics are especially important for the quantized global-throttle ablation:
actual_update_norm
update_cosine
update_angle_rad
update_radius_ratio
raw_update_norm is the norm of the intended update before storage effects:
actual_update_norm is computed after the update has been applied and after any quantized variable storage has been enforced:
This catches silent learning death. A run can look numerically stable because the applied update has underflowed to zero; that is not a successful controller result.
update_cosine compares the applied update to the intended update:
For pure global throttling, this should stay close to 1 because the throttle scales the full update vector uniformly. If quantization, clipping, or a future row/column projection changes the update direction, this cosine should fall.
The phase-distortion diagnostics are derived from the same quantities:
update_angle_rad stores beta_t; update_radius_ratio stores r_t.
These are offline software diagnostics. They are meant to visualize and compare numerical update distortion during ablations. They are not currently proposed as hardware controller inputs because the high-precision raw update may not exist as a physical hardware signal.
Return Valueโ
return FitHistory(**{k: np.asarray(v) for k, v in history.items()})
Use:
h = model.train_instrumented(...)
h.plot_results()
_forward_with_precision(...)โ
def _forward_with_precision(self, x, precision, *, training):
...
Manual forward path with fake quantization hooks.
Current behavior:
- Quantize input with
precision.dtype("input", "value"). - For each Dense layer:
- quantize kernel as
"weight", - matrix multiply,
- quantize dot-product as
"accumulator", - add quantized bias if present,
- quantize layer output as
"activation".
- quantize kernel as
- For non-Dense layers:
- call layer normally,
- quantize output as
"activation".
The quantizers use STE inside GradientTape, so the forward value is quantized but gradients still flow.
Extension point:
If Conv layers are added, implement their explicit forward path here.
_quantize_gradients(...)โ
def _quantize_gradients(self, grads, trainable_vars, precision):
...
For each gradient:
- Find the owning layer with
_layer_and_field_for_variable. - Lookup
precision.dtype(layer_name, "gradient"). - Quantize the gradient if a dtype exists.
Missing gradient dtype means floating point gradient.
_quantize_variable_storage(...)โ
def _quantize_variable_storage(self, var, precision):
...
After the update is applied, this quantizes the stored variable:
- Dense kernel uses
"weight", - Dense bias uses
"bias", - other trainable variables use
"value".
This models fixed-point storage.
_layer_and_field_for_variable(...)โ
def _layer_and_field_for_variable(self, var) -> tuple[str, str]:
...
Maps a Keras variable to:
(layer_name, field_name)
Examples:
dense0/kernel -> ("dense0", "weight")
dense0/bias -> ("dense0", "bias")
This function is central to PrecisionDict integration. If a new layer type has special trainable variables, update this mapping.
_same_variable(a, b)โ
@staticmethod
def _same_variable(a, b) -> bool:
...
Defensive Keras variable comparison helper. It first checks identity, then tries path, then falls back to name.
This avoids fragile behavior across Keras/TensorFlow versions.
_rail_max_for_variables(...)โ
def _rail_max_for_variables(self, vars_, precision, *, fields):
...
Computes max saturation and near-rail fractions across trainable variables for selected fields.
Used for:
weight_saturation_fraction_max
weight_near_rail_fraction_max
_rail_max_for_tensors(...)โ
def _rail_max_for_tensors(self, tensors, trainable_vars, precision, *, field):
...
Computes max rail pressure across non-variable tensors, currently gradients.
Used for:
gradient_saturation_fraction_max
gradient_near_rail_fraction_max
LinearBlockModelโ
@dataclass
class LinearBlockModel(BaseModel):
num_hidden: list = field(default_factory=lambda: [64, 64])
activation: Optional[Union[str, tf.keras.layers.Activation]] = None
use_batchnorm: bool = False
use_bias: bool = True
name: str = "LinearBlockModel"
Dense-block model used by the first ablations.
Each block is:
Dense -> optional Activation -> optional BatchNorm
__post_init__()โ
def __post_init__(self):
super().__post_init__()
self.model = self._build_model(...)
Builds the Keras model immediately after dataclass initialization.
_build_model(input_shape, output_shape, verbose=True)โ
def _build_model(self, input_shape, output_shape, verbose=True) -> tf.keras.Model:
...
Build process:
- Create Keras input named
model_input. - Add Dense blocks using stable names:
dense0
dense1
...
- Add optional activations:
activation0
activation1
...
- Add optional batchnorm:
batchnorm0
batchnorm1
...
- If the last Dense output dimension does not match the dataset output, add a final Dense layer.
Usage:
model = enabol.LinearBlockModel(
dataset=dataset,
num_hidden=[8, 2],
activation="relu",
use_batchnorm=False,
use_bias=True,
)
Extension Notesโ
When adding a new model class:
- Subclass
BaseModel. - Build
self.modelin__post_init__(). - Use stable layer names.
- Avoid reserved layer names
inputandloss. - If the model has non-Dense trainable variables, update
_layer_and_field_for_variable. - If fake quantization must happen inside special ops, update
_forward_with_precision.
When adding a new optimizer:
- Keep the flattened global update available for logging.
- Log intended update and actual applied update.
- Keep
update_cosinemeaningful. - Add precision fields for optimizer state, such as
momentum,adam_m, oradam_v.