Skip to main content

๐Ÿ“ฆ Models / NN Module Reference

Status:ValidIn progress
Tags:ImplementationModel
Last modified:
Source:enabol/nn/

What this page covers

This page is a coder-facing reference for the enabol.nn package: the shared BaseModel, the current dense-only LinearBlockModel, quantization hooks, controllers, optimizer-style update rules, telemetry, and the custom instrumented training loop. It also tracks model families from old_enabol/nn.py that have not been ported yet.

Module Mapโ€‹

GroupObjectsPurpose
Model classesBaseModel, LinearBlockModelOwn dataset linkage, Keras model construction, simple training, and the public instrumented-training entry point.
BaseModel public methodssummary, reinitialize_weights, train, train_instrumentedUser-facing model utilities and training entry points.
BaseModel precision helpers_forward_with_precision, _quantize_gradients, _quantize_variable_storage, rail helpersInternal fake-quantization and diagnostic hooks.
ControllersBaseController, GlobalThrottleOrder*, QuantizationAwareOrder2ControllerOwn controller parameters, alpha computation, state, and controller-specific log fields.
Update rulesBaseUpdateRule, SGDUpdateRulePropose raw optimizer deltas before controller intervention.
TelemetryCurvatureSensor, HistoryRecorderCollect tensor-native signals and materialize NumPy histories.
Trainer/apply pathInstrumentedTrainer, UpdateApplierOrchestrate the online loop and apply controller-approved updates.

Model Coverageโ€‹

This table tracks active model classes and legacy model families that are still missing from the current enabol module. Priority is intentionally blank for implemented models.

Model familyClass / sourceStatusPriorityNotes / Documentation
Base model and training harnessBaseModelValidInternal parent class documented below.
Dense linear blocksLinearBlockModel (enabol.nn.models)ValidMDL-DENSE1-LINEAR-NOBIAS-000
Generic MLPMLPModelMissingHighNeeded for two-layer and deeper dense ablations.
Bounded activation layersClippedReLU, ClippedReLUAdaptiveMissingMediumNeeded when activation rails become part of the controller study.
Fusion CNN familyFusionModel, TinyFusionModel, NanoFusionModel, NanoFusionModel16MissingMediumUseful for TinyML image-like regression once dense ablations are stable.
Generic CNN classifierCNNMissingMediumUseful once MNIST/Fashion-MNIST/CIFAR datasets are ported.
Jet tagging MLPJetTaggingModelMissingMediumRelevant to hls4ml/physics workflows.
AutoencodersAEModel, cAEModelMissingLowPort only if reconstruction experiments return.
Model TODOs
  • Port a minimal MLPModel or extend LinearBlockModel enough to support two-layer dense experiments.
  • Decide whether bounded activations belong in enabol.nn.models or a separate enabol.nn.activations module.
  • Add model registry pages only when a model is used by an experiment.

Imports And Dependenciesโ€‹

from abc import ABC
from dataclasses import dataclass, field
from typing import Any, Dict, Mapping, Union, Optional, Tuple

import numpy as np
import tensorflow as tf

Local dependencies:

from ..dataset import BaseDataset
from ..history import FitHistory
from ..precision import PrecisionDict, ensure_precision_dict
from ..quantization import quantize_tensor, rail_stats
from ..utils import ...
from .controller import BaseController, make_controller
from .training import InstrumentationConfig, InstrumentedTrainer

Important utilities:

UtilityPurpose
flatten_tensorsConvert trainable variables or gradients into one global vector.
half_mse_batch_lossLoss used by analytic Hessian experiments.
tensor_l2_normStable tensor norm for logging.
safe_cosineUpdate direction diagnostic.
analytic_single_dense_hessianExact Hessian for one-layer no-bias regression.
stability_metrics_from_hessianComputes eta * lambda_max and update-map spectral radius.

Classesโ€‹

BaseModelโ€‹

@dataclass(eq=False)
class BaseModel(ABC):
dataset: BaseDataset
loss: Union[str, tf.keras.losses.Loss] = "mse"
optimizer: Union[str, tf.keras.optimizers.Optimizer] = "sgd"
metrics: list = field(default_factory=lambda: [])
model: Optional[tf.keras.Model] = field(init=False, default=None)
input_shape: Optional[Tuple[int, ...]] = field(init=False, default=None)
output_shape: Optional[Tuple[int, ...]] = field(init=False, default=None)
name: str = "BaseModel"
verbose: bool = False
seed: Optional[int] = None

BaseModel owns the dataset reference, the Keras model object, and the custom training loop.

Subclasses are responsible for constructing self.model.

__post_init__()โ€‹

def __post_init__(self):
self.input_shape = self.dataset.input_shape[1:]
self.output_shape = self.dataset.output_shape[1:]

Removes the batch dimension from dataset shapes.

Example:

dataset.X.shape == (1000, 4)
model.input_shape == (4,)

_compile(...)โ€‹

def _compile(self, optimizer=None, loss=None, metrics=None, **kwargs):
...

Thin wrapper over self.model.compile(...).

Used by the simple train() method. The instrumented ablation loop does not use Keras optimizer application because it needs direct control over every update.

summary()โ€‹

def summary(self) -> None:
...

Prints Keras model summary if the model exists.

Usage:

model.summary()

reinitialize_weights()โ€‹

def reinitialize_weights(self):
...

Loops over Keras layers and reassigns kernels/biases using the layer initializers.

Usage:

model.reinitialize_weights()

Used before comparing baseline and controller runs so they start from comparable initial conditions.

train(...)โ€‹

def train(self, X, Y, epochs=10, batch_size=32) -> dict[str, np.ndarray]:
...

Simple Keras-style training loop:

  1. Builds a tf.data.Dataset.
  2. Calls _compile().
  3. Uses the compiled optimizer and loss.
  4. Applies gradients with optimizer.apply_gradients.
  5. Returns only loss history.

This is mostly a convenience method. Ablation work should use train_instrumented().

train_instrumented(...)โ€‹

def train_instrumented(
self,
X,
Y,
epochs=10,
batch_size=32,
learning_rate=0.05,
shuffle=True,
loss_mode="half_mse",
controller=None,
metrics=None,
precision_dict=None,
) -> FitHistory:
...

This is the main ablation loop.

Inputsโ€‹
ArgumentPurpose
X, YNumPy training arrays.
learning_rateBase SGD learning rate eta.
loss_mode"half_mse" for Hessian-clean experiments or "keras_mse".
controllerController object or registry string. None uses NoController. Controller-specific fields such as chi, eps, and curvature_ema_rho live on the controller object.
metricsOptional MetricsConfig. None preserves the legacy full metric list; MetricsConfig() records the compact core profile. Metric-only settings such as reference_A and analytic Hessian logging live here.
precision_dictOptional PrecisionDict; None means full floating point.
Training Loop Structureโ€‹

The instrumented loop is easiest to understand as a top-level online training loop with clickable internal blocks. The explorer starts at train_instrumented(...); click a block to inspect its local flowchart and pseudo-code, then use the back arrow to return to the parent loop.

train_instrumented(...)top-level online training loop
false
next batch
Start
More batches?epoch / batch loop
true
Return FitHistory
Algorithm: train_instrumented(...)
  1. input data X,YX,Y, learning rate ฮท\eta, controller settings, precision map PP
  2. Pโ†ensure_precision_dictโก(precision_dict)P \leftarrow \operatorname{ensure\_precision\_dict}(\texttt{precision\_dict})
  3. validate_modelโก(P,modelโก)\operatorname{validate\_model}(P,\operatorname{model})
  4. Dโ†batchโก(Datasetโก(X,Y))D \leftarrow \operatorname{batch}(\operatorname{Dataset}(X,Y))
  5. for each epoch and batch do
  6. ฮธtโ†flattenโก(trainableย variablesโก)\theta_t \leftarrow \operatorname{flatten}(\operatorname{trainable\ variables})
  7. y^t,Lt,Gtโ†ForwardGradientBlockโก(batchโก,P)\hat{y}_t,L_t,G_t \leftarrow \operatorname{ForwardGradientBlock}(\operatorname{batch},P)
  8. Ct,Stโ†CurvatureProxyBlockโก(ฮธt,Gt)C_t,S_t \leftarrow \operatorname{CurvatureProxyBlock}(\theta_t,G_t)
  9. ฮฑt,ฮทteffโ†ControllerBlockโก(Ct,St,ฯ‡,ฮท)\alpha_t,\eta^{\mathrm{eff}}_t \leftarrow \operatorname{ControllerBlock}(C_t,S_t,\chi,\eta)
  10. ฮธt+1โ†UpdateStorageBlockโก(ฮธt,Gt,ฮทteff,P)\theta_{t+1} \leftarrow \operatorname{UpdateStorageBlock}(\theta_t,G_t,\eta^{\mathrm{eff}}_t,P)
  11. MetricsBlockโก(โ€ฆ)\operatorname{MetricsBlock}(\ldots)
  12. ฮธtโˆ’1,Gtโˆ’1โ†ฮธt,Gt\theta_{t-1},G_{t-1} \leftarrow \theta_t,G_t
  13. end for
  14. return FitHistoryโก(H)\operatorname{FitHistory}(H)
The top-level loop shows where each internal block is called. Click any block in the diagram to inspect it.
Metric Blockโ€‹

The loop returns a FitHistory whose keys are selected by MetricsConfig. The exact set of populated metrics depends on the enabled profiles, reference_A, analytic Hessian logging, and precision_dict.

MetricSymbol / expressionWhat it measuresNotes
lossLtL_tCurrent batch loss.Usually half_mse in clean Hessian experiments.
rmseMSEโก\sqrt{\operatorname{MSE}}Output error scale.Easier to read than raw loss.
theta_normโˆฅฮธtโˆฅ2\lVert \theta_t \rVert_2Global parameter magnitude.Detects growth or collapse.
grad_normโˆฅGtโˆฅ2\lVert G_t \rVert_2Global gradient magnitude.Detects exploding or vanishing gradients.
raw_update_normโˆฅฮ”ฮธrawโˆฅ2\lVert \Delta\theta_{\mathrm{raw}} \rVert_2Intended update magnitude before storage effects.Uses the throttled raw update direction.
actual_update_normโˆฅฮธt+1โˆ’ฮธtโˆฅ2\lVert \theta_{t+1}-\theta_t \rVert_2Applied update magnitude after quantization/storage.Detects silent update underflow.
update_cosinecosโก(ฮ”ฮธactual,ฮ”ฮธraw)\cos(\Delta\theta_{\mathrm{actual}},\Delta\theta_{\mathrm{raw}})Direction preservation.Near 1 means little update rotation.
update_angle_radฮฒt=arccosโก(cosโกt)\beta_t=\arccos(\cos_t)Angular distortion in radians.Used by phase-style plots.
update_radius_ratiort=โˆฅฮ”ฮธactualโˆฅ2/(โˆฅฮ”ฮธrawโˆฅ2+ฮต)r_t=\lVert\Delta\theta_{\mathrm{actual}}\rVert_2/(\lVert\Delta\theta_{\mathrm{raw}}\rVert_2+\varepsilon)Applied-vs-intended update radius.Values near 0 indicate update death.
curvature_proxyCt=โˆฅGtโˆ’Gtโˆ’1โˆฅ/(โˆฅฮธtโˆ’ฮธtโˆ’1โˆฅ+ฮต)C_t=\lVert G_t-G_{t-1}\rVert/(\lVert\theta_t-\theta_{t-1}\rVert+\varepsilon)Online curvature/update-field sensitivity proxy.Controller input candidate.
curvature_emaStS_tSmoothed curvature proxy.Reduces step-to-step noise.
alphaฮฑt\alpha_tApplied global throttle.1 in baseline mode.
alpha_wouldฮฑtwould\alpha^{\mathrm{would}}_tController value that would be applied.Logged even when controller is disabled.
eta_effฮทteff=ฮฑtฮท\eta^{\mathrm{eff}}_t=\alpha_t\etaEffective learning rate.Compare against stability bounds.
forward_gain_spectralapproximate โˆlโˆฅWlโˆฅ2\prod_l \lVert W_l\rVert_2Forward gain proxy.Static gain diagnostic, not the closed-loop condition.
hessian_lambda_maxฮปmaxโก(Ht)\lambda_{\max}(H_t)True local curvature when available.Exact only for supported toy cases.
stability_margin_lambda_rawฮทฮปmaxโก(Ht)\eta\lambda_{\max}(H_t)Raw SGD stability margin.Quadratic stability boundary is near 2.
stability_margin_lambda_ctrlฮฑtฮทฮปmaxโก(Ht)\alpha_t\eta\lambda_{\max}(H_t)Throttled stability margin.Should stay below the boundary in stable controller runs.
spectral_radius_rawฯ(Iโˆ’ฮทHt)\rho(I-\eta H_t)Raw local update-map spectral radius.Stable if below 1 in the quadratic local model.
spectral_radius_ctrlฯ(Iโˆ’ฮฑtฮทHt)\rho(I-\alpha_t\eta H_t)Throttled local update-map spectral radius.Controller should reduce this when curvature rises.
weight_error_froโˆฅWtโˆ’AโˆฅF\lVert W_t-A\rVert_FTeacher-weight error.Only meaningful when reference_A is supplied.
finite/divergence flagsboolean indicatorsNumerical run health.Used to stop or label unstable trajectories.

With quantization enabled, the loop also logs rail-pressure maxima:

MetricWhat it measuresInterpretation
weight_saturation_fraction_maxMax fraction of weights at fixed-point rails.High values mean storage rails are active.
weight_near_rail_fraction_maxMax fraction of weights near rails.Early warning before saturation.
gradient_saturation_fraction_maxMax fraction of gradients at rails.Indicates gradient clipping by dtype.
gradient_near_rail_fraction_maxMax fraction of gradients near rails.Early warning for gradient rail pressure.
update_saturation_fraction_maxMax fraction of updates at rails.Indicates update dtype is too tight or learning rate too high.
update_underflow_fraction_maxMax fraction of nonzero updates below half a quantum.Indicates learning may silently die.
Update Geometry Diagnosticsโ€‹

Two metrics are especially important for the quantized global-throttle ablation:

actual_update_norm
update_cosine
update_angle_rad
update_radius_ratio

raw_update_norm is the norm of the intended update before storage effects:

ฮ”ฮธraw=โˆ’ฮฑtฮทGt.\Delta\theta_{\mathrm{raw}} = -\alpha_t\eta G_t.

actual_update_norm is computed after the update has been applied and after any quantized variable storage has been enforced:

ฮ”ฮธactual=ฮธt+1โˆ’ฮธt.\Delta\theta_{\mathrm{actual}} = \theta_{t+1}-\theta_t.

This catches silent learning death. A run can look numerically stable because the applied update has underflowed to zero; that is not a successful controller result.

update_cosine compares the applied update to the intended update:

cosโกt=โŸจฮ”ฮธactual,ฮ”ฮธrawโŸฉโˆฅฮ”ฮธactualโˆฅ2โˆฅฮ”ฮธrawโˆฅ2+ฮต.\cos_t = \frac{ \left\langle \Delta\theta_{\mathrm{actual}}, \Delta\theta_{\mathrm{raw}} \right\rangle }{ \left\|\Delta\theta_{\mathrm{actual}}\right\|_2 \left\|\Delta\theta_{\mathrm{raw}}\right\|_2 +\varepsilon }.

For pure global throttling, this should stay close to 1 because the throttle scales the full update vector uniformly. If quantization, clipping, or a future row/column projection changes the update direction, this cosine should fall.

The phase-distortion diagnostics are derived from the same quantities:

ฮฒt=arccosโก(cosโกt)\beta_t = \arccos(\cos_t) rt=โˆฅฮ”ฮธactualโˆฅ2โˆฅฮ”ฮธrawโˆฅ2+ฮต.r_t = \frac{ \left\|\Delta\theta_{\mathrm{actual}}\right\|_2 }{ \left\|\Delta\theta_{\mathrm{raw}}\right\|_2 +\varepsilon }.

update_angle_rad stores beta_t; update_radius_ratio stores r_t.

These are offline software diagnostics. They are meant to visualize and compare numerical update distortion during ablations. They are not currently proposed as hardware controller inputs because the high-precision raw update may not exist as a physical hardware signal.

Return Valueโ€‹
return FitHistory(**{k: np.asarray(v) for k, v in history.items()})

Use:

h = model.train_instrumented(...)
h.plot_results()

_forward_with_precision(...)โ€‹

def _forward_with_precision(self, x, precision, *, training):
...

Manual forward path with fake quantization hooks.

Current behavior:

  1. Quantize input with precision.dtype("input", "value").
  2. For each Dense layer:
    • quantize kernel as "weight",
    • matrix multiply,
    • quantize dot-product as "accumulator",
    • add quantized bias if present,
    • quantize layer output as "activation".
  3. For non-Dense layers:
    • call layer normally,
    • quantize output as "activation".

The quantizers use STE inside GradientTape, so the forward value is quantized but gradients still flow.

Extension point:

If Conv layers are added, implement their explicit forward path here.

_quantize_gradients(...)โ€‹

def _quantize_gradients(self, grads, trainable_vars, precision):
...

For each gradient:

  1. Find the owning layer with _layer_and_field_for_variable.
  2. Lookup precision.dtype(layer_name, "gradient").
  3. Quantize the gradient if a dtype exists.

Missing gradient dtype means floating point gradient.

_quantize_variable_storage(...)โ€‹

def _quantize_variable_storage(self, var, precision):
...

After the update is applied, this quantizes the stored variable:

  • Dense kernel uses "weight",
  • Dense bias uses "bias",
  • other trainable variables use "value".

This models fixed-point storage.

_layer_and_field_for_variable(...)โ€‹

def _layer_and_field_for_variable(self, var) -> tuple[str, str]:
...

Maps a Keras variable to:

(layer_name, field_name)

Examples:

dense0/kernel -> ("dense0", "weight")
dense0/bias -> ("dense0", "bias")

This function is central to PrecisionDict integration. If a new layer type has special trainable variables, update this mapping.

_same_variable(a, b)โ€‹

@staticmethod
def _same_variable(a, b) -> bool:
...

Defensive Keras variable comparison helper. It first checks identity, then tries path, then falls back to name.

This avoids fragile behavior across Keras/TensorFlow versions.

_rail_max_for_variables(...)โ€‹

def _rail_max_for_variables(self, vars_, precision, *, fields):
...

Computes max saturation and near-rail fractions across trainable variables for selected fields.

Used for:

weight_saturation_fraction_max
weight_near_rail_fraction_max

_rail_max_for_tensors(...)โ€‹

def _rail_max_for_tensors(self, tensors, trainable_vars, precision, *, field):
...

Computes max rail pressure across non-variable tensors, currently gradients.

Used for:

gradient_saturation_fraction_max
gradient_near_rail_fraction_max

LinearBlockModelโ€‹

@dataclass
class LinearBlockModel(BaseModel):
num_hidden: list = field(default_factory=lambda: [64, 64])
activation: Optional[Union[str, tf.keras.layers.Activation]] = None
use_batchnorm: bool = False
use_bias: bool = True
name: str = "LinearBlockModel"

Dense-block model used by the first ablations.

Each block is:

Dense -> optional Activation -> optional BatchNorm

__post_init__()โ€‹

def __post_init__(self):
super().__post_init__()
self.model = self._build_model(...)

Builds the Keras model immediately after dataclass initialization.

_build_model(input_shape, output_shape, verbose=True)โ€‹

def _build_model(self, input_shape, output_shape, verbose=True) -> tf.keras.Model:
...

Build process:

  1. Create Keras input named model_input.
  2. Add Dense blocks using stable names:
dense0
dense1
...
  1. Add optional activations:
activation0
activation1
...
  1. Add optional batchnorm:
batchnorm0
batchnorm1
...
  1. If the last Dense output dimension does not match the dataset output, add a final Dense layer.

Usage:

model = enabol.LinearBlockModel(
dataset=dataset,
num_hidden=[8, 2],
activation="relu",
use_batchnorm=False,
use_bias=True,
)

Extension Notesโ€‹

When adding a new model class:

  1. Subclass BaseModel.
  2. Build self.model in __post_init__().
  3. Use stable layer names.
  4. Avoid reserved layer names input and loss.
  5. If the model has non-Dense trainable variables, update _layer_and_field_for_variable.
  6. If fake quantization must happen inside special ops, update _forward_with_precision.

When adding a new optimizer:

  1. Keep the flattened global update available for logging.
  2. Log intended update and actual applied update.
  3. Keep update_cosine meaningful.
  4. Add precision fields for optimizer state, such as momentum, adam_m, or adam_v.