Skip to main content

๐Ÿงญ Trainable Precision Schema

Status:ValidIn progress
Tags:ImplementationPrecisionHls4mlTrainable
Last modified:

What this page covers

This page records the precision relationships needed by ENABOL and hls4ml-trainable. It is intentionally incomplete. Unknown relationships are marked with ? so we can fill them in as controller, optimizer, and layer rules become clear.

Purposeโ€‹

PrecisionDict is the user-facing description of hardware data types. It should stay compact in notebooks, but the compiler eventually needs a richer trainable schema:

input / activation / weight / bias / loss
gradient / accumulator / update / optimizer state / controller metric

The goal is not to force users to specify every internal type forever. The goal is to identify which type can be inferred from which base type, and which type must be explicitly supplied until ENABOL has a reliable inference rule.

Current Compile Behaviorโ€‹

Today, ENABOL accepts compact semantic fields such as:

PrecisionDict({
"input": {"value": wide_activation},
"dense0": {
"weight": wide_weight,
"activation": wide_activation,
"gradient": wide_gradient,
"update": wide_update,
"accumulator": wide_accumulator,
},
"loss": {"value": wide_loss},
})

The bridge expands aliases:

User fieldCurrent hls4ml-trainable expansion
gradientgrad_in, grad_out, weight_grad, bias_grad, loss_grad
updateraw_update, update, optimizer_state
accumulatorgradient_accum, controller_metric
loss.valueloss

That last alias is useful for bring-up but is not mathematically correct as a long-term policy. controller_metric has different range and fractional-bit requirements from gradient_accum.

Tensor Familiesโ€‹

For a Dense layer, the trainable path currently uses these tensor families:

FamilySymbolMeaningCurrent sourceInference status
Input / activationxLayer input activation.input.value or layer activation?
Output activationyLayer output activation / forward result.layer activation / result?
WeightWStored trainable weight.layer weightBase candidate
BiasbStored trainable bias.layer bias, or hls4ml default?
Loss valueLScalar loss.loss.value?
Loss seeddL/dyLoss endpoint gradient seed.gradient / loss_grad?
Backprop input gradientdL/dxGradient sent to previous layer.gradient / grad_out?
Weight gradientG_WAveraged dL/dW at batch end.gradient / weight_grad?
Bias gradientG_bAveraged dL/db at batch end.gradient / bias_grad?
Gradient accumulatorsum(G)Batch accumulator before shift average.accumulator / gradient_accum?
Raw optimizer updateDelta theta_rawOptimizer-proposed update before alpha.update / raw_update?
Applied updateDelta thetaAlpha-scaled update before cast into stored parameters.update?
Optimizer states_optMomentum/Adam/etc. state, currently SGD placeholder.update / optimizer_state?
Controller alphaalphaGlobal scalar throttle.alpha or default?
Controller metricm_ctrlSquared norms and inequality terms used by controller laws.controller_metric, currently aliased from accumulator if omittedKnown partial rule below

Known Rule: Parameter Precision To Actual Update Telemetryโ€‹

The controller metric stores squared geometry. For GT-0 and GT-1, the controller law now uses raw optimizer update geometry:

raw_update_norm_sq = ||Delta theta_raw||^2
dgrad_norm_sq = ||G_t - G_{t-1}||^2
stability_lhs_raw = learning_rate^2 * dgrad_norm_sq
stability_lhs_ctrl = alpha^2 * stability_lhs_raw
stability_rhs = chi^2 * (raw_update_norm_sq + epsilon^2)
actual_update_norm_sq = ||theta_after - theta_before||^2

For SGD, raw_update_norm_sq already contains the learning rate because the raw update is -learning_rate * gradient. The separate actual_update_norm_sq telemetry measures the movement after alpha and fixed-point assignment. These squared quantities mean controller_metric cannot be copied blindly from gradient_accum, and it usually needs more fractional bits than the stored weight.

Let a stored parameter type have:

WL_theta = total word length
IWL_theta = integer word length
F_theta = WL_theta - IWL_theta
q_theta = 2^-F_theta

The smallest nonzero one-parameter movement is approximately q_theta, so the smallest one-term squared movement is:

q_theta^2 = 2^(-2 * F_theta)

Therefore, if we want actual movement telemetry to resolve one stored-parameter LSB, controller_metric must have enough fractional bits to represent at least that squared movement:

F_controller_metric >= 2 * F_theta + guard_bits

For bring-up, use a few guard bits because:

  • several products are evaluated in the controller law,
  • learning_rate^2 * dgrad_sq can be smaller than dgrad_sq,
  • epsilon^2 may be tiny,
  • small models have few terms in the norm sum, so summation may not rescue underflow.

Example:

weight_t = ap_fixed<16,6>
F_theta = 10
q_theta^2 = 2^-20

A controller metric with only 14 fractional bits cannot represent one squared weight LSB:

ap_fixed<28,14> -> F = 14 -> LSB = 2^-14
2^-20 < 2^-14 -> dtheta_sq can quantize to zero

For this case, a bring-up type like the following is more appropriate:

controller_metric: ap_fixed<48,16,AP_RND,AP_SAT>

This gives 32 fractional bits and enough integer range for the current small Dense tests.

Relationship Tableโ€‹

This table is the starting point for the final precision inference pass.

Target typeDepends onRule
activation / resultinput range, weight range, fan-in, activation function?
accuminput precision, weight precision, fan-in?
lossprediction precision, target precision, loss kind, output dimension?
loss_gradprediction precision, target precision, loss kind?
weight_gradinput precision, loss/next gradient precision, batch size?
bias_gradloss/next gradient precision, batch size?
gradient_accumunaveraged gradient precision, batch size?
raw_updateoptimizer kind, learning rate, gradient precision, optimizer state?
updateraw update precision, alpha precision, stored parameter precision?
optimizer_stateoptimizer kind, raw update precision, gradient precision?
alphacontroller kind, controller rails, smoothing rule?
controller_metricstored parameter precision, gradient precision, learning rate, controller lawF_controller_metric >= 2 * F_theta + guard_bits for actual update telemetry; raw-update controller law also depends on update/gradient precision

Notebook Guidance For Nowโ€‹

Until ENABOL implements automatic trainable precision inference, specify controller_metric explicitly when using global throttling:

wide_controller_metric = dtypes.ap_fixed(WL=48, IWL=16, QMODE="AP_RND", OMODE="AP_SAT")

precision_dict = PrecisionDict({
"input": {"value": wide_activation},
"dense0": {
"weight": wide_weight,
"activation": wide_activation,
"gradient": wide_gradient,
"update": wide_update,
"accumulator": wide_accumulator,
"controller_metric": wide_controller_metric,
},
"loss": {"value": wide_loss},
})

If controller_metric is omitted today, ENABOL may still alias it from accumulator. That is acceptable only for smoke tests where the controller trace is not being interpreted numerically.

Implementation Directionโ€‹

The target behavior is:

  1. User provides a compact PrecisionDict.
  2. ENABOL expands semantic fields into hls4ml-trainable fields.
  3. ENABOL applies explicit user overrides first.
  4. ENABOL infers missing internal fields from known rules.
  5. ENABOL warns when a fallback alias is used for a field with known non-equivalent requirements.

This belongs to ENB-023: infer hls4ml trainable precisions from ENABOL model, dataset, optimizer, and controller.

Change Logโ€‹

TaskDateFilesSummary
ENB-0282026-05-23site/docs/implementation/precision-schema.mdAdded the first trainable precision relationship document and recorded the stored-parameter movement telemetry to controller_metric fractional-bit rule.