Skip to main content

๐Ÿงฎ Quantization Module Reference

Status:ValidIn progress
Tags:ImplementationQuantizationPrecision
Last modified:

What this page covers

This page is a coder-facing reference for enabol/quantization.py: NumPy quantization, TensorFlow fake quantization with a straight-through estimator, rail/underflow statistics, and the TensorFlow implementations of HLS-style rounding, clipping, and wrapping.

Module Mapโ€‹

GroupObjectsPurpose
StatisticsRailStats, rail_stats(...)Record saturation, near-rail pressure, and underflow pressure.
Public quantizersquantize_np(...), quantize_tensor(...)Apply an HLSDataType to NumPy values or TensorFlow tensors.
TensorFlow internals_quantize_tensor_value(...), _round_tensor(...), _clip_tensor(...), _wrap_tensor(...)Implement HLS-style quantization behavior for tensors.
Quantization TODOs
  • Add focused tests comparing NumPy dtype quantization against TensorFlow tensor quantization for every supported QMODE / OMODE pair.
  • Decide whether wraparound modes are useful for ablations or should be avoided in favor of saturation-first experiments.
  • Add optional richer rail statistics if future controllers use hardware-feasible saturation proxies.

Imports And Dependenciesโ€‹

from __future__ import annotations

from dataclasses import dataclass
from typing import Any

import numpy as np
import tensorflow as tf

from .dtypes import HLSDataType, ap_fixed, ap_int, ap_ufixed, ap_uint

quantization.py bridges the dtype descriptions in dtypes.py and the TensorFlow training path in nn.py.

Data Classesโ€‹

RailStatsโ€‹

@dataclass(frozen=True)
class RailStats:
min_value: float
max_value: float
saturation_fraction: float
near_rail_fraction: float
underflow_fraction: float

RailStats is the small diagnostics object returned by rail_stats(...).

FieldMeaning
min_valueLower representable rail for the dtype.
max_valueUpper representable rail for the dtype.
saturation_fractionFraction of values at or beyond either rail.
near_rail_fractionFraction of values whose magnitude is near the maximum rail magnitude.
underflow_fractionFraction of nonzero values smaller than half a quantization step.

These values are used as software observables for quantized ablations. They are also candidates for hardware-feasible controller proxies later, but only after we decide which statistics are cheap enough to implement on edge.

Public Functionsโ€‹

quantize_np(x, dtype)โ€‹

def quantize_np(x: Any, dtype: HLSDataType | None) -> Any:
if dtype is None:
return x
return dtype(x)

Applies a dtype object to a NumPy-like value. If dtype is None, the input is returned unchanged, which is the default floating-point path.

Usage:

from enabol import dtypes
from enabol.quantization import quantize_np

weight_dtype = dtypes.ap_fixed(12, 4, "AP_RND", "AP_SAT")
W_q = quantize_np(W, weight_dtype)

This path delegates to the dtype implementation in dtypes.py, so it is the reference behavior for NumPy-side checks.

quantize_tensor(x, dtype, *, ste=True)โ€‹

def quantize_tensor(x: tf.Tensor, dtype: HLSDataType | None, *, ste: bool = True) -> tf.Tensor:
...

Applies fake quantization to a TensorFlow tensor.

If dtype is None, the tensor is returned unchanged. Otherwise, the forward value is quantized by _quantize_tensor_value(...).

With the default ste=True, the function uses a straight-through estimator:

return x + tf.stop_gradient(q - x)

This means:

  • forward pass value: quantized,
  • backward pass gradient: identity through the quantizer.

That is the right default for fake quantization inside tf.GradientTape because hard rounding has zero or undefined gradients almost everywhere.

Usage:

x_q = quantize_tensor(x, activation_dtype, ste=True)
STE is a software approximation

The straight-through estimator is useful for ablation studies, but it is not the same thing as hardware arithmetic. It lets us study the effect of quantized forward/update values while still allowing TensorFlow to produce gradients.

rail_stats(x, dtype, *, near_ratio=0.95)โ€‹

def rail_stats(x: Any, dtype: HLSDataType | None, *, near_ratio: float = 0.95) -> RailStats:
...

Computes saturation and rail-pressure statistics for an array-like value.

If the array is empty or dtype is None, it returns neutral values:

RailStats(np.nan, np.nan, 0.0, 0.0, 0.0)

For a real dtype, it computes:

saturationย fraction=#{xiโ‰คxminโ€…โ€Šโˆจโ€…โ€Šxiโ‰ฅxmax}N.\text{saturation fraction} = \frac{\#\{x_i \leq x_{min} \;\lor\; x_i \geq x_{max}\}}{N}.

The near-rail fraction is:

#{โˆฃxiโˆฃโ‰ฅrโ‹…maxโก(โˆฃxminโˆฃ,โˆฃxmaxโˆฃ)}N,\frac{\#\{|x_i| \geq r \cdot \max(|x_{min}|, |x_{max}|)\}}{N},

where r = near_ratio.

The underflow fraction is measured when the dtype exposes a quantum:

#{xiโ‰ 0โ€…โ€Šโˆงโ€…โ€Šโˆฃxiโˆฃ<12ฮ”}N,\frac{\#\{x_i \neq 0 \;\land\; |x_i| < \frac{1}{2}\Delta\}}{N},

where \Delta is the quantization step.

Usage:

stats = rail_stats(weights, weight_dtype)
print(stats.saturation_fraction)
print(stats.near_rail_fraction)

TensorFlow Internal Functionsโ€‹

These helpers are implementation details. They are documented because anyone adding new HLS dtype behavior must keep the TensorFlow path aligned with dtypes.py.

_quantize_tensor_value(x, dtype)โ€‹

def _quantize_tensor_value(x: tf.Tensor, dtype: HLSDataType) -> tf.Tensor:
...

Converts a TensorFlow tensor into the scaled integer domain, rounds it, applies overflow behavior, and scales it back.

Algorithm: TensorFixedPointQuantize
  1. if dtype is fixed-point then
  2. sโ†2WLโˆ’IWLs \leftarrow 2^{WL-IWL}
  3. else
  4. sโ†1s \leftarrow 1
  5. end if
  6. xsโ†xโ‹…sx_s \leftarrow x \cdot s
  7. zโ†roundโกQMODE(xs)z \leftarrow \operatorname{round}_{QMODE}(x_s)
  8. if OMODE == AP_WRAP then
  9. zqโ†wrapโก(z,WL)z_q \leftarrow \operatorname{wrap}(z, WL)
  10. else if OMODE is saturation-like then
  11. zqโ†clipโก(z,WL)z_q \leftarrow \operatorname{clip}(z, WL)
  12. end if
  13. return zq/sz_q / s
This mirrors the NumPy dtype path but operates on TensorFlow tensors inside the training loop.

Supported dtype families:

DtypeScaleSigned?Default qmode/omode behavior
ap_fixed2^fractional_bitsyesuses dtype QMODE / OMODE
ap_ufixed2^fractional_bitsnouses dtype QMODE / OMODE
ap_int1yesAP_TRN, AP_WRAP
ap_uint1noAP_TRN, AP_WRAP

Unsupported dtype classes raise TypeError.

_round_tensor(x, qmode)โ€‹

def _round_tensor(x: tf.Tensor, qmode: str) -> tf.Tensor:
...

TensorFlow implementation of supported HLS-style rounding modes.

QMODEBehavior
AP_TRN, AP_TRN_ZEROTruncate toward zero.
AP_RND, AP_RND_CONVTensorFlow round-to-nearest.
AP_RND_INFRound half toward positive infinity.
AP_RND_MIN_INFRound half toward negative infinity.
AP_RND_ZERORound to nearest, ties toward zero.

Unsupported modes raise NotImplementedError.

_clip_tensor(x, WL, *, signed)โ€‹

def _clip_tensor(x: tf.Tensor, WL: int, *, signed: bool) -> tf.Tensor:
...

Clips integer-domain values to the representable range.

Signed:

[โˆ’2WLโˆ’1,2WLโˆ’1โˆ’1].[-2^{WL-1}, 2^{WL-1}-1].

Unsigned:

[0,2WLโˆ’1].[0, 2^{WL}-1].

This is used for AP_SAT, AP_SAT_ZERO, and AP_SAT_SYM in the current implementation.

Saturation modes are currently grouped

AP_SAT, AP_SAT_ZERO, and AP_SAT_SYM currently use the same TensorFlow clipping path. If future experiments depend on the subtle differences between these modes, update this function and add tests against the NumPy dtype behavior.

_wrap_tensor(x, WL, *, signed)โ€‹

def _wrap_tensor(x: tf.Tensor, WL: int, *, signed: bool) -> tf.Tensor:
...

Wraps integer-domain values modulo 2^WL.

Unsigned values use:

xโ€Šmodโ€Š2WL.x \bmod 2^{WL}.

Signed values use the same modulo operation, then reinterpret values at or above the sign bit as negative:

xsigned={xwrappedโˆ’2WL,xwrappedโ‰ฅ2WLโˆ’1xwrapped,otherwise.x_{signed} = \begin{cases} x_{wrapped} - 2^{WL}, & x_{wrapped} \geq 2^{WL-1} \\ x_{wrapped}, & \text{otherwise}. \end{cases}

Wraparound is hardware-realistic, but it can make early ablation plots harder to interpret. Saturation should remain the first-line setting for stability experiments unless wrap behavior is the object being tested.

Integration Pointsโ€‹

quantization.py is used by BaseModel.train_instrumented(...) through three main paths:

Training-loop pathQuantization role
_forward_with_precision(...)Quantizes inputs, weights, biases, and activations during the forward pass.
_quantize_gradients(...)Quantizes gradients before the update is formed.
_quantize_variable_storage(...)Quantizes stored trainable variables after applying the update.

The precision choices are supplied by PrecisionDict. For example:

from enabol import PrecisionDict, dtypes

precision = PrecisionDict({
"dense0": {
"weight": dtypes.ap_fixed(12, 4, "AP_RND", "AP_SAT"),
"gradient": dtypes.ap_fixed(16, 6, "AP_RND", "AP_SAT"),
"update": dtypes.ap_fixed(16, 4, "AP_RND", "AP_SAT"),
},
})

Extension Notesโ€‹

When adding a new dtype behavior, keep these three paths synchronized:

Checklist: AddQuantizationBehavior
  1. Update NumPy dtype behavior in dtypes.py
  2. Update TensorFlow fake-quantization behavior in quantization.py
  3. Add tests comparing both implementations on the same input values
  4. Update the implementation documentation and any affected experiment notes
If these paths diverge, software ablations can stop representing the intended hardware arithmetic.