Skip to main content

MDL-DENSE1-LINEAR-NOBIAS-000: One Dense Linear Student

Status:ValidIn progress
Tags:ModelDenseLinearLin1
Last modified:

Summary​

This is the smallest student model used in the first global-throttle ablations. It has one Dense layer, no activation, and no bias. It is designed to match the linear teacher in DS-AFFINE-LINEAR-000.

FieldValue
Model IDMDL-DENSE1-LINEAR-NOBIAS-000
Model familyDense linear
Input dimensiond_in = 4
Output dimensiond_out = 2
Layers1 Dense
ActivationNone
BiasDisabled
Stable layer namedense0
Trainable parameters4 x 2 = 8 Keras kernel parameters

Forward Pass​

The student model computes:

y^=Wx.\hat{y} = W x.

In Keras Dense storage convention, the kernel has shape:

Wkeras∈RdinΓ—dout.W_{\mathrm{keras}} \in \mathbb{R}^{d_{\mathrm{in}} \times d_{\mathrm{out}}}.

The math convention used in the experiment docs is:

Wmath=WkerasT∈RdoutΓ—din.W_{\mathrm{math}} = W_{\mathrm{keras}}^T \in \mathbb{R}^{d_{\mathrm{out}} \times d_{\mathrm{in}}}.

Diagram​

Training Role​

This model is useful because the one-layer no-bias regression loss has an exact Hessian. That lets the ablation compare:

  • analytic stability margin,
  • online curvature proxy,
  • selected global throttle alpha_t,
  • quantized update distortion.

Used By​

ExperimentRole
EXP-000AFloat global-throttle sanity check.
EXP-000BQuantized global-throttle sanity check.

Implementation​

Implemented through LinearBlockModel with:

LinearBlockModel(
dataset=dataset,
num_hidden=[],
activation=None,
use_bias=False,
)

Reference: Model implementation