Alan, Seda, Manu sync
Updates on batchnorm and convolutional layers
We discussed the current progress of Andrew and Darrel on training models with batchnorm. They basically ran the MSWC notebook but added batchnorm (no training, no propagation of gradients): just training dense layers. The main question was does adding batchnorm layers to a model make it inherently more robust to drift?. The results that they got is that apparently it does not, which is good news for 🚂ENABOL. In fact, the accuracy seems to be better with batchnorm overall and we can also recover much more accuracy on hardware.
Conclusion: batchnorm improves accuracy (this is why people use them), but 🚂ENABOL is still useful and required for batch-norm-based models.
Next steps for @Andrew Tieu and @Darrel Zhao
- Re-run the
MSWCnotebook (with or without batchnorm) but now training the convolutional layers. This requires two things:- Making sure that the gradients still flow through the dense layers (even if they are frozen), to confirm the logic of the
apply_updateflag is consistent indense_backpass. - Making sure that the backpass for the
conv2dlayer is correct. - This will force them to dig into the
cppcode and understand how backprop works
- Making sure that the gradients still flow through the dense layers (even if they are frozen), to confirm the logic of the
- Later, re-run the experiment but now adding batchnorm layers as well (no training of batchnorm layers yet, just training conv layers). Important: this assumes that the backprop for batchnorm is correct, see Alan tasks below.
Updates on column throttling, -budgeting
Alan has been checking what the throttling of columns looked like on a per-layer basis, and for different budgets. He realized that the distribution of throttling was very uneven (meaning, some layer would get a lot of throttling, while others almost none). This is a big problem because it means that while learning the direction of the updates is changing. This is why when we had the full RowScale learning didn't work. We are fixing this with the global throttling (see notes on Manu updates below).
Next steps for @Alan Guo
- First, settle on a testing workflow that always logs three points for any test (basically the three points we have in our paper):
- baseline accuracy on HARDWARE without any drift (this is error introduced due to quantization, and the theoretical maximum we could recover on hardware).
- accuracy on HARDWARE with drift at
epoch=0(this is the first point of the training curve, it shows how bad the accuracy degrades without 🚂ENABOL). - accuracy on HARDWARE with drift at
epoch=END(this is the last point of the training curve, it shows how much we recovered with 🚂ENABOL).
- Correct the backprop pass for batchnorm layers (even if we are not training them, we still need to make sure the backprop chain is correct).
- First, I would do this manually, modifying the
cppcode directly infirmware, before trying to do thetemplatesandvivado-writeretc. - Later, we'll put it back onto
hls4mlto make it automatic.
- First, I would do this manually, modifying the
- Consider using always
training/validationsplit data to make our results more robust.
Updates on global throttling
Manu has been working on a new version of the stability algorithm that uses a global throttle instead of a per-layer independent budgeting/throttle. It also created this documentation page and ran the initial ablation tests on a dummy model to test the hypothesis exp-000-a and exp-000-b. The results show that indeed there is a divergence zone that will make the quantized model be unstable, and that global throttling can fix this exp-001. It's left for us to decide how we can reconcile this with the old -budgeting idea: are we just saying that this is a new implementation of the idea? Or this is ruling out our previous method?
Next steps for @Manu B. Valentin
- Re-run the ablation tests but now set instead of to confirm that the system is able to always keep training stable (in the first tests, because , a big enough learning rate would still be able to diverge in the first step, without the global throttle doing its job).
- Run the rest of ablation tests and check that the controller still works.
- Distribution shift.
- Transient spike in gradients, or inputs, or some path that would cause instabilities and divergence.
- Quantization ablations (rail saturation / underflow, dead-zone).
- Pull latest
hls4mlcode and start integrating the global throttle mechanism into it. - Be able to do
csimwith the global throttle before leaving for AMD so I can hand this off.