Log-Normal Multiplicative Dynamics for Stable Low-Precision Deep Learning

Abstract

We propose Log-Normal Multiplicative Dynamics (LMD), a new algorithm to stabilize low-precision deep learning. LMD is motivated by the robustness of biological neural networks where synaptic spine sizes follow a log-normal distribution and fluctuate via noisy multiplicative dynamics. So far, no algorithm has successfully incorporated all these synaptic properties to train large models. LMD incorporates these properties through a variational Bayesian formulation that uses a log-normal posterior over weights. We evaluate LMD with low-precision forward-pass matrix multiplications, including training a Vision Transformer and GPT-2 from scratch. Our findings demonstrate the utility of biologically inspired approaches for designing energy-efficient algorithms for stable low-precision deep learning.