Training machine learning models on edge devices has definite advantages for security, privacy and latency. However, techniques such as Deep Neural Networks (DNNs) are unsuitable given the resource constraints on such devices. Optimizing DNNs is especially challenging due to the nonconvex nature of their loss function. While gradient-based methods that use back-propagation have been crucial to neural network adoption, optimal convergence of the loss function is still time-consuming, volatile, and needs many finely tuned hyperparameters. One key hyperparameter is the learning rate. A high learning rate can produce fast results faster but at the increased risk of the model never converging. In this talk, I explain one of the advances from our lab that show that by manipulating the model weights directly using their distributions over batch-wise updates, we can achieve significant intermediate improvements in training convergence, and add more robustness to the optimization process with negligible cost of additional training time. More importantly, this approach allows deep neural networks to be trained at higher than usual learning rates resulting in fewer epochs which reduces resource use and allows for lower total training time.

Power has always been a challenge for battery-operated always-on devices, and with additional privacy and communication requirements pushing more data processing to the device, power is an even more critical constraint. Aspinity will discuss how analog ML cultivates an always-on system architecture that mimics the brain’s ability to use a small amount of energy up front to determine which data are important before committing higher power resources to further analysis. This approach allows designers to partition more efficient always-on systems that determine which sensor data are important while the data are still analog and subsequently eliminate the digitization and higher-power analysis of irrelevant data that will simply be thrown away.