tinyML Talks: Suitability of Forward-Forward and PEPITA Learning to MLCommons-Tiny benchmarks

On-device learning challenges the restricted memory and computation requirements imposed by its deployment on tiny devices. Current training algorithms are based on backpropagation which requires storing intermediate activations to compute the backward pass and to update the weights into the memory.
Recently ”Forward-only algorithms” have been proposed as biologically plausible alternatives to backpropagation. At the same time, they remove the need to store the intermediate activations which potentially lower the power consumption due to memory read and write operations, thus, opening to new opportunities for further savings. This talk investigates quantitatively the improvements in terms of complexity and memory usage brought by PEPITA and Forward-Forward computing approaches with respect to backpropagation on the MLCommons-Tiny benchmarks set as case studies. It was observed that the reduction in activations’ memory provided by ”Forward-only algorithms” does not affect total RAM in Fully-connected networks. On the other hand, Convolutional neural networks benefit the most from such reduction due to lower parameters-activations ratio. In the context of the latter, a memory-efficient version of PEPITA reduces, on average, one third of the total RAM with respect to backpropagation, introducing only a third more complexity.
Forward-Forward brings average memory reduction to 40%, and it involves additional computation at inference that, depending on the benchmarks studied, can be costly on micro-controllers.


September 19, 2023



Contact us



Timezone: PDT

Suitability of Forward-Forward and PEPITA Learning to MLCommons-Tiny benchmarks

Danilo PAU, Technical Director, IEEE and ST Fellow


Danilo PAU, Technical Director, IEEE and ST Fellow


Danilo Pau – Technical Director, IEEE & ST Fellow – System Research and Applications
One year before graduating from the Polytechnic University of Milan in 1992, Danilo PAU joined STMicroelectronics, where he worked on HDMAC and MPEG2 video memory reduction, video coding, embedded graphics, and computer vision. Today, his work focuses on developing solutions for deep learning tools and applications.

Since 2019 Danilo is an IEEE Fellow, serves as Industry Ambassador coordinator for IEEE Region 8 South Europe and Member of the Machine Learning, Deep Learning and AI in the CE (MDA) Technical Stream Committee IEEE Consumer Electronics Society (CESoc). With over 80 patents, 98 publications, 113 MPEG authored documents and more than 37 invited talks/seminars at various worldwide Universities and Conferences, Danilo’s favorite activity remains mentoring undergraduate students, MSc engineers and PhD students from various universities in Italy, US, France and India.

Schedule subject to change without notice.