tinyML EMEA Innovation Forum 2022

Until now, embedded DL at the edge has been limited to simple Neural Networks such as CNNs.

InferSens is focused on leveraging new silicon architectures to enable more sophisticated models at the edge, that are able to unlock new meaning from sensor data while operating on a nanoPower budget compatible with a multi-year battery life.

This talk presents a real-world application of such networks to help automate the manual monitoring of water systems which is done in hundreds of millions of properties worldwide to prevent outbreaks such as Legionnaire’s disease. Our non-invasive and easy to deploy flow sensors infer flow and monitor temperature in real-time.

Nano-sized autonomous unmanned aerial vehicles (UAVs), i.e., nano-UAVs, are compelling
flying robots characterized by a stringent form factor and payload, such as 10 cm diameter
and a few tens of grams in weight. These fundamental traits challenge the onboard sensorial
and computational capabilities, allowing only for limited microcontroller-class computational units (MCUs) mapped to a sub-100 mW power envelope. At the same time, making a UAV fully autonomous means to fulfill its mission with the only resources available onboard, i.e., avoiding any external infrastructure or off-board computation.
To some extent, achieving such an ambitious goal of a fully autonomous nano-UAV can be
seen as the embodiment of the extreme edge computing paradigm. The computational/memory limitations are exacerbated by the paramount need for real-time
mission-critical execution of complex algorithms on a flying cyber-physical system (CPS).
Enabling timely computation on a nano-UAV ultimately would lead to i. new application
scenarios, otherwise prevented for bigger UAVs, ii. increased safety in the human-robot
interaction, and iii. reduced cost of versatile robotic platforms.
In recent years, many researchers have pushed the state-of-the-art in the onboard
intelligence of nano-UAVs by leveraging machine learning and deep learning techniques as
an alternative approach to the traditional and computationally expensive, geometrical, and
computer vision-based methods. This talk presents our latest research effort and
achievements by delivering holistic and vertical-integrated solutions, which frame
energy-efficient ultra-low power MCUs with deep neural network vision-based algorithms,
quantization techniques, and data augmentation pipelines.
In this presentation, we will follow our two keystone works in the nanorobotics area, i.e., the
seminal PULP-Dronet [1,2] and the recent PULP-Frontnet [3] project, as concrete examples
to introduce our latest scientific contributions. In detail, we will address several fundamental research questions, such as “how to shrink the number of operations and memory footprint of convolutional neural networks (CNNs) for autonomous navigation” [4], “how to improve the generalization capabilities of tiny CNNs for human-robot interaction” [5] and “how to combine the CPS’ state with vision-based CNNs for enhancing the performance of an autonomous nano-UAV” [6]. Finally, we will support our key findings with thorough in-field evaluations of our methodologies and resulting closed-loop end-to-end robotic demonstrators.

[1] Palossi, Daniele, Antonio Loquercio, Francesco Conti, Eric Flamand, Davide Scaramuzza, and
Luca Benini. “A 64-mW DNN-based visual navigation engine for autonomous nano-drones.” IEEE
Internet of Things Journal 6, no. 5 (2019): 8357-8371.
[2] Palossi, Daniele, Francesco Conti, and Luca Benini. “An open source and open hardware deep
learning-powered visual navigation engine for autonomous nano-UAVs.” In 2019 15th International
Conference on Distributed Computing in Sensor Systems (DCOSS), pp. 604-611. IEEE, 2019.
[3] Palossi, Daniele, Nicky Zimmerman, Alessio Burrello, Francesco Conti, Hanna Müller, Luca Maria
Gambardella, Luca Benini, Alessandro Giusti, and Jérôme Guzzi. “Fully onboard ai-powered
human-drone pose estimation on ultralow-power autonomous flying nano-uavs.” IEEE Internet of
Things Journal 9, no. 3 (2021): 1913-1929.
[4] Lorenzo Lamberti, Vlad Niculescu, Michał Barciś, Lorenzo Bellone, Enrico Natalizio, Luca Benini
and Daniele Palossi. “Tiny-PULP-Dronets: Squeezing Neural Networks for Faster and Lighter
Inference on Multi-Tasking Autonomous Nano-Drones.” In 2022 IEEE 4th International Conference on
Artificial Intelligence Circuits and Systems (AICAS), IEEE, 2022.
[5] Cereda, Elia, Marco Ferri, Dario Mantegazza, Nicky Zimmerman, Luca M. Gambardella, Jérôme
Guzzi, Alessandro Giusti, and Daniele Palossi. “Improving the Generalization Capability of DNNs for
Ultra-low Power Autonomous Nano-UAVs.” In 2021 17th International Conference on Distributed
Computing in Sensor Systems (DCOSS), pp. 327-334. IEEE, 2021.
[6] Cereda, Elia, Stefano Bonato, Mirko Nava, Alessandro Giusti, and Daniele Palossi. “Vision-State
Fusion: Improving Deep Neural Networks for Autonomous Robotics.” arXiv preprint arXiv:2206.06112
(2022).

Neuromorphic sensing and processing hold an important promise for creating autonomous tiny drones. Both promise to be lightweight and highly energy efficient, while allowing for high-speed perception and control. For tiny drones, these characteristics are essential, as these vehicles are very agile but extremely restricted in terms of size, weight and power. In my talk, I present our work on developing neuromorphic perception and control for tiny autonomous drones. I delve into the approach we followed for having spiking neural networks learn visual tasks such as optical flow estimation. Furthermore, I explain our ongoing effort to integrate these networks in the control loop of autonomously flying drones.
Schematic summary of the presentation:
– Introduction: tiny drones, specifications and constraints
– How to make tiny drones autonomous? Drawing inspiration from nature
– Neuromorphic sensing and computing
o Event cameras and spiking neural networks
o Goal: Fully neuromorphic, vision-based autonomous flight
o Work 1: Vertical landings using event-based optical flow
o Works 2 and 3: Unsupervised and self-supervised learning of event-based optical
flow using spiking neural networks
o Work 4: Neuromorphic control for high-speed landings
– Conclusion

Detection, tracking and classification of objects in real-time from a video is a very important but costly problem in computer vision. Typically, this is achieved by running a window based CNN and sliding it over the full image to obtain the areas of interest (i.e. location of the object) and then identifying it. This approach requires the image to be stationary such that the model can be run over the entire object over multiple passes. A second approach involves a one shot one pass model such as YOLO, where the entire image is passed and the localization and identification is done simultaneously. While this approach is better for real-time systems compared to the former due to its requirements of multiple passes, they typically require a lot of memory and thus consume a lot of power to achieve this. Furthermore, previous work along these lines also demonstrated that it is challenging to achieve good accuracy using spiking convolutional neural networks for such a model. The problem becomes more costly to solve due to the state-holding nature of spiking neurons and the requirement of using memory to store those states.

Locating and identifying the objects are two inherently different problems and if they are solved separately this might reduce the network size significantly. This is partially achieved in the field of neuromorphic engineering by the use of dynamic vision sensors, which only pass the information on the change in luminosity as events for individual pixels with corresponding timestamps rather than passing an entire frame. This eliminates the background which is unrelated for the objects to be tracked. These events can easily be clustered into separate clusters by their spatial locations. In this work, we implement such a clustering algorithm to track the objects, preprocess it such that each cluster has the same output size and then pass these events to a spiking neural network for identification. The pipeline is implemented and tested for tracking of single and multiple objects, as well as identification of these objects on novel DynapCNN™ asynchronous spiking convolutional neural network processor.