About tinyML Asia

Exercising regularly is essential for maintaining a healthy lifestyle. Studies have shown that knowing parameters such as calories burnt, step counts, and miles travelled helps track fitness goals and maintain motivation [1]. Modern fitness trackers aim to provide these metrics in real-time, with features like ubiquitous connectivity and lightweight. Most fitness trackers are either wearables or camera-based. Wearable fitness trackers can cause discomfort during exercise, and exchanging users’ data over the internet poses a privacy threat [2] [3]. The camera-based fitness trackers also often pose a privacy risk. Certain works prevent this by hiding the face of users through complex algorithms, adding to the computation cost requiring high-end processing computers to make readings available in real-time [4]. We propose tinyRadar as a contactless fitness tracker which identifies the exercise performed by the user and computes its repetition counts. It comes in a small form factor and preserves the user’s privacy as it provides point cloud data, making it a suitable choice for a fitness tracker in a smart home setting.

As can be seen from Figure 1, it comprises a Texas Instruments (TI) IWR1843 mmWave radar board as a sensing and processing modality and an ESP32 module for results transmission to the user’s smartphone through Bluetooth Low Energy (BLE). The mmWave radar processes the information received from the target environment to create a Velocity-Time map which contains unique signatures for different human activities. A three-layered Convolutional Neural Network (CNN) deployed on the radar board classifies the exercise performed by the user with the Velocity-Time map as input in one of the following eight categories: Crossover toe touch, Crunches, Jogging, Lateral squats, Lunges, Hand rotation, Squats, and Rest. The repetition count algorithm runs parallelly on board to compute the repetition counts. It provides a real-time subject-independent classification accuracy of 97% and repetition counts with an accuracy of ±4 counts.

References:

1. https://www.runnersworld.com/news/a40774442/fitness-trackers-help-you-move-more-study/
2. Fitnessarmbaender – Nur zwei von zwoelf sind gut, test.de, December 27, 2015. Retrieved onJanuary 6, 2016
3. Sly, Liz (29 January 2018). ”U.S. soldiers are revealing sensitive and dangerous information byjogging”. The Washington Post. Retrieved 29 January 2018.
4. Rushil Khurana, Karan Ahuja, Zac Yu, Jennifer Mankoff, Chris Harrison, and Mayank Goel. 2018.GymCam: Detecting, Recognizing and Tracking Simultaneous Exercises in Unconstrained Scenes. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 4, Article 185 (December 2018), 17 pages. https://doi.org/10.1145/3287063

Figure 1: tinyRadar as a contactless fitness tracker. (a) IWR1843 mmWave radar board and ESP32 integrated inside a housing (b) Breakout view of the tinyRadar assembly where the radar board and ESP32 are connected using an interface board (c) Functional block diagram showing the RF front end, radar signal processing and classification module along with the result transmission module comprising of ESP32. The results are transmitted over BLE to a smartphone.

Tiny machine learning (TinyML), which targets energy efficiency and low cost in AI-to-device integration, has been proven to be an effective implementation for data-intensive smart IoT devices. But device heterogeneity and MCU con- straints are two main challenges against further application based on tiny ML [1]. Conventional computer architecture, like Von Neumann architecture, separates the memory and computing part, will incur significant costs in transferring data on TinyML devices. To overcome this barrier, emerging architecture called compute-in-memory (CIM) shed new light by eliminating the boundary of memory and storage. This is especially beneficial for tinyML models that run inference on power-constrained devices. Among different flavours of CIM architectures, SRAM-based CIM gains increasing interests due to its great compatibility with the advanced technology and performance advantages [2].

Device heterogeneity has introduced tremendous advantages to today’s computing landscape, yet it also leads to costly design efforts and longer time to market. This is even more critical in architecting CIM, where customization efforts are required [1]. Learning from the history, reconfigurability has offered great opportunities for leveraging the performance tradeoffs and design costs. It has enabled generations of new architectures that can be reconfigured at different levels. Ex- isting CIM architectures were usually designed under the con- straint of very specific applications. For example, [3] proposed a 12T cell design useful for bit-wise XNOR calculation. [4] proposed two peripheral designs for bit-serial and bit-parallel vector accelerators for throughput-oriented and efficiency- oriented scenarios respectively. Design in [5] looked at full- system design of DNN computing cache architecture. There has been a few recent CIM work that attempted to integrate the recongigurability. [6] presented a system-level design that offers reconfigurable precision supports for cloud computing. [7] proposed a CIM macro supporting reconfigurable bit- wise operations. These work showed feasibility of adopting reconfigurable CIM architectures in AI applications and also great potential of integrating more functionalities on a single macro.

Inspired by these work, we propose to develop a cross-layer reconfigurable CIM solution, where opportunities need to be gained from cells, array, peripherals, to architecture and system designs. In this talk, we will show the progress of the first step on the roadmap where we design a novel all-digital SRAM cell macro that is able to support multiple commonly used operators from TinyML models in an all-in-one fashion.

Multiple compute-in-memory cell designs integrate the computing logic circuits mainly in the memory array pe- ripherals. This approach highly relies on operands’ physical locations, and performance improvement is thus limited. In order to support more flexibility for operands while still guaranteeing less latency for simple arithmetic operations, part of the computation workload along with the reconfigurable capability are integrated within the SRAM cell in this work. We introduce a novel 10T multi-logic SRAM cell that can support at least five basic operation modes, they are writ- ing, reading, and bit-wise AND/OR/XOR logical calculations. More complex functions were then be built upon these cell- level operations with extended circuitry. In Table I, we list the current supported function modes from the proposed CIM macro.

It is worth to mention that pipelined scheme has been em- ployed at the peripherals to further improve the performance. The design also makes use of concurrent reading of two adja- cent cells and separates the carry-ripple adder into two stages to reduce latency in addition operations. The proposed design has been implemented and simulated in Cadence Virtuoso with an industry-standard 28nm process node. Since it is purely digital, the design is synthesizable and compatible with the standard top-down digital implementation flow. In the talk, we will present detailed simulation results, design methodology and the comparisons against other state of the art designs. The final goal of this project is to deliver a system level compute-in-memory design approach that can support a wide range of tiny ML algorithms.

As deep learning advances, edge devices and lightweight neural networks are becoming even more important. In order to reduce latency in the AI accelerator, it is important not only to reduce FLOPs but also to increase hardware efficiency. By analyzing the hardware performance of Arm Ethos-U65 NPU using Arm Virtual Hardware, we discovered that the latency of the convolution layer showed a staircase pattern varying on configuration. To utilize Arm Ethos-U65 NPU fully, the parameter of the compression methods should be set to include consideration of the latency pattern. By applying device characteristics of Arm Ethos-U65 NPU to NetsPresso which is a hardware-aware AI optimization platform, we adjusted the parameters of structured pruning and filter decomposition to fit a multiple-of-step size of the latency staircase pattern. In the image classification task, we validated the hardware-aware model compression increased FLOPs and accuracy at the same latency.

The neuromorphic industry is growing rapidly, with significant contributions from large players such as Intel and IBM. SynSense is developing approaches for machine vision, bio- and industrial-signal processing, based on binary event-based neuronal architectures and communication emulated on low-power neuromorphic hardware. Our advantages are our long experience in designing low-power neuromorphic circuits, and our expertise in theory of neuromorphic computation.

Xylo is a new family of ultra-low-power (<1mW) neuromorphic ML inference processors designed by SynSense. Xylo is a fully programmable, ultra-low power neuromorphic architecture supporting feed forward, recurrent, reservior and other complex spiking neural networks. Xylo-A2 also integrates a dedicated analog Audio-Front-End supporting low-power real-time processing of single-channel audio signals.

Xylo provides efficient simulation of spiking leaky integrate-and-fire neurons with exponential input synapses. Xylo is highly configurable, and supports individual synaptic and membrane time-constants, thresholds and biases for each neuron. Xylo supports arbitrary network architectures, including recurrent networks, for up to 1000 neurons. With the adaption of SynSense’s novel spiking neural network algorithms, Xylo is able to power TinyML applications continuously at a few hundred uW.

Deep learning and machine learning have enjoyed great pop- ularity and achieved huge success in the last decade. With the wide adoption of inferences on tinyML or Edge AI platforms, it becomes increasingly critical to develop optimal models that can fit the limited hardware budget [1]. However, state-of-the- art models with higher accuracy also come with large sets of parameters which requires significant amount of computing resources and memory footprint. As a result, this limits the deployment of real-time DNN models on the edge, such as real- time health monitoring, autonomous driving, real-time transla- tion and various other application domains. Quantization has been demonstrated as one of the most efficient model compres- sion solutions for reducing the size and runtime of a network via reduction of bit-width [2], [3]. Quantization reduces the precision of network components to the extent that it is more lightweight but without a “noticeable” difference in efficacy. The most straightforward way of applying quantization is called uniform quantization, where the resulting quantized values (aka quantization levels) are uniformly space. For example, uniform INT8 has been widely used in inference AI models [4]. As the demand for inference speed keeps increasing, and the on-device storage becomes limited, lower precision quantization scheme such as INT4 has been employed. But uniform scheme (e.g. all layers are quantized to 4 bits) can cause significant accuracy loss.

In order to further reduce the hardware consumption and compress the model, mixed precision quantization (MPQ) was proposed [5]–[9], where some layers maintain higher precision and others are kept lower precision quantization. Therefore, it strikes a better balance between accuracy and hardware cost [10]–[16]. However, many of these techniques were purely accuracy driven without considering hardware implementation costs. When it comes to the edge inference units, the limitation set by hardware resources requires a very fine-grained design space searching process that is able to provide explainable tradeoffs. This certainly applies to the problem of finding the best mixed-precision quantization scheme at the edge that needs to be both hardware and application friendly. While there were few existing MPQ solutions that introduced hardware awareness in the search process, the searching is either com- puting intensive or model specific. Thus a light and explainable hardware-aware MPQ methodology is required. In this work,

we propose a novel MPQ searching algorithm that firstly “sam- ples” the layer-wise sensitivity with respect to a newly proposed metric, which incorporates both accuracy (from application perspective) and proxy of cost (from hardware perspective). Based on the hardware budget and accuracy requirements, a candidate search space can be decided, and the optimal MPQ scheme is further explored. Evaluation results show that the proposed solution achieves 3%-11% higher inference accuracy with similar hardware cost compared to the state of the art MPQ strategies. At the hardware level, we propose a new processing- in-memory (PIM) architecture that tightly integrates the optimal MPQ policies as part of the processor pipeline through Instruc- tion Set Architecture (ISA) and micro-architecture co-design. To summarize, this work makes the following contributions.

• We define a new metric that takes consideration of hard- ware cost and accuracy loss simultaneously while perform- ing the search of optimal MPQ schemes.
• We propose a light MPQ searching methodology by an- alyzing single layer sensitivity first and then narrowing down the search space. The proposed strategy outperforms other existing hardware-aware MPQ search solutions for lightweight neural network models.
• We look into the adaptation of the optimal MPQ schemes at the hardware level and propose a tightly integrated layer-wise quantized mixed-precision unit that resides in the CPU pipeline. The customized architecture allows seamless processing of both AI and non-AI tasks on the same hardware.
• In this talk, we will discuss the details of the proposed search methodology for layer-wise hardware-aware MPQ, including the design space exploration, hardware architecture design methodology and evaluation results.