In Search of the Best Machine Learning Models and Software Support for TinyML

At the moment, the TinyML industry is still at its nascent stage. ABI Research forecasts the global shipments of TinyML devices will reach 6.3 billion by 2030, with key addressable markets including consumer devices, industrial and manufacturing, and transport & logistics, as well as smart cities applications. However, deploying Machine Learning (ML) models in TinyML application remains a very challenging task. Traditional edge AI applications are generally supported by high-performance general-purpose chipset, or by dedicated neural network accelerators. These models are process workload demanding and are not optimized for compute and power constrained terminals, such as IoT devices. In contrast, TinyML devices generally run on batteries and rely on power efficient Arm Cortex-M microcontroller (MCU) that consumes milliwatts or microwatts of power. This requires TinyML engineers to maximize the available resources in a TinyML device.

One of the ways to achieve that is through the selection of the most optimal machine learning model. Currently, there are three main options for machine learning models in TinyML:

• Tree-Based Model: Tree-based models are classical machine learning models that rely on a series of if-then rules to generate predictions from one or more decision trees. Tree-based models are able to learn from past experiences, using either supervised, unsupervised, or reinforcement learning, and adapt to new situations and circumstances. Random Forest, XGBoost, and Gradient Boosting Machine are some of the most popular tree-based models. While supported by major AI frameworks, developers need to rely on proprietary solutions from vendors such as Ekkono Solutions, ImagiMob, and Qeexo to deploy tree-based models on TinyML devices.
• Artificial Neural Network (ANN): ANNs are based on a collection of connected units or nodes called artificial neurons, which loosely model the neurons in a biological brain. Neurons and edges typically have a weight that can be adjusted as the network is being trained. Typically, neurons are aggregated into layers. The more layers an ANN has, the more accurate it becomes. Popular ANNs include convolutional neural networks (CNN) and recurrent neural networks (RNN). Key vendors include Qualcomm, GreenWaves Technologies, Syntiant, and XMOS.
• Spiking Neural Network (SNN): SNN are artificial neural networks that more closely mimic natural neural networks. SNN relies on event-driven network architecture, where the spike or signal will only be triggered when a firing threshold is reached, minimizing the power required as compared to a fully connected ANN. Key vendors include BrainChip, GrAI Matter Labs, and SynSense.

At the moment, most TinyML projects utilize ANNs. A major part of the reason is due to developers’ familiarity with TensorFlow Lite for Microcontroller, an AI framework designed for TinyML that is based on Google’s popular AI framework. TensorFlow Lite for Microcontroller is able to convert ANNs into smaller size models that are optimized for TinyML devices. This has generated a lot of interests among developers in TinyML, as developers have utilized ANNs for wide range of applications such as always on machine vision, audio and speech processing, and time-series data processing. Apart from Google, AI chipset vendors such as Qualcomm, GreenWaves Technologies, Syntiant, and XMOS have also launched neural network accelerator with a wide range of language support (Java, Swift, Objective-C, C++, and Python), model optimization, quantization, and hardware acceleration support.

However, while ANNs are known for their high performance and accuracy, they also require high compute and memory bandwidth. As such, some vendors, such as Qeexo, are championing tree-based models as an alternative solution. In general, tree-based models generally require less training data, have lower latency, are smaller in size, and do not need a significant amount of RAM during inference. This allows AI devices to provide much better user experience while benefiting from longer battery life and lower Bill of Materials (BoM) cost. Currently, developers that utilize tree-based model will need to perform their own model conversion and compression. Qeexo aims to address that gap in the market through their Model Converter solution.

Qeexo Model Converter supports tree-based models developed using the Open Neural Network Exchange (ONNX) format. In order to optimize the models for edge devices, the Model Converter will apply quantization to the algorithms to reduce memory footprint and power consumption. It is also able to reduce the input models to a desired size by applying a tree-pruning technique, making the models ideal for time-series data processing at the edge. Qeexo believes its solution will enable more developers to embrace TinyML and fully capitalize on the promise of the technology.

Aside from tree-based models and ANN, some players in the industry believe that Spiking Neural Network (SNN) may address some of the pain points in TinyML deployment. SNNs in neuromorphic computing are good at performing massive parallel computing and one-shot training, requiring less energy than the regular artificial neural network. This creates an opportunity for neuromorphic chipset vendors, such as GrAI Matter Labs and SynSense, to develop ultra-low-powered chipsets for TinyML. However, SNN remains niche. SNNs have much larger computational costs to train, have less readily available dataset, and the developer community around SNN is less mature as compared to tree-based models and ANNs. As such, some neuromorphic chipset startups such as BrainChip have pivoted to support ANNs.

At the end of the day, there is no one-size-fits-all solution for TinyML. Each model type has its purposes. Tree-based models are better for time-series data, but ANN still holds significant advantages in machine vision, audio, and speech processing. To achieve the best TinyML performance, machine learning engineers need to have strong knowledge in both ML and embedded system design. Most ML models are developed using high level programming languages, such as R and Python, while embedded system design relies on low level programming languages, such as C++. This creates a significant barrier to entry as well as opportunities for vendors and solution providers with the right domain expertise to step in and offer their support.

Hence, tools from the likes of Qeexo will play a critical role in increasing TinyML adoption. Considering the commercial potential of TinyML, more and more startups offer compelling alternatives to Qeexo’s solution. Nota specializes in AI model compression using a combination of uniform quantization and channel expansion. Edge Impulse, Ekkono Solutions, Imagimob, and SensiML all offer custom model compression techniques and development expertise to embed tree-based models and ANNs into MCU devices. Neuton, a subsidiary of Bell Integrator, is a startup that offers a proprietary self-organizing perceptron neural network that uses neuron-by-neuron network structure growth and creates minimum-size neural networks for TinyML devices with no loss of accuracy. The success of these solutions is dependent on the willingness of end users to accept the tradeoff between flexibility and performance. Regardless, TinyML software and service provider market holds much potential and should not be underestimated.