Embedded AI
Introduction
What Question Does It Ask?
Embedded AI studies running trained models on devices with limited memory, compute and energy, such as phones, cameras or microcontrollers. The course covers shrinking a model by quantization, pruning and distillation, moving it onto the device, and measuring latency and energy; the extreme form is called TinyML.
At a glance
resource-constrained inference
trained model, device constraints
small, fast model
latency, energy, memory
medium
training data already exists
Introduction
A Short History
Hinton, Vinyals and Dean introduce knowledge distillation: a large model's knowledge passes to a small one.
Han et al. combine pruning and quantization in Deep Compression, shrinking networks by tens of times.
MobileNets is published and TensorFlow Lite announced: deep networks on phones become routine.
Google announces the Edge TPU accelerator for edge devices.
The first tinyML Summit is held; Warden and Situnayake publish the book TinyML.
MCUNet runs ImageNet classification on a microcontroller with over 70% accuracy.
The Course
Core Concepts
8-bit or smaller integers instead of 32-bit floats.
Removing unimportant weights, channels or layers.
A small student model mimics a large teacher.
MobileNet, depthwise separable convolution.
The thin software layer that runs the model on the device.
NPU, DSP, Edge TPU: hardware built for neural networks.
A few hundred KB of SRAM and flash for weights and activations.
Two performance metrics as important as accuracy.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Embedded hardware: MCUs, memory and energy budgets
- Sensor data collection and preprocessing
- Small neural networks and efficient architectures
- Post-training quantization
- Quantization-aware training
- Pruning and sparse networks
- Knowledge distillation
- Model conversion: TensorFlow Lite and ONNX
- Inference on microcontrollers: TFLite Micro, CMSIS-NN
- Accelerators: NPUs, DSPs and the Edge TPU
- Measuring latency, memory, energy; MLPerf Tiny
- On-device learning and over-the-air model updates
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
Fit the Model on a Microcontroller
A vibration sensor on a motor must recognize faults by itself, but its processor has 512 KB of memory. Choose how many bits to store each weight in; size, speed and accuracy change together.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Deep Learning | quantization, pruning, small models |
| Microprocessors | memory, registers, interrupts and peripherals: understanding device limits |
| Programming with C | the language of embedded software; pointers and static memory management |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you train a convolutional network and report its validation accuracy? | Deep Learning |
| Can you write C code with pointers and without dynamic memory allocation? | A C programming course |
| Can you flash a program to a microcontroller and read data from a sensor? | A microprocessors course and an Arduino starter kit |
| Can you explain how a number is represented as an 8-bit integer? | Binary numbers and fixed point from digital design |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| application | This course applies the methods of Deep Learning | quantization, pruning, small models |
| shares methods | Shares methods with Robotics and Robot Vision | real-time on-device inference |
| shares methods | Shares methods with Machine Learning Systems and MLOps | deploying and updating models on devices |
| shares methods | Shares methods with GPU Programming and Parallel Computing | hardware accelerators |
Differences
Where It Differs (1/2)
| Criterion | Embedded AI | Deep Learning | GPU & Parallel Computing |
|---|---|---|---|
| Core question | How do AI models run on small, low-power devices? | How do many-layered networks learn useful representations from raw data? | How do thousands of cores share the same computation? |
| Approach | resource-constrained inference | multilayer representation learning | data-parallel computation |
| Input | trained model, device constraints | raw data: images, text, audio | parallelizable algorithm |
| Output | small, fast model | learned representation and prediction | accelerated program |
| Success measure | latency, energy, memory | accuracy, loss, scaling | speedup, memory bandwidth |
| Math load | medium | high: derivatives, matrices, probability | low-medium |
| Data needs | training data already exists | large; needs GPUs | none |
- Deep Learning: Deep learning grows the model for accuracy; this course shrinks the same model while losing little accuracy.
- GPU & Parallel Computing: GPU programming gets peak performance from large accelerators; this course runs inference on devices drawing a few milliwatts.
Differences
Where It Differs (2/2)
| Criterion | Embedded AI | MLOps | Robotics |
|---|---|---|---|
| Core question | How do AI models run on small, low-power devices? | How is a model taken from the lab into production and kept running there? | How does a machine sense the physical world, know where it is and move? |
| Approach | resource-constrained inference | software engineering meets ML | sense-plan-act loop |
| Input | trained model, device constraints | trained model, data pipeline | sensor measurements |
| Output | small, fast model | a running, monitored service | motor commands, path |
| Success measure | latency, energy, memory | latency, uptime, data drift | task success, safety |
| Math load | medium | low | high: transformations, control |
| Data needs | training data already exists | streaming production data | physical experiments and simulation |
- MLOps: MLOps deploys and monitors models on servers; this course embeds the model in devices that may have no connection.
- Robotics: Robotics treats perception, localization and motion as a whole; this course makes the robot's model run in real time.
Practice
Applications
An always-listening model in speakers and phones.
Object detection without sending images to the cloud.
Fall, step and heart rhythm analysis on a watch.
Catching fault signs in motor vibration.
Photo enhancement, offline translation.
Sound and pest detection on battery sensors.
Practice
Risks and Ethics
Quantization and pruning barely move average accuracy, yet rare classes and minority groups can lose far more.
A faulty or vulnerable model spread over thousands of devices in the field is hard to recall.
Always-listening microphones and cameras can monitor bystanders unawares, even when data stays on the device.
The model file on a device can be read and copied, and reverse engineering opens it to adversarial attacks.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Well-known datasets
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| Models are trained on the small device. | Training is mostly done on a server; only inference runs on the device. |
| Quantization always ruins accuracy. | 8-bit quantization costs most models very little accuracy; quantization-aware training reduces the loss further. |
| Fewer parameters means a faster model. | Speed depends on memory access, operation types and hardware support; parameter count is not enough. |
| Embedded AI is only needed without internet. | For privacy, latency and energy, on-device processing is preferred even when a connection exists. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Warden and Situnayake, TinyML, O'Reilly, 2019.
- Sze, Chen, Yang and Emer, Efficient Processing of Deep Neural Networks, Morgan & Claypool, 2020.
- Han, Mao and Dally, "Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding", ICLR, 2016.
- Howard et al., "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications", arXiv:1704.04861, 2017.
- Hinton, Vinyals and Dean, "Distilling the Knowledge in a Neural Network", arXiv:1503.02531, 2015.