Embedded AI

Guide to AI and Data Courses · Part VI: Systems and Applications
Resource limitsQuantizationPruning and distillationEfficient architecturesDevices and acceleratorsNeighbouring courses
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

What Question Does It Ask?

How do AI models run on small, low-power devices?

Embedded AI studies running trained models on devices with limited memory, compute and energy, such as phones, cameras or microcontrollers. The course covers shrinking a model by quantization, pruning and distillation, moving it onto the device, and measuring latency and energy; the extreme form is called TinyML.

At a glance

Approach

resource-constrained inference

Input

trained model, device constraints

Output

small, fast model

Success measure

latency, energy, memory

Math load

medium

Data needs

training data already exists

Introduction

A Short History

2015

Hinton, Vinyals and Dean introduce knowledge distillation: a large model's knowledge passes to a small one.

2016

Han et al. combine pruning and quantization in Deep Compression, shrinking networks by tens of times.

2017

MobileNets is published and TensorFlow Lite announced: deep networks on phones become routine.

2018

Google announces the Edge TPU accelerator for edge devices.

2019

The first tinyML Summit is held; Warden and Situnayake publish the book TinyML.

2020

MCUNet runs ImageNet classification on a microcontroller with over 70% accuracy.

The Course

Core Concepts

Quantization

8-bit or smaller integers instead of 32-bit floats.

Pruning

Removing unimportant weights, channels or layers.

Knowledge distillation

A small student model mimics a large teacher.

Efficient architectures

MobileNet, depthwise separable convolution.

Inference engine

The thin software layer that runs the model on the device.

Accelerators

NPU, DSP, Edge TPU: hardware built for neural networks.

Memory budget

A few hundred KB of SRAM and flash for weights and activations.

Latency and energy

Two performance metrics as important as accuracy.

The Course

A Typical Workflow

Trainon a server,full precisionShrinkquantize,pruneConvertTFLite, ONNXDeployMCU, phone,cameraRun andmeasurelatency,energy
Example: A small keyword-spotting network is trained on a computer with 32-bit floats, quantized to 8-bit integers and loaded onto a microcontroller with a few hundred kilobytes of memory. The device listens all the time, never sends audio to the cloud and uses very little power.

The Course

What Does the Course Cover?

  1. Embedded hardware: MCUs, memory and energy budgets
  2. Sensor data collection and preprocessing
  3. Small neural networks and efficient architectures
  4. Post-training quantization
  5. Quantization-aware training
  6. Pruning and sparse networks
  7. Knowledge distillation
  8. Model conversion: TensorFlow Lite and ONNX
  9. Inference on microcontrollers: TFLite Micro, CMSIS-NN
  10. Accelerators: NPUs, DSPs and the Edge TPU
  11. Measuring latency, memory, energy; MLPerf Tiny
  12. On-device learning and over-the-air model updates

Skills you will gain

converting models with TensorFlow Liteapplying 8-bit quantizationdeploying a model to Arduinoan end-to-end project in Edge Impulsemeasuring latency, memory and energy

A typical semester; the order and coverage vary between universities.

Try It

Fit the Model on a Microcontroller

A vibration sensor on a motor must recognize faults by itself, but its processor has 512 KB of memory. Choose how many bits to store each weight in; size, speed and accuracy change together.

● normal ● imbalance ● bearing fault. The network is trained here (63 weights); size is scaled to a real 300k-weight model.
Embedded AI treats shrinking the model as part of the job: 8-bit quantization usually keeps accuracy and cuts memory to a quarter. At 2 bits the decision boundaries fall apart.

Context

Prerequisites

Courses worth knowing before this one, and why they are needed.

CourseWhy it is needed
Deep Learningquantization, pruning, small models
Microprocessorsmemory, registers, interrupts and peripherals: understanding device limits
Programming with Cthe language of embedded software; pointers and static memory management

Context

Are You Ready for This Course?

Ask yourself; for every "no", the right-hand column shows where to start.

Ask yourselfIf not, start with
Can you train a convolutional network and report its validation accuracy?Deep Learning
Can you write C code with pointers and without dynamic memory allocation?A C programming course
Can you flash a program to a microcontroller and read data from a sensor?A microprocessors course and an Arduino starter kit
Can you explain how a number is represented as an 8-bit integer?Binary numbers and fixed point from digital design
If you can say "yes" to most of these, you are ready; one or two gaps can be closed during the semester.

Context

Neighbouring Courses

subfieldbuilds onshares methodsapplication

Overlap

Overlapping Courses

RelationCourseShared ground
applicationThis course applies the methods of Deep Learningquantization, pruning, small models
shares methodsShares methods with Robotics and Robot Visionreal-time on-device inference
shares methodsShares methods with Machine Learning Systems and MLOpsdeploying and updating models on devices
shares methodsShares methods with GPU Programming and Parallel Computinghardware accelerators

Differences

Where It Differs (1/2)

CriterionEmbedded AIDeep LearningGPU & Parallel Computing
Core questionHow do AI models run on small, low-power devices?How do many-layered networks learn useful representations from raw data?How do thousands of cores share the same computation?
Approachresource-constrained inferencemultilayer representation learningdata-parallel computation
Inputtrained model, device constraintsraw data: images, text, audioparallelizable algorithm
Outputsmall, fast modellearned representation and predictionaccelerated program
Success measurelatency, energy, memoryaccuracy, loss, scalingspeedup, memory bandwidth
Math loadmediumhigh: derivatives, matrices, probabilitylow-medium
Data needstraining data already existslarge; needs GPUsnone
  • Deep Learning: Deep learning grows the model for accuracy; this course shrinks the same model while losing little accuracy.
  • GPU & Parallel Computing: GPU programming gets peak performance from large accelerators; this course runs inference on devices drawing a few milliwatts.

Differences

Where It Differs (2/2)

CriterionEmbedded AIMLOpsRobotics
Core questionHow do AI models run on small, low-power devices?How is a model taken from the lab into production and kept running there?How does a machine sense the physical world, know where it is and move?
Approachresource-constrained inferencesoftware engineering meets MLsense-plan-act loop
Inputtrained model, device constraintstrained model, data pipelinesensor measurements
Outputsmall, fast modela running, monitored servicemotor commands, path
Success measurelatency, energy, memorylatency, uptime, data drifttask success, safety
Math loadmediumlowhigh: transformations, control
Data needstraining data already existsstreaming production dataphysical experiments and simulation
  • MLOps: MLOps deploys and monitors models on servers; this course embeds the model in devices that may have no connection.
  • Robotics: Robotics treats perception, localization and motion as a whole; this course makes the robot's model run in real time.

Practice

Applications

Wake words

An always-listening model in speakers and phones.

Smart cameras

Object detection without sending images to the cloud.

Wearables

Fall, step and heart rhythm analysis on a watch.

Predictive maintenance

Catching fault signs in motor vibration.

On-phone AI

Photo enhancement, offline translation.

Farming and environment

Sound and pest detection on battery sensors.

Practice

Risks and Ethics

The hidden cost of shrinking

Quantization and pruning barely move average accuracy, yet rare classes and minority groups can lose far more.

Devices that cannot be updated

A faulty or vulnerable model spread over thousands of devices in the field is hard to recall.

Unnoticed surveillance

Always-listening microphones and cameras can monitor bystanders unawares, even when data stays on the device.

Model theft

The model file on a device can be read and copied, and reverse engineering opens it to adversarial attacks.

In depth: Explainable AI and AI Ethics.

Practice

Tools and Datasets

Tools and libraries

TensorFlow LiteONNX RuntimeExecuTorchEdge ImpulseCMSIS-NNArduinoNVIDIA Jetson

Well-known datasets

Speech CommandsVisual Wake WordsMNISTCIFAR-10ImageNetMLPerf Tiny
Success is not measured by accuracy alone: model size, peak memory, latency and energy go in the same table.

Practice

Common Misconceptions

Common beliefIn fact
Models are trained on the small device.Training is mostly done on a server; only inference runs on the device.
Quantization always ruins accuracy.8-bit quantization costs most models very little accuracy; quantization-aware training reduces the loss further.
Fewer parameters means a faster model.Speed depends on memory access, operation types and hardware support; parameter count is not enough.
Embedded AI is only needed without internet.For privacy, latency and energy, on-device processing is preferred even when a connection exists.

Next Steps

After This Course

Courses that build on or extend this one

This course is mostly an end point: it builds on the others.

Related careers

Embedded software engineerEdge AI engineerMachine learning engineerHardware-software co-designerIoT solutions architect
Embedded AI connects deep learning to hardware: from here you move on to robotics, autonomous vehicles and Internet of Things applications. For more depth in hardware, accelerator design and compilers come next; on the software side, MLOps and updating fleets of devices.

Next Steps

Further Reading

  • Warden and Situnayake, TinyML, O'Reilly, 2019.
  • Sze, Chen, Yang and Emer, Efficient Processing of Deep Neural Networks, Morgan & Claypool, 2020.
  • Han, Mao and Dally, "Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding", ICLR, 2016.
  • Howard et al., "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications", arXiv:1704.04861, 2017.
  • Hinton, Vinyals and Dean, "Distilling the Knowledge in a Neural Network", arXiv:1503.02531, 2015.

The End

Chapter 36: Embedded AI
← Previous chapter · Map of the field · Next chapter →
1 / 1 Sercan KÜLCÜ. All rights reserved.