Machine Learning Systems and MLOps
Introduction
What Question Does It Ask?
MLOps applies software engineering and DevOps practice to ML so that models reach production and run reliably there. The course covers versioning data, code and models together, automating the pipeline, deploying, and monitoring the model in production.
At a glance
software engineering meets ML
trained model, data pipeline
a running, monitored service
latency, uptime, data drift
low
streaming production data
Introduction
A Short History
The first DevOpsDays is held in Ghent; DevOps, uniting development and operations, spreads.
Docker 1.0 is released and Kubernetes announced: containers become the common unit of deployment.
Sculley et al. expose hidden technical debt in ML systems: model code is a small part of the system.
Google presents TFX and Uber presents Michelangelo; in-house ML platforms come into view.
Databricks open-sources MLflow: experiment tracking and a model registry.
Andrew Ng calls for data-centric AI: focus on data quality more than on the model.
The Course
Core Concepts
Automated steps from data preparation to deployment.
Tracking and rolling back data, code and model together.
Logging the parameters, metrics and outputs of every run.
Approved model versions kept in one place.
The same features used in training and in production.
Continuous integration, delivery and continuous training.
Running as an online API or as a batch prediction job.
Production data moving away from training data; must be monitored.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- ML systems in production: requirements, technical debt
- Data pipelines, data validation and versioning
- Feature engineering and feature stores
- Experiment tracking and reproducible training
- Model evaluation: slice analysis and model tests
- Packaging: containers and the model registry
- Model serving: online APIs and batch prediction
- Faster inference: latency, scale and cost
- Deployment strategies: canary, shadow, A/B tests
- Monitoring: data drift, concept drift, alerts
- Continuous training and CI/CD pipelines
- Responsible ML: auditing, documentation, model cards
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
The Model Silently Degrades in Production
A bank's fraud model flags unusually large transactions, while prices rise a little every week. Advance the weeks, watch drift (PSI) and accuracy, and retrain the model when the alarm goes off.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Machine Learning | deploying, monitoring and retraining models |
| Operating Systems | processes, memory and resource management: the concepts containers rely on |
| Object Oriented Programming with Java | the habit of writing modular, testable code |
| Software Engineering | version control (Git), testing and continuous integration |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you train a classifier and evaluate it with cross-validation? | Machine Learning |
| Can you create and merge branches in Git and resolve a merge conflict? | A short Git tutorial and a software engineering course |
| Can you manage processes, environment variables and permissions from the shell? | An operating systems course and an intro to the Linux shell |
| Can you write a modular Python project that has unit tests? | Object-oriented programming, then a short pytest tutorial |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| application | This course applies the methods of Machine Learning | deploying, monitoring and retraining models |
| shares methods | Shares methods with Data Science | experiment tracking, data versioning |
| shares methods | Shares methods with Big Data Systems | data pipelines |
| shares methods | Shares methods with Embedded AI | deploying and updating models on devices |
Differences
Where It Differs (1/2)
| Criterion | MLOps | Machine Learning | Big Data |
|---|---|---|---|
| Core question | How is a model taken from the lab into production and kept running there? | How does a program generalize from examples? | How is data that does not fit on one machine stored and processed? |
| Approach | software engineering meets ML | statistical learning | distributed storage and computation |
| Input | trained model, data pipeline | labeled or unlabeled examples | huge, fast-flowing data |
| Output | a running, monitored service | a model that makes predictions | processed data, query results |
| Success measure | latency, uptime, data drift | error on unseen data | scalability, latency, cost |
| Math load | low | high: linear algebra, probability, optimization | low |
| Data needs | streaming production data | moderate to large | very large |
- Machine Learning: ML teaches how to build a good model; this course teaches how to keep it running reliably in production for months.
- Big Data: Big data systems store and process data at scale; this course manages the lifecycle of models trained on that data.
Differences
Where It Differs (2/2)
| Criterion | MLOps | Data Science | Embedded AI |
|---|---|---|---|
| Core question | How is a model taken from the lab into production and kept running there? | How do we ask the right questions of data and turn the answers into decisions? | How do AI models run on small, low-power devices? |
| Approach | software engineering meets ML | question, data, analysis, story | resource-constrained inference |
| Input | trained model, data pipeline | raw, messy data | trained model, device constraints |
| Output | a running, monitored service | report, visualization, model | small, fast model |
| Success measure | latency, uptime, data drift | impact on decisions, accuracy | latency, energy, memory |
| Math load | low | medium: statistics | medium |
| Data needs | streaming production data | moderate to large | training data already exists |
- Data Science: Data science runs an analysis that answers a question; this course builds a service that runs, is monitored and is renewed.
- Embedded AI: Embedded AI fits the model onto a small device; this course deploys, versions and monitors it on servers.
Practice
Applications
Models answering millions of requests a day.
Decisions in milliseconds, frequent retraining.
Batch prediction jobs that run every night.
Models tested continuously with A/B tests.
Auditable, versioned and monitored models.
Serving large models at scale and at high cost.
Practice
Risks and Ethics
Without drift monitoring a model degrades with no errors; many demand forecasts lost their accuracy overnight in COVID-19.
If a feature is computed differently in training and production, a model that tested well predicts wrongly live.
A model retrained on the outcomes of its own decisions, such as a recommender or credit model, amplifies its own bias.
A model whose training data and code are unknown cannot be explained in a complaint or audit, nor rolled back.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Classic example scenarios
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| Once the model is trained, the job is done. | Data changes in production; an unmonitored model silently degrades over time. |
| MLOps is just installing tools. | Tools support the process; the real work is versioning, testing, ownership and rollback. |
| High test accuracy means good production performance. | Production data may differ from training data; latency and cost are part of success too. |
| Most of an ML system is the model. | Model code is a small part next to data, configuration, serving and monitoring infrastructure. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Huyen, Designing Machine Learning Systems, O'Reilly, 2022.
- Burkov, Machine Learning Engineering, True Positive Inc., 2020.
- Lakshmanan, Robinson and Munn, Machine Learning Design Patterns, O'Reilly, 2020.
- Sculley et al., "Hidden Technical Debt in Machine Learning Systems", NeurIPS, 2015.