Big Data Systems
Introduction
What Question Does It Ask?
Big data systems study how to store and process data that does not fit on one machine, or arrives faster than one machine can keep up, by spreading it over many machines. The course deals with the trade-offs between partitioning, replication, fault tolerance and consistency.
At a glance
distributed storage and computation
huge, fast-flowing data
processed data, query results
scalability, latency, cost
low
very large
Introduction
A Short History
Laney describes the data management challenge as volume, velocity and variety (the 3 Vs).
The Google File System paper: a fault-tolerant distributed file system built from cheap machines.
Dean and Ghemawat publish MapReduce: a simple programming model across thousands of machines.
Hadoop, an open-source counterpart of GFS and MapReduce, becomes its own Apache project.
Spark is released as open source: keeping intermediate results in memory speeds up iterative jobs.
LinkedIn open-sources Kafka: a distributed log system for event streams.
The Course
Core Concepts
The size of data, how fast it arrives, how many formats.
Splitting files into blocks and replicating them across machines.
Map, shuffle, reduce: moving computation to the data.
Spark: chaining steps without writing to disk.
Processing an unbounded event stream as it arrives.
Key-value, document, wide-column and graph stores.
Under a network partition, choosing consistency or availability.
Spreading load and not losing data when a machine fails.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- The scaling problem: scaling up and scaling out
- Distributed file systems: GFS and HDFS
- The MapReduce programming model
- Spark: RDDs, DataFrames and Spark SQL
- Data models: relational, document, wide-column
- Storage engines: LSM trees, B-trees, columnar files
- Partitioning and load balancing
- Replication and consistency models
- CAP, distributed transactions, consensus: Paxos, Raft
- Event streams: Kafka and publish-subscribe
- Stream processing: windows, event time, Flink
- Data lakes, warehouses and workflow orchestration
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
Word Count over Search Logs
A site's search logs do not fit on one machine; worker machines must count together how often each word was searched. Go step by step: Map → Shuffle → Reduce. Change the number of workers and the combiner, and watch the time.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Operating Systems | processes, threads, file systems and the cost of I/O |
| Object Oriented Programming with Java | Hadoop, Spark and Kafka run on the JVM |
| Databases and SQL | the relational model, indexes, transactions |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you explain the difference between a process and a thread? | An operating systems course: processes and concurrency |
| Can you write an SQL query with JOIN and GROUP BY? | SQL and the relational model from a database course |
| Can you write a program with classes and collections in Java or Python? | An object-oriented programming course in Java |
| Can you move files and run a script from the Linux command line? | A short introduction to the Linux command line |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| shares methods | Shares methods with Data Mining | scalable mining |
| shares methods | Shares methods with Machine Learning Systems and MLOps | data pipelines |
| shares methods | Shares methods with GPU Programming and Parallel Computing | parallel and distributed computing |
Differences
Where It Differs (1/2)
| Criterion | Big Data | Data Mining | Data Science |
|---|---|---|---|
| Core question | How is data that does not fit on one machine stored and processed? | How are hidden, useful patterns found in large piles of data? | How do we ask the right questions of data and turn the answers into decisions? |
| Approach | distributed storage and computation | discovery-driven analysis | question, data, analysis, story |
| Input | huge, fast-flowing data | large tables or transaction data | raw, messy data |
| Output | processed data, query results | rules, clusters, outliers | report, visualization, model |
| Success measure | scalability, latency, cost | interestingness, support, confidence | impact on decisions, accuracy |
| Math load | low | medium | medium: statistics |
| Data needs | very large | large | moderate to large |
- Data Mining: Data mining asks which patterns to look for; big data systems ask how that search runs on many machines.
- Data Science: Data science draws questions and conclusions from data; big data systems make that possible when data outgrows one machine.
Differences
Where It Differs (2/2)
| Criterion | Big Data | GPU & Parallel Computing | MLOps |
|---|---|---|---|
| Core question | How is data that does not fit on one machine stored and processed? | How do thousands of cores share the same computation? | How is a model taken from the lab into production and kept running there? |
| Approach | distributed storage and computation | data-parallel computation | software engineering meets ML |
| Input | huge, fast-flowing data | parallelizable algorithm | trained model, data pipeline |
| Output | processed data, query results | accelerated program | a running, monitored service |
| Success measure | scalability, latency, cost | speedup, memory bandwidth | latency, uptime, data drift |
| Math load | low | low-medium | low |
| Data needs | very large | none | streaming production data |
- GPU & Parallel Computing: A GPU speeds up computation with thousands of cores in one machine; big data systems spread data over machines, where network and disk are the bottleneck.
- MLOps: MLOps manages the model life cycle; big data systems build the data pipelines and storage that feed it.
Practice
Applications
Errors and traffic from server logs.
Tracking user behavior in real time.
The histories of millions of users.
Continuous measurements from sensors.
Real-time fraud alerts.
Crawling and indexing web pages.
Practice
Risks and Ethics
Misconfigured replication or Kafka records deleted when retention expires can destroy data without anyone noticing.
Collecting personal data because storage is cheap widens the breach surface; GDPR and similar laws require purpose limits.
A person's data spreads into backups, replicas and derived tables; fully honouring a deletion request is hard.
A cluster left running or an unnecessary full table scan means a large bill and wasted energy.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Well-known datasets
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| Big data just means a lot of data. | Velocity and variety matter as much as size; the real question is whether the data fits on one machine. |
| A distributed system is always faster. | On small data a single machine (pandas, SQL) is often faster; distribution has overhead. |
| Doubling the machines halves the time. | Shuffles, network traffic and the slowest task limit the speedup. |
| NoSQL has replaced SQL. | NoSQL offers different consistency trade-offs; SQL engines such as Spark SQL and Hive are widespread. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Kleppmann, Designing Data-Intensive Applications, O'Reilly, 2017.
- White, Hadoop: The Definitive Guide, 4th ed., O'Reilly, 2015.
- Chambers and Zaharia, Spark: The Definitive Guide, O'Reilly, 2018.
- Dean and Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters", OSDI, 2004.