Big Data Systems

Guide to AI and Data Courses · Part IV: Data
The three VsDistributed storageMapReduce and SparkStream processingNoSQL and CAPNeighboring courses
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

What Question Does It Ask?

How is data that does not fit on one machine stored and processed?

Big data systems study how to store and process data that does not fit on one machine, or arrives faster than one machine can keep up, by spreading it over many machines. The course deals with the trade-offs between partitioning, replication, fault tolerance and consistency.

At a glance

Approach

distributed storage and computation

Input

huge, fast-flowing data

Output

processed data, query results

Success measure

scalability, latency, cost

Math load

low

Data needs

very large

Introduction

A Short History

2001

Laney describes the data management challenge as volume, velocity and variety (the 3 Vs).

2003

The Google File System paper: a fault-tolerant distributed file system built from cheap machines.

2004

Dean and Ghemawat publish MapReduce: a simple programming model across thousands of machines.

2006

Hadoop, an open-source counterpart of GFS and MapReduce, becomes its own Apache project.

2010

Spark is released as open source: keeping intermediate results in memory speeds up iterative jobs.

2011

LinkedIn open-sources Kafka: a distributed log system for event streams.

The Course

Core Concepts

Volume, velocity, variety

The size of data, how fast it arrives, how many formats.

Distributed file system

Splitting files into blocks and replicating them across machines.

MapReduce

Map, shuffle, reduce: moving computation to the data.

In-memory processing

Spark: chaining steps without writing to disk.

Stream processing

Processing an unbounded event stream as it arrives.

NoSQL

Key-value, document, wide-column and graph stores.

CAP theorem

Under a network partition, choosing consistency or availability.

Partitioning, replication

Spreading load and not losing data when a machine fails.

The Course

A Typical Workflow

CollectKafka,event streamsStoreHDFS, objectstorageProcessSpark, FlinkQuerydistributed SQLServedashboard,model, report
Example: A video site collects billions of view events a day with Kafka, partitions them by date in object storage, computes nightly per-user viewing summaries with Spark and writes the results to tables that analysts can query with SQL.

The Course

What Does the Course Cover?

  1. The scaling problem: scaling up and scaling out
  2. Distributed file systems: GFS and HDFS
  3. The MapReduce programming model
  4. Spark: RDDs, DataFrames and Spark SQL
  5. Data models: relational, document, wide-column
  6. Storage engines: LSM trees, B-trees, columnar files
  7. Partitioning and load balancing
  8. Replication and consistency models
  9. CAP, distributed transactions, consensus: Paxos, Raft
  10. Event streams: Kafka and publish-subscribe
  11. Stream processing: windows, event time, Flink
  12. Data lakes, warehouses and workflow orchestration

Skills you will gain

writing distributed jobs in PySparkwriting Kafka producers and consumerscolumnar storage with Parquetdata modelling in Cassandra or MongoDBdiagnosing a slow Spark job

A typical semester; the order and coverage vary between universities.

Try It

Word Count over Search Logs

A site's search logs do not fit on one machine; worker machines must count together how often each word was searched. Go step by step: Map → Shuffle → Reduce. Change the number of workers and the combiner, and watch the time.

A big data course teaches moving computation to the data: Map and Reduce tasks run independently, so adding workers speeds things up, but network traffic and the slowest worker set a limit. The combiner cuts the number of pairs on the network.

Context

Prerequisites

Courses worth knowing before this one, and why they are needed.

CourseWhy it is needed
Operating Systemsprocesses, threads, file systems and the cost of I/O
Object Oriented Programming with JavaHadoop, Spark and Kafka run on the JVM
Databases and SQLthe relational model, indexes, transactions

Context

Are You Ready for This Course?

Ask yourself; for every "no", the right-hand column shows where to start.

Ask yourselfIf not, start with
Can you explain the difference between a process and a thread?An operating systems course: processes and concurrency
Can you write an SQL query with JOIN and GROUP BY?SQL and the relational model from a database course
Can you write a program with classes and collections in Java or Python?An object-oriented programming course in Java
Can you move files and run a script from the Linux command line?A short introduction to the Linux command line
If you can say "yes" to most of these, you are ready; one or two gaps can be closed during the semester.

Context

Neighbouring Courses

subfieldbuilds onshares methodsapplication

Overlap

Overlapping Courses

RelationCourseShared ground
shares methodsShares methods with Data Miningscalable mining
shares methodsShares methods with Machine Learning Systems and MLOpsdata pipelines
shares methodsShares methods with GPU Programming and Parallel Computingparallel and distributed computing

Differences

Where It Differs (1/2)

CriterionBig DataData MiningData Science
Core questionHow is data that does not fit on one machine stored and processed?How are hidden, useful patterns found in large piles of data?How do we ask the right questions of data and turn the answers into decisions?
Approachdistributed storage and computationdiscovery-driven analysisquestion, data, analysis, story
Inputhuge, fast-flowing datalarge tables or transaction dataraw, messy data
Outputprocessed data, query resultsrules, clusters, outliersreport, visualization, model
Success measurescalability, latency, costinterestingness, support, confidenceimpact on decisions, accuracy
Math loadlowmediummedium: statistics
Data needsvery largelargemoderate to large
  • Data Mining: Data mining asks which patterns to look for; big data systems ask how that search runs on many machines.
  • Data Science: Data science draws questions and conclusions from data; big data systems make that possible when data outgrows one machine.

Differences

Where It Differs (2/2)

CriterionBig DataGPU & Parallel ComputingMLOps
Core questionHow is data that does not fit on one machine stored and processed?How do thousands of cores share the same computation?How is a model taken from the lab into production and kept running there?
Approachdistributed storage and computationdata-parallel computationsoftware engineering meets ML
Inputhuge, fast-flowing dataparallelizable algorithmtrained model, data pipeline
Outputprocessed data, query resultsaccelerated programa running, monitored service
Success measurescalability, latency, costspeedup, memory bandwidthlatency, uptime, data drift
Math loadlowlow-mediumlow
Data needsvery largenonestreaming production data
  • GPU & Parallel Computing: A GPU speeds up computation with thousands of cores in one machine; big data systems spread data over machines, where network and disk are the bottleneck.
  • MLOps: MLOps manages the model life cycle; big data systems build the data pipelines and storage that feed it.

Practice

Applications

Log analysis

Errors and traffic from server logs.

Clickstream

Tracking user behavior in real time.

Recommendation backends

The histories of millions of users.

Internet of Things

Continuous measurements from sensors.

Financial transactions

Real-time fraud alerts.

Search index

Crawling and indexing web pages.

Practice

Risks and Ethics

Silent data loss

Misconfigured replication or Kafka records deleted when retention expires can destroy data without anyone noticing.

Keep-everything habits

Collecting personal data because storage is cheap widens the breach surface; GDPR and similar laws require purpose limits.

Honouring the right to erasure

A person's data spreads into backups, replicas and derived tables; fully honouring a deletion request is hard.

Cloud bills and energy

A cluster left running or an unnecessary full table scan means a large bill and wasted energy.

In depth: Explainable AI and AI Ethics.

Practice

Tools and Datasets

Tools and libraries

Apache HadoopApache SparkApache KafkaApache FlinkApache CassandraMongoDBApache Hive

Well-known datasets

Common CrawlWikipedia dumpsNYC TLC taxi recordsGDELTGoogle Books Ngram

Practice

Common Misconceptions

Common beliefIn fact
Big data just means a lot of data.Velocity and variety matter as much as size; the real question is whether the data fits on one machine.
A distributed system is always faster.On small data a single machine (pandas, SQL) is often faster; distribution has overhead.
Doubling the machines halves the time.Shuffles, network traffic and the slowest task limit the speedup.
NoSQL has replaced SQL.NoSQL offers different consistency trade-offs; SQL engines such as Spark SQL and Hive are widespread.

Next Steps

After This Course

Courses that build on or extend this one

This course is mostly an end point: it builds on the others.

Related careers

Data engineerBig data architectData platform engineerCloud engineer
Big data systems are the infrastructure of the data courses. From here you can move to MLOps, which takes models to production, to GPU and parallel computing, which speeds up computation, or to the data mining and data science that run on top of this infrastructure.

Next Steps

Further Reading

  • Kleppmann, Designing Data-Intensive Applications, O'Reilly, 2017.
  • White, Hadoop: The Definitive Guide, 4th ed., O'Reilly, 2015.
  • Chambers and Zaharia, Spark: The Definitive Guide, O'Reilly, 2018.
  • Dean and Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters", OSDI, 2004.

The End

Chapter 27: Big Data Systems
← Previous chapter · Map of the field · Next chapter →
1 / 1 Sercan KÜLCÜ. All rights reserved.