Designing Data-Intensive Applications Part 4: Batch & Stream Processing

Up until now in our DDIA series, we have focused primarily on OLTP Request-Response Architectures. A client sends a request (e.g. searching for a product, placing an order), a database queries or updates a few indexed rows, and returns a response within milliseconds.

However, modern data systems frequently need to process vast volumes of data where a request-response loop is entirely inappropriate:

• How do you compute daily analytics across 50 billion user click events?
• How do you continuously update search indexes, machine learning models, and real-time fraud detection alerts as transactions occur?

In Part 4 of our Designing Data-Intensive Applications masterclass series, we explore Batch Processing (handling bounded historical data) and Stream Processing (handling unbounded real-time data).

1. Batch Processing: Computing over Bounded Data

A batch processing job takes a large volume of input data, runs a computation task to process or aggregate it, and writes an output dataset.

The key defining characteristic of batch processing is that it operates on bounded data—data of a known, fixed size that has a clear start and end.

The Philosophy of Batching: Lessons from Unix Pipes

Modern distributed batch processing frameworks were directly inspired by Unix text processing tools. In Unix, simple, specialized programs can be chained together using pipes (|):

cat access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -n 5

This simple Unix pipeline:

1. Reads web server access logs.
2. Extracts the URL path from each line (awk).
3. Sorts the URL lines alphabetically (sort).
4. Counts duplicate URLs (uniq -c).
5. Sorts the counts in reverse numerical order (sort -rn).
6. Emits the top 5 most visited pages (head -n 5).

Why Unix Pipes are Architecturally Brilliant:

• Uniform Interface: Everything is a stream of bytes (stdin/stdout).
• Separation of Logic: Small, reusable single-purpose tools.
• Immutability: Input files are never modified in-place; output is written to a new stream.

Distributed Batch Processing: MapReduce

When your access log is 50 Terabytes, a single machine’s CPU and disk cannot process it. We must distribute the work across a cluster of hundreds of nodes.

Pioneered by Google in 2004 and open-sourced via Apache Hadoop, MapReduce scales batch computations across thousands of machines.

INPUT FILE (HDFS Chunks)
   │
   ├───> [ MAPPER 1 ] ────> (Key, Value) ──┐
   ├───> [ MAPPER 2 ] ────> (Key, Value) ──┼──> [ SHUFFLE & SORT ] ──> [ REDUCER ] ──> OUTPUT
   └───> [ MAPPER 3 ] ────> (Key, Value) ──┘    (Group by Key)         (Aggregate)

The MapReduce Lifecycle:

1. Map Phase: Map tasks run in parallel on nodes where data blocks physically reside (data locality). The Mapper extracts key-value pairs from raw input records.
2. Shuffle and Sort Phase: This is the heart of MapReduce. The framework automatically partitions, network-transfers, and sorts all intermediate key-value pairs across the cluster so that all values sharing the exact same key land on the exact same Reducer node.
3. Reduce Phase: The Reducer processes the grouped values for each key and writes the final aggregated result to a distributed file system (HDFS).

Distributed Join Algorithms in Batch Engines

Joining two large datasets (e.g. User_Profiles and User_Click_Logs) across a distributed cluster is a complex operation. Batch engines use three primary join strategies:

1. Sort-Merge Join

Both datasets are partitioned by the join key (e.g. user_id) using the Map phase. During the Shuffle phase, records are sorted by user_id. Each Reducer receives the sorted records for a set of keys and merges them in a single linear scan!

2. Broadcast Hash Join (Map-Side Join)

If one dataset is small enough to fit completely into memory (e.g., a 10MB Countries lookup table) while the other is huge (1TB Events log), the batch engine broadcasts the small table to every Mapper node. Each Mapper loads the small table into an in-memory Hash Map and joins records on the fly without any expensive network shuffle!

3. Partitioned Hash Join

If both datasets are partitioned by the same join key and have the same number of partitions, Mappers only need to load corresponding partition pairs into local hash tables.

Modern Dataflow Engines: Apache Spark & Flink

While MapReduce was revolutionary, it had a major flaw: after every Map and Reduce phase, intermediate state was forcibly written to disk in HDFS. A complex pipeline required 5 chained MapReduce jobs, forcing 5 expensive disk I/O cycles.

Modern dataflow engines like Apache Spark and Apache Flink represent batch jobs as a Directed Acyclic Graph (DAG) of transformations. They keep intermediate dataset partitions in RAM whenever possible, resulting in 10x-100x speedups over traditional Hadoop MapReduce.

2. Stream Processing: Computing over Unbounded Data

In the real world, data does not stop arriving at midnight for a batch job to process. Transactions happen constantly, sensor logs stream continuously, users click perpetually.

Stream Processing operates on unbounded data—data that is continuously generated with no end in sight.

BATCH PROCESSING:   [ Fixed Bounded Dataset ] ──> Job Run ──> [ Output ]
STREAM PROCESSING:  ── Event 1 ── Event 2 ── Event 3 ──> Continuous Processing ──> Continuous Output

Message Brokers: AMQP Queues vs Log-Based Streams

To process streaming events, we need a Message Broker to transport events from producers to consumers.

There are two fundamental types of message brokers:

1. TRADITIONAL JMS / AMQP BROKER (RabbitMQ, ActiveMQ)
Producer ──> [ Queue ] ──> Consumer A (Deletes message on ACK!)
                      └──> Consumer B (Worker Queue style)

2. LOG-BASED STREAM BROKER (Apache Kafka, AWS Kinesis)
Producer ──> [ Partitioned Append-Only Log ]
             Offset 0 | Offset 1 | Offset 2 | Offset 3 | Offset 4
                ▲                     ▲
                │                     │
           Consumer Group A      Consumer Group B
           (Reads at Offset 1)   (Reads at Offset 3)

Key Differences:

Apache Kafka Architecture

Apache Kafka models messages as Partitions of an Append-Only Disk Log.

TOPIC: "user-purchases"
  ├── Partition 0: [Offset 0][Offset 1][Offset 2][Offset 3]... ──> Read by Consumer 1
  ├── Partition 1: [Offset 0][Offset 1][Offset 2]...          ──> Read by Consumer 2
  └── Partition 2: [Offset 0][Offset 1][Offset 2][Offset 3]... ──> Read by Consumer 3

1. A Topic represents a logical stream of events.
2. Topics are divided into Partitions spread across broker nodes for parallelism.
3. Every message inside a partition is assigned a sequential, monotonically increasing integer called an Offset.
4. Consumers track their own offset position. If a consumer crashes or a bug is deployed, you fix the bug, reset the offset back to 3 days ago, and replay the exact sequence of events!

Time in Stream Processing: Event Time vs Processing Time

A major trap in stream processing is confusing Event Time with Processing Time:

• Event Time: The timestamp when the event physically occurred on the device (e.g. 10:00:00.123 when a user clicked a button on a mobile phone).
• Processing Time: The timestamp when the stream processing engine processed the event (e.g. 10:05:30.000 when the event arrived at the server after a network outage).

If a mobile phone loses connection in a tunnel and uploads 5 hours of offline clicks at once, processing those events using Processing Time produces distorted analytics! Production stream processors always use Event Time alongside Watermarks to handle late-arriving data.

Windowing Strategies

Stream analytics require grouping continuous events into temporal boundaries called Windows:

1. TUMBLING WINDOW (Fixed size, Non-overlapping)
| 10:00 - 10:05 | 10:05 - 10:10 | 10:10 - 10:15 |

2. HOPPING / SLIDING WINDOW (Fixed size, Overlapping)
| 10:00 - 10:05 |
     | 10:01 - 10:06 |
          | 10:02 - 10:07 |

3. SESSION WINDOW (Dynamic size based on inactivity gap)
| User Active (Events) | ... (5 min gap) ... | User Active (Events) |

3. Event Sourcing & Change Data Capture (CDC)

One of the most transformative concepts in modern system design is treating events as the primary source of truth.

Change Data Capture (CDC)

In traditional web architectures, your application writes directly to PostgreSQL, and then attempts to manually update Redis, Elasticsearch, and analytics databases:

App Code ──> 1. WRITE Postgres
         ──> 2. WRITE Redis (Fails halfway due to network drop! Out of Sync!)
         ──> 3. WRITE Elasticsearch (Fails!)

This dual-write pattern introduces subtle data corruption.

Change Data Capture (CDC) solves this cleanly: The application writes only to the primary PostgreSQL database. A CDC tool (like Debezium) tails the database’s internal Write-Ahead Log (WAL), extracts every mutation, and streams those changes as events into Apache Kafka!

App ──> WRITE Postgres ──(WAL Log)──> Debezium CDC ──> Kafka Stream
                                                          │
                               ┌──────────────────────────┼──────────────────────────┐
                               ▼                          ▼                          ▼
                         Elasticsearch                  Redis                    Snowflake

Event Sourcing

Pioneered in Domain-Driven Design (DDD), Event Sourcing takes CDC a step further: instead of storing current application state, you store the full history of state-changing events.

• Traditional DB: Account Balance = $150
• Event Sourcing Log:
  • Event 1: Account Created
  • Event 2: Deposited $200
  • Event 3: Withdrew $50

The current state ($150) is simply a derived read projection computed by replaying the event log from offset 0! If business requirements change 6 months later, you can write a new projection function, replay the event log, and build an entirely new dataset retroactively!

Summary of Part 4

In Part 4, we mastered data processing paradigms:

1. Batch Processing works on bounded data using tools like MapReduce and Apache Spark, leveraging Map-Side and Reduce-Side joins.
2. Stream Processing handles unbounded real-time data using log-based brokers like Apache Kafka.
3. Kafka’s Offset Architecture enables consumer decoupling, fast parallel reads, and complete message replayability.
4. Event Time processing with watermarking is essential to avoid distorted metrics caused by network latency or offline devices.
5. Change Data Capture (CDC) and Event Sourcing turn the database WAL into an event stream, eliminating dual-write bugs and making data integration seamless.

Up next: Designing Data-Intensive Applications Part 5: The Future of Data Systems (Unbundling the Database & End-to-End Correctness).