Designing Data-Intensive Applications Part 1: Foundations of Data Systems

When building modern web applications, the code you write in your web framework is rarely the hardest problem. The true engineering bottleneck is managing data.

How do you guarantee that a user’s financial balance is never lost when a hard drive crashes? How do you scale when a sudden viral event surges traffic by 100x? How do you organize data storage so that a team of 50 developers can evolve the application without breaking production?

In this 5-part masterclass series, we distill the core principles of Martin Kleppmann’s landmark book, Designing Data-Intensive Applications (DDIA).

In Part 1, we unpack the foundational architecture: Reliability, Scalability, Maintainability, Data Models, and Storage Engines (LSM-Trees vs B-Trees).

1. The Three Pillars of Data Systems

Many applications today are data-intensive rather than compute-intensive. Raw CPU power is rarely the limiting factor; instead, the challenges lie in the amount of data, the complexity of data, and the speed at which it changes.

Any data system must satisfy three fundamental requirements:

                  ┌─────────────────────────────────────────┐
                  │      DATA-INTENSIVE APPLICATION         │
                  └────────────────────┬────────────────────┘
                                       │
         ┌─────────────────────────────┼─────────────────────────────┐
         ▼                             ▼                             ▼
  ┌──────────────┐              ┌──────────────┐              ┌──────────────┐
  │ RELIABILITY  │              │ SCALABILITY  │              │MAINTAINABILITY│
  │ Faults ≠     │              │ Coping with  │              │ Operability, │
  │ Failures     │              │ Load Growth  │              │ Simplicity   │
  └──────────────┘              └──────────────┘              └──────────────┘

Pillar 1: Reliability (Tolerating Faults)

Reliability means that the system continues to work correctly (performing the desired function at the expected performance level) even in the face of adversity (hardware faults, software bugs, human error).

It is crucial to distinguish between a fault and a failure:

• A fault is defined as one component of the system deviating from its spec.
• A failure is when the system as a whole stops providing the required service to the user.

We cannot prevent faults entirely, but we can design fault-tolerant architectures that prevent faults from triggering system failures.

Types of Faults:

1. Hardware Faults: Hard drives crash, RAM corrupts, power grids drop out. Mitigation: Dual power supplies, RAID arrays, hot-swappable CPUs, and multi-node redundancy.
2. Software Errors: Systematic bugs that lie dormant until triggered by an edge case (e.g., a runaway process consuming all CPU, cascading failures). Mitigation: Process isolation, thorough testing, telemetry monitoring, and circuit breakers.
3. Human Error: Humans configure systems incorrectly. Studies show human operational error is the leading cause of production outages. Mitigation: Decouple environments (staging vs production), provide sandbox testing, enable fast rollbacks, and automate deployment pipelines.

Pillar 2: Scalability (Coping with Growth)

Scalability is the term we use to describe a system’s ability to cope with increased load. It is not a single one-dimensional metric (“Is system X scalable?”); rather, it is answered by asking: “If the load grows in a specific way, what are our options for adding compute to maintain performance?”

Describing Load: Load Parameters

First, you must quantify load using load parameters:

• Requests per second to a web server
• Ratio of reads to writes in a database
• Simultaneous active users in a chat room
• Hit rate on a cache

Case Study: Twitter’s Timeline Architecture (Fan-out)

Twitter faced a famous scalability challenge with its timeline rendering:

• Operation 1 (Post Tweet): A user posts a new tweet (12k requests/sec average, 30k/sec peak).
• Operation 2 (Home Timeline): A user views their home timeline (300k requests/sec).

Twitter evaluated two main architectural approaches to handle this fan-out load:

APPROACH A: Relational Join on Read (Old Twitter)
User Posts Tweet ──> Insert into Global Tweets Table
User Views Timeline ──> SELECT * FROM tweets JOIN follows WHERE follows.follower_id = user.id
Result: Cheap Writes, Extremely Expensive Reads at 300k req/sec!

APPROACH B: Fan-Out Cache on Write (Modern Twitter)
User Posts Tweet ──> Lookup Followers ──> Insert Tweet into EVERY Follower's Redis Timeline
User Views Timeline ──> READ Redis Timeline Cache (O(1) lookup!)
Result: Extremely Fast Reads, Expensive Writes for High-Follower Users (e.g. celebrities)!

Twitter eventually adopted a hybrid model: For 99% of users, they use Approach B (fan-out on write). However, for celebrities with tens of millions of followers, tweets are excluded from fan-out and are instead merged on-read into the follower’s timeline!

Describing Performance

When load increases, how is performance impacted?

• In batch processing systems, we look at throughput (number of records processed per second).
• In online real-time systems, we measure response time (the time between a client sending a request and receiving a response).

Important: Always measure response times using percentiles (p50, p95, p99, p999), never averages! The average hides outliers. A p99 response time of 2 seconds means that 1 out of 100 requests takes 2+ seconds. For e-commerce sites (like Amazon), p999 users are often the most valuable customers with the largest shopping carts.

Pillar 3: Maintainability (Operability, Simplicity, Evolvability)

Software maintenance accounts for the vast majority of a system’s lifetime cost. To minimize engineering pain, design for:

1. Operability: Make it easy for operations teams to keep the system running smoothly (good telemetry, clear operational docs, standard defaults).
2. Simplicity: Abstract away accidental complexity. Keep architecture intuitive so new engineers can understand it easily.
3. Evolvability (Extensibility): Make it easy to modify the system in the future when requirements change (agile, modular design).

2. Data Models and Query Languages

A data model is perhaps the single most important part of building software because it affects not only how the software is written, but how we think about the problem we are solving.

┌─────────────────────────────────────────────────────────────────┐
│                    APPLICATION CODE (Objects)                   │
├─────────────────────────────────────────────────────────────────┤
│           DATA MODEL (Relational / Document / Graph)            │
├─────────────────────────────────────────────────────────────────┤
│            PHYSICAL DATA FORMAT (SSTables, B-Trees)             │
├─────────────────────────────────────────────────────────────────┤
│                 HARDWARE (Bytes on Disk / Flash)                │
└─────────────────────────────────────────────────────────────────┘

Relational Model vs Document Model

The Relational Model (SQL, formalized by Edgar Codd in 1970) organizes data into relations (tables), where each relation is an unordered collection of tuples (rows).

The Document Model (NoSQL, e.g. MongoDB) targets applications where data comes in self-contained documents, and relationships between documents are rare.

Object-Relational Impedance Mismatch

If data is stored in relational tables, an awkward translation layer is required between the objects in application code (classes, arrays) and database tables/columns. ORMs (like Hibernate, Prisma, TypeORM) reduce boilerplates, but cannot completely hide the architectural mismatch.

Graph-Like Data Models

If your application has mostly many-to-many relationships and complex interconnected entities (like a social network, fraud detection network, or knowledge graph), relational models require agonizing multi-join queries, and document models break down entirely.

A Graph Data Model consists of two atomic objects:

1. Vertices (Nodes / Entities)
2. Edges (Relationships / Links)

      (Person: Alice) ───[:FRIEND_WITH]───> (Person: Bob)
             │                                    │
       [:WORKS_AT]                            [:LIVES_IN]
             │                                    │
             ▼                                    ▼
    (Company: Acme Corp) ───[:LOCATED_IN]───> (City: New York)

Cypher Query Language Example

Cypher is a declarative query language for property graphs (popularized by Neo4j):

MATCH (person:Person)-[:LIVES_IN]->(city:City {name: 'New York'})
MATCH (person)-[:WORKS_AT]->(company:Company)
RETURN person.name, company.name

Declarative query languages (like SQL and Cypher) are superior to imperative code because the database query optimizer can reorder execution steps, parallelize execution, and select indexes automatically without code changes.

3. Storage & Retrieval: Under the Hood of Databases

How does a database physically store the data you write, and how does it retrieve it efficiently?

At the most primitive level, a database is just a file where you append records. Consider a 2-line shell script key-value store:

db_set () {
    echo "$1,$2" >> database.db
}

db_get () {
    grep "^$1," database.db | sed -e "s/^$1,//" | tail -n 1
}

• db_set has incredible performance (O(1) append-only write).
• db_get has terrible performance (O(n) linear search through the whole file).

To speed up reads, databases construct Indexes. An index is an additional side-structure derived from the primary data. Adding an index speeds up reads, but slows down writes because every index must be updated whenever data is written!

Hash Indexes (Bitcask Engine)

Keep an in-memory hash map where every key maps to a byte offset in an append-only log file on disk.

IN-MEMORY HASH TABLE                 ON-DISK APPEND-ONLY LOG
┌─────────────┬─────────────┐        ┌────────────────────────────┐
│ Key         │ Offset      │        │ 000: {"user":"alice","age":28}
├─────────────┼─────────────┤        ├────────────────────────────┤
│ "user:123"  │ 000         │ ────>  │ 064: {"user":"bob","age":34}  
│ "user:456"  │ 064         │        ├────────────────────────────┤
└─────────────┴─────────────┘        │ 128: {"user":"alice","age":29} <── Update

• Compaction: When the log gets large, merge segments and throw away duplicate keys, retaining only the latest update per key.
• Limitation: All keys must fit in RAM. If you have billions of keys, the hash table overflows RAM. Range queries (e.g. find all keys between 100 and 200) are also impossible.

SSTables and LSM-Trees (Log-Structured Merge-Trees)

To solve the limitations of Hash Indexes, we require that the sequence of key-value pairs in the log files is sorted by key. This data format is called a Sorted String Table (SSTable).

How LSM-Tree Storage Engines Work (LevelDB, RocksDB, Cassandra):

1. Write arrives ──> Written to In-Memory Balanced Tree (MemTable)
                           │
                           ▼ (When MemTable reaches ~2MB)
2. Flush to Disk ──> Written as SSTable File on Disk (Sorted by Key)
                           │
                           ▼
3. Background ───> Merging & Compaction (Mergesort of SSTables)

           [IN-MEMORY]
            MemTable (Red-Black Tree or SkipList)
               │
   ┌───────────┴───────────┐ (Flush to Disk)
   ▼                       ▼
[DISK Level 0]        [DISK Level 0]
SSTable 1             SSTable 2
(Sorted: A..M)        (Sorted: N..Z)
   │                       │
   └───────────┬───────────┘ (Compaction)
               ▼
[DISK Level 1] Merged SSTable (Sorted & De-duplicated)

Why LSM-Trees Are Fast:

• Writes are sequential disk I/O (extremely fast on both HDDs and NVMe SSDs).
• Range queries are fast because files are sorted.
• Bloom Filters: To prevent checking every SSTable file for a non-existent key, LSM-trees use Bloom Filters (probabilistic memory structures that quickly tell you if a key definitely does not exist).

B-Trees (The Standard Relational Storage Engine)

While LSM-trees are popular in NoSQL databases, B-Trees remain the standard index structure in almost all relational databases (PostgreSQL, MySQL InnoDB, Oracle).

Unlike LSM-trees, which write variable-size segment files sequentially, B-Trees break the database down into fixed-size pages (traditionally 4KB) and read/write one page at a time.

                          [ROOT PAGE]
                        [ 100 | 250 ]
                       /      |      \
         ┌────────────┘       │       └────────────┐
         ▼                    ▼                    ▼
   [LEAF PAGE 1]        [LEAF PAGE 2]        [LEAF PAGE 3]
   Keys: 0..99          Keys: 100..249       Keys: 250..999

• Each page contains keys and references to child pages.
• Finding a key requires traversing O(log n) page references from root to leaf.
• Write-Ahead Log (WAL): To make B-Tree writes crash-resilient (since modifying a page in-place could crash halfway through), every page modification is first written to an append-only WAL file before modifying the page on disk.

Comparing LSM-Trees vs B-Trees

4. OLTP vs OLAP (Column-Oriented Storage)

In the early days of databases, the same database served both transactional applications and management analytics. Today, we decouple these systems:

• OLTP (Online Transaction Processing): High volume of small queries (reads/writes by key), low latency, user-facing applications.
• OLAP (Online Analytics Processing): Low volume of huge queries aggregating millions of rows (SUM, AVG, COUNT), analytics data warehouses (Snowflake, BigQuery, ClickHouse).

Column-Oriented Storage Layout

Relational OLTP databases store data row-by-row:

Row 1: [ID:1, Name:Alice, Age:28, Salary:85000, Dept:Engineering] Row 2: [ID:2, Name:Bob, Age:34, Salary:92000, Dept:Sales]

If an OLAP analytics query runs: SELECT AVG(Salary) FROM employee, a row-oriented database must load all columns for all millions of rows from disk into memory, wasting massive I/O bandwidth!

An OLAP Column-Oriented Database stores all values from each column together on disk:

Salary File: [85000, 92000, 105000, 78000, ...]
Age File:    [28,    34,    41,     29,    ...]
Dept File:   [Eng,   Sales, Eng,    HR,    ...]

Now, computing AVG(Salary) only loads the small Salary file from disk!

Column Compression (Bitmap Encoding)

Columnar storage allows immense compression ratios (e.g. 10x-100x) using Bitmap Encoding and Run-Length Encoding because values in a column repeat frequently.

Summary of Part 1

In this foundational deep dive, we learned:

1. Reliability means tolerating faults to prevent complete system failures.
2. Scalability requires describing load parameters and measuring performance with percentiles (p99), not averages.
3. Maintainability requires operability, simplicity, and evolvability.
4. LSM-Trees use append-only SSTables for blazing-fast writes, while B-Trees overwrite 4KB pages in-place for predictable read performance.
5. Column-Oriented Storage fuels modern OLAP data warehouses by reading only the target columns necessary for big data analytics.

Up next: Designing Data-Intensive Applications Part 2: Distributed Data (Replication, Partitioning, & Quorums).