Key Takeaways
Design data systems around guarantees, not database fashion or scale slogans.
Three tests for a system. A data-intensive application is judged by reliability, scalability, and maintainability. Reliability means continuing to do the right thing when hardware, software, or people fail. Scalability is not a badge a product either earns or lacks: it means having a workable response to a particular kind of growing load. Maintainability means making the system operable, understandable, and changeable by the people who inherit it.
Build for ordinary trouble. A study of large internet services found operator configuration mistakes caused more outages than hardware faults, which accounted for only 10 to 25 percent. Good systems therefore make errors easier to spot and reverse, with monitoring, safe testing environments, and gradual rollouts. The goal is not to prevent every fault, but to stop a fault from becoming a user-visible failure.
What's valuable here is the shift from product labels to observable guarantees. Reliability engineering in aviation and medicine uses the same discipline: define hazards, identify failure paths, and design checks around them. That approach also exposes a sociotechnical truth: software reliability depends on interfaces and operating practices, not only code. The limitation is that the three goals can conflict. For example, a highly automated system may reduce routine work while making unusual recovery harder. Design reviews should therefore ask not only whether the system meets its target, but who can diagnose and repair it when its assumptions fail.
Choose data models by relationships and access patterns, not by hype.
Shape follows relationships. A résumé with jobs, education, and contact details is naturally document-like: the pieces usually travel together, so storing them together can avoid extra queries. But when records link to many other records, such as people, schools, companies, and recommendations, a relational or graph model handles those connections more naturally. The history matters: document databases revive some strengths and weaknesses of older hierarchical databases, especially around many-to-many relationships and joins.
Schemas are a choice, not an absence. Schema-on-read means the database accepts varied records and the application interprets their structure when reading. Relational schema-on-write checks structure when data is stored. The former eases heterogeneous data; the latter makes expectations explicit. Declarative queries such as SQL also let an optimizer choose indexes and join order, unlike hand-coded traversal paths.
The practical question is not whether documents or tables are more modern, but where complexity should live. A document store may simplify writes while pushing joins, consistency work, and migration logic into application code. This resembles a broader principle in programming language design: abstractions do not eliminate complexity, they relocate it. A useful design exercise is to map the relationships that features are likely to introduce, not just the first screen the application must render. Neither schema flexibility nor strict validation is universally safer; the right choice depends on whether data variation is genuine or merely unmanaged.
Match storage engines to the work they perform most often.
Indexes trade writes for reads. An append-only log makes writes simple and sequential, but finding a key requires an index. A hash index can locate keys quickly when the keys fit in memory, while sorted-string tables and log-structured merge trees organize data into sorted files that are merged in the background. They often handle write-heavy workloads well. B-trees instead update fixed-size pages in place, making them a mature choice for many transactional databases and range queries.
Analytics asks a different question. Online transaction processing usually fetches a few records by key. Analytics often scans millions of rows while reading only a handful of columns. Column-oriented storage avoids loading unused fields and compresses repeated values. Neither design wins every benchmark: the actual read-write mix, data size, and latency target determine which engine fits.
This is a reminder that performance comes from matching data layout to access patterns, not from choosing the newest storage technology. The same idea appears in computer architecture: caches are effective when access has locality, and vectorized operations are effective when similar work can be performed on batches. The book also gives a useful warning against treating benchmark results as universal. A write-optimized engine may look excellent until compaction competes with production traffic. Performance tests should reflect realistic data distributions and include tail latency, not just average throughput.
Measure the workload before scaling, and design for the slowest requests.
Twitter's bottleneck was fan-out. In figures from 2012, Twitter handled about 4,600 tweet writes per second on average, but roughly 300,000 home-timeline reads per second. Precomputing timelines made reads cheap, yet each tweet then had to be copied to about 75 followers on average, or roughly 345,000 timeline writes per second. A celebrity with tens of millions of followers made that strategy costly, so Twitter used a hybrid: precompute ordinary users' timelines and fetch celebrity tweets when a timeline is read.
Averages conceal user pain. Measure response-time distributions, including p95 and p99, because one slow backend call can delay an entire request that depends on many services. Queueing often drives the worst delays, especially near capacity. Scalability begins by naming the load parameters that matter, then testing how those parameters affect throughput and response time.
The Twitter example turns architecture into arithmetic. The important insight is that moving work from reads to writes is not free; it changes where the system pays, and skew determines whether the change succeeds. Queueing theory reinforces the point: as utilization approaches capacity, waiting time can rise sharply even before average demand seems dramatic. Percentiles make that visible, although they should be interpreted with care across populations and time windows. A p99 target is meaningful only when the measurement method and the user-facing request boundary are clearly defined.
Treat replicas as potentially stale, and partitions as workload decisions.
Replication copies; partitioning divides. Replicas can improve availability, bring data closer to users, and serve more reads. With asynchronous replication, however, a follower may lag behind its leader. Users can see their own update disappear, observe data move backward in time, or encounter a reply before the question that caused it. Read-your-writes, monotonic reads, and consistent-prefix reads are distinct guarantees, not synonyms for eventual consistency.
Partition keys shape the load. Range partitioning supports efficient scans but can overload the newest time range. Hash partitioning spreads ordinary keys more evenly but scatters related keys and makes range queries expensive. A celebrity account can still become a hot key under either scheme. Secondary indexes add another trade-off: local indexes make writes simpler but may require queries to contact every partition; global indexes make reads more targeted but complicate updates.
A replica is not simply a spare copy. It is a copy with a timing relationship to other copies, and the application must decide which delays users can tolerate. This resembles the distinction between consistency and freshness in distributed caches: a result can be valid yet old. Partitioning likewise encodes assumptions about future queries and data distribution. The useful discipline is to test skew explicitly, since averages hide exceptional keys. Stronger guarantees can simplify application behavior, but requiring them for every read may waste the geographic and throughput benefits that replication was meant to provide.
Use transactions to protect invariants, not as a vague promise of safety.
ACID needs interpretation. Atomicity makes a group of writes all succeed or all roll back. Isolation controls interference between concurrent transactions. Durability protects committed data against specified failures. The consistency in ACID refers to application-defined invariants, such as balanced accounts; the database cannot infer every rule that makes an application valid.
Weak isolation leaves traps. Snapshot isolation lets a transaction read a stable view, often using multiple versions of each record. But two doctors can each see that two colleagues are on call, independently take themselves off duty, and both commit. The result violates the rule that one doctor must remain available. This write skew changes separate records, so preventing lost updates is not enough. Serializable isolation prevents such anomalies, though its implementations can add coordination or aborts. Distributed two-phase commit can coordinate commits across machines, but a failed coordinator can leave participants waiting.
The chapter's most useful correction is that an isolation-level name is not a sufficient specification. The concrete question is which interleavings the database forbids. This is closely related to formal methods, where correctness is expressed as invariants and allowed state transitions rather than reassuring labels. The doctors' schedule makes a subtle concurrency bug legible: each transaction is locally reasonable, while their combined result is unsafe. Teams should document business invariants and test the database's behavior against them. Transactions reduce error-handling complexity, but they do not repair incorrect application logic.
In a distributed system, a timeout signals uncertainty, not failure.
Silence has several causes. A request may be lost, delayed in a network queue, processed by a paused server, or completed with its reply lost. A timeout cannot distinguish these cases. Clocks add another uncertainty: machines drift, and a wall clock can jump. A process paused during garbage collection may resume believing an expired lease is still valid.
Make decisions collectively. Consensus protocols use overlapping votes to agree on leaders and ordered decisions despite crashes and network faults. Linearizability makes reads and writes appear to act on one current copy, but it can require waiting for coordination. CAP is often misrepresented as choosing any two of three properties. Its useful point is narrower: during a network partition, a system requiring linearizable results may have to reject or delay some requests. Fencing tokens help prevent an outdated lease holder from writing after a newer leader takes over.
The key conceptual move is to separate what a node believes from what the cluster has agreed. This is a form of epistemic humility built into protocol design: absence of evidence is not evidence of a crash. Consensus offers a disciplined way to make certain decisions, but it is not a universal ingredient for every data operation. The distinction between safety and progress is especially important: a system may stop accepting work to avoid making a corrupt decision. That is often preferable to a fast answer that silently violates an invariant.
Design data formats for old and new code to coexist.
Stored data outlives deployments. During rolling upgrades, new and old application versions may run together, and records written years ago may still be read today. Backward compatibility lets new code read old data; forward compatibility lets old code tolerate data written by newer code. JSON is convenient, but weakly specifies some types, including large integers. Language-specific serialization can tie data to one programming language and create security risks when decoding untrusted input.
Schema-based binary formats manage change explicitly. Protocol Buffers and Thrift identify fields by stable numeric tags, so names can change but tags must not be reused. Avro matches fields by name and resolves differences between the writer's and reader's schemas. In all cases, adding a field must be compatible with data and code that do not yet know it. Compatibility makes gradual deployment possible without demanding a single, risky system-wide update.
Compatibility is a temporal interface: it connects not just different services, but different moments in a system's life. This is why schema design resembles public API design, even when a format is used only internally. The strongest practice is to test old-reader and new-reader combinations before deployment, rather than assuming that a format's reputation guarantees safe evolution. There is a real trade-off: strict schemas improve validation and documentation, while flexible formats can absorb heterogeneous inputs. The engineering goal is explicit change, not change-free data.
Use durable logs to derive, replay, and synchronize specialized views.
One source can feed many tools. A system of record holds authoritative data; indexes, caches, recommendations, and analytics are derived views optimized for different uses. Updating each independently creates dual-write races: the database may receive changes A then B while a search index receives B then A. Change data capture avoids this by turning database writes into an ordered stream that downstream systems can consume.
Logs make recovery practical. A log-based broker retains messages and lets consumers track their position, so a consumer can restart or replay history. Event sourcing takes the idea further by recording meaningful application actions as immutable events and rebuilding current state from them. Deterministic processing and idempotent updates help prevent retries from applying changes twice. Batch jobs can rebuild large views; stream processors keep them fresh between rebuilds.
The log is powerful because it separates the authoritative event from the many representations built for convenience. This resembles accounting ledgers and version control: corrections are added as new facts rather than erasing the history that explains the current state. The approach can improve recovery and team independence, but replayability is not automatic. Old code, external side effects, schema changes, and data-retention requirements all complicate a supposedly simple replay. Durable history also creates privacy obligations, because retaining every event can make deletion difficult.
Protect data integrity end to end, and treat personal data as power.
Freshness and correctness differ. A derived view may be briefly stale and later catch up; corrupted or missing data usually does not repair itself. End-to-end operation IDs can suppress duplicate requests when a client retries after losing a response. Audits, checksums, and replayable derivations can reveal corruption that lower-level guarantees miss.
Engineering choices affect people. Predictive systems can reproduce historical discrimination, while feedback loops can deepen disadvantage: a poor score may restrict access to work, worsening the circumstances that produced the score. Behavioral tracking can also transfer control over private information from individuals to organizations. The book argues that consent is weak when people cannot understand data use or realistically opt out. Data collection should be limited to justified purposes, and systems should support accountability, correction, and deletion.
The connection between technical integrity and human rights is not ornamental. Data models determine what can be inferred, retained, corrected, and forgotten. Auditing supports operational trust, while accountability and appeal support social trust. Research on algorithmic fairness also shows that removing an explicit protected attribute does not remove proxies for it, such as location or purchasing patterns. At the same time, data analysis can support valuable services and public benefits. The hard question is not whether data use is good or bad in the abstract, but who gains control, who bears errors, and what remedies exist.
Analysis
Designing Data-Intensive Applications is best understood as an engineering guide to the costs hidden behind apparently simple operations: storing a value, reading a replica, adding an index, or retrying a request. Its organizing achievement is to replace product categories with durable questions about workload, failure, ordering, and change. The book moves from individual-machine foundations to distributed coordination, then to derived data and streams, making a case that the architecture of a modern application is often the interaction among specialized systems rather than one database.
Its deepest connective idea is that state is both a useful current view and the accumulated result of events. Once that distinction is clear, replication logs, CDC, event sourcing, batch recomputation, and stream processing become related techniques for moving information while preserving provenance. This is more than a performance strategy. It offers a way to recover from bugs, evolve schemas, and make dataflows intelligible to future engineers.
The argument is strongest when it insists on precise guarantees. Eventual consistency is not a complete application contract; ACID labels do not reveal which anomalies remain; and timeouts cannot prove remote failure. The book thereby teaches readers to ask what a system promises under specific faults, rather than accept slogans such as NoSQL, web-scale, or exactly-once.
Its proposals also need operational restraint. Event-driven architectures can multiply infrastructure, replay can reproduce harmful decisions, and asynchronous views may be unacceptable for high-stakes actions. The most appropriate design may be a simple relational database on one machine, not an elaborate distributed pipeline. Likewise, integrity mechanisms cannot decide which social objectives a predictive system should optimize. Technical rigor must include the consequences of data collection and the remedies available to affected people. The lasting lesson is disciplined composition: use the smallest set of tools that satisfies explicit requirements, make assumptions testable, and keep the flow of data understandable.
Review Summary
Designing Data-Intensive Applications is highly praised as an essential read for software engineers and developers. Readers appreciate its comprehensive coverage of data storage, distributed systems, and modern database concepts. The book is lauded for its clear explanations, practical examples, and insightful diagrams. Many consider it a mini-encyclopedia of data engineering, offering valuable knowledge for both beginners and experienced professionals. While some find certain sections challenging or overly academic, most agree it provides a solid foundation for understanding complex data systems and architectures.
People Also Read
Glossary
OLTP
Fast record-level operationsOnline transaction processing describes interactive workloads that read or modify a small number of records, usually located by key. Web applications, banking transactions, and inventory updates are typical examples. OLTP systems prioritize low response time and handling many concurrent requests, unlike analytic systems that scan large datasets to calculate aggregates.
OLAP
Large-scale data analysisOnline analytic processing describes workloads that scan large numbers of records to calculate summaries, comparisons, and aggregates. Analysts might use OLAP to compare sales by store, product, or time period. Such workloads often benefit from column-oriented storage because a query can read only the few columns it needs.
LSM-tree
Sorted, merged write structureA log-structured merge tree stores recent writes in memory, flushes them to sorted files, and merges those files in the background. This design favors sequential writes and can support high write throughput and range scans. Reads may need to check multiple files, and compaction must keep pace with incoming writes.
MVCC
Multiple versions for readersMulti-version concurrency control keeps multiple versions of data so transactions can read a consistent snapshot while other transactions continue writing. A transaction sees versions visible at its chosen point in time, while obsolete versions are later reclaimed. MVCC supports snapshot isolation and commonly allows readers and writers to proceed without blocking one another.
linearizability
Reads see the latest writeLinearizability makes a distributed data item behave as though it were a single copy updated atomically. Once a write completes, later reads must observe it. This strong recency guarantee simplifies application reasoning, but maintaining it may require coordination and can reduce availability or increase latency during network failures.
serializability
Transactions act one at a timeSerializability guarantees that concurrent transactions have the same overall effect as some execution in which each transaction ran to completion before the next began. It prevents race conditions such as write skew, where separate updates jointly violate an invariant. Serializability is distinct from linearizability, which concerns the recency of individual reads and writes.
change data capture
Publish database changesChange data capture observes writes made to a database and publishes them as events for other systems to consume. A search index, cache, or analytics store can apply the changes in order and stay derived from the authoritative database. CDC may read a database's replication log or use triggers, and consumers must account for lag and recovery.
event sourcing
Record actions as eventsEvent sourcing stores application-level events as an append-only history, then derives current application state by processing that history. Events represent meaningful actions, such as a reservation being made or cancelled, rather than only the resulting row changes. Replay can reconstruct state and aid debugging, but preserving and deleting historical data require deliberate policies.
schema-on-read
Interpret structure when readingSchema-on-read allows stored records to have varying structures, leaving the application to interpret fields when it reads them. It is useful when data is heterogeneous or controlled by external producers. It does not mean there is no schema: the reading code still assumes a structure, but the database does not enforce it at write time.
idempotence
Retries do not repeat effectsAn operation is idempotent when performing it multiple times has the same effect as performing it once. Setting a record to a fixed value is idempotent; incrementing a counter is not. Idempotence makes retries safer after failures or lost acknowledgments, often by carrying a unique operation identifier or message offset through the system.
CQRS
Separate write and read modelsCommand query responsibility segregation separates the model used to accept changes from the models optimized for reading. A write stream can feed distinct views, such as a search index, cache, or user-facing summary. This supports different access patterns and gradual evolution, but derived views may be temporarily stale and require reliable update and recovery mechanisms.
FAQ
What's Designing Data-Intensive Applications about?
- Focus on Data Systems: The book explores the principles and practices behind building reliable, scalable, and maintainable data-intensive applications. It covers various architectures, data models, and the trade-offs involved in designing these systems.
- Enduring Principles: Despite rapid technological changes, the book emphasizes fundamental principles that remain constant across different systems, equipping readers to make informed decisions about data architecture.
- Real-World Examples: Martin Kleppmann uses examples from successful data systems to illustrate key concepts, making complex ideas more accessible through practical applications.
Why should I read Designing Data-Intensive Applications?
- Comprehensive Overview: The book provides a thorough examination of data systems, making it suitable for software engineers, architects, and technical managers. It covers a wide range of topics, from storage engines to distributed systems.
- Improved Decision-Making: By understanding the trade-offs of various technologies, readers can make better architectural decisions for their applications, crucial for meeting performance and reliability requirements.
- Curiosity and Insight: For those curious about how data systems work, the book offers deep insights into the internals of databases and data processing systems, encouraging critical thinking about application design.
What are the key takeaways of Designing Data-Intensive Applications?
- Reliability, Scalability, Maintainability: The book emphasizes these three principles as essential for building robust data-intensive applications.
- Understanding Trade-offs: It highlights the importance of understanding trade-offs in system design, such as the CAP theorem, which states that "you can only pick two out of consistency, availability, and partition tolerance."
- Data Models and Replication: The choice of data model significantly impacts application performance, and the book discusses various replication strategies and their implications for consistency.
What are the best quotes from Designing Data-Intensive Applications and what do they mean?
- "Technology is a powerful force in our society.": This quote underscores the dual nature of technology, serving as a reminder of the ethical responsibilities in building data systems.
- "The truth is the log. The database is a cache of a subset of the log.": This encapsulates the idea of event sourcing, where the log of events is the authoritative source, and the database provides a read-optimized view.
- "If you understand those principles, you’re in a position to see where each tool fits in.": Highlights the importance of grasping fundamental principles to effectively utilize various technologies.
How does Designing Data-Intensive Applications define reliability, scalability, and maintainability?
- Reliability: Refers to the system's ability to function correctly even in the face of faults, involving design strategies to tolerate hardware failures, software bugs, and human errors.
- Scalability: Concerns how well a system can handle increased load, requiring strategies like partitioning and replication to cope with growth in data volume, traffic, or complexity.
- Maintainability: Focuses on how easily a system can be modified and updated over time, emphasizing simplicity, operability, and evolvability for productive team work.
What is the CAP theorem in Designing Data-Intensive Applications?
- Consistency, Availability, Partition Tolerance: The CAP theorem states that in a distributed data store, it is impossible to simultaneously guarantee all three properties.
- Trade-offs in Design: Emphasizes the trade-offs system designers must make, such as sacrificing availability during network failures to prioritize consistency and partition tolerance.
- Historical Context: Introduced by Eric Brewer in 2000, the theorem has significantly influenced the design of distributed systems.
How does Designing Data-Intensive Applications explain data models and query languages?
- Data Models: Compares various data models, including relational, document, and graph models, each with strengths and weaknesses, crucial for selecting the right one based on application needs.
- Query Languages: Discusses different query languages like SQL for relational databases and those for NoSQL systems, essential for effectively interacting with data.
- Use Cases: Emphasizes that different applications have different requirements, guiding informed decisions about data architecture.
What are the different replication methods in Designing Data-Intensive Applications?
- Single-Leader Replication: Involves one node as the leader processing all writes and replicating changes to followers, common but can lead to bottlenecks.
- Multi-Leader Replication: Allows multiple nodes to accept writes, improving flexibility and availability but introducing complexities in conflict resolution.
- Leaderless Replication: Any node can accept writes, improving availability but requiring careful management of consistency.
How does Designing Data-Intensive Applications address schema evolution?
- Schema Changes: Discusses the inevitability of application changes requiring corresponding data schema changes, emphasizing backward and forward compatibility.
- Encoding Formats: Explores various encoding formats like JSON, XML, and binary formats, highlighting trade-offs associated with each for schema evolution.
- Practical Strategies: Provides advice on handling schema changes in real-world applications, ensuring old and new data versions can coexist without issues.
What is the significance of event sourcing in Designing Data-Intensive Applications?
- Immutable Event Log: Involves storing all changes as an immutable log of events, allowing easy reconstruction of the current state by replaying the log.
- Separation of Concerns: Enables multiple views of data from the same log, allowing for easier application evolution over time.
- Auditability and Recovery: Provides a clear audit trail of changes, simplifying recovery from errors by rebuilding the state from the event log.
How does Designing Data-Intensive Applications propose handling network partitions?
- Network Faults: Explains that network partitions can lead to inconsistencies across replicas, complicating distributed system design.
- Handling Partitions: Discusses strategies like the CAP theorem, which states a system can only guarantee two of three properties: Consistency, Availability, and Partition Tolerance.
- Practical Implications: Emphasizes designing systems that tolerate network faults and continue operating effectively.
What are the ethical considerations in Designing Data-Intensive Applications?
- Responsibility of Engineers: Stresses the ethical implications of data collection and usage, including awareness of potential biases and discrimination in algorithms.
- Impact of Predictive Analytics: Discusses risks associated with predictive analytics, urging careful consideration of data-driven decisions and their consequences.
- Surveillance Concerns: Raises concerns about surveillance capabilities, advocating for user privacy, transparency, and control over personal data.
Download PDF
Download EPUB
.epub digital book format is ideal for reading ebooks on phones, tablets, and e-readers.