Start free trial
Trending New Genres Authors
EnglishEnglish
EspañolSpanish
简体中文Chinese
繁體中文Chinese (Traditional)
FrançaisFrench
DeutschGerman
日本語Japanese
PortuguêsPortuguese
ItalianoItalian
한국어Korean
РусскийRussian
NederlandsDutch
العربيةArabic
PolskiPolish
हिन्दीHindi
Tiếng ViệtVietnamese
SvenskaSwedish
ΕλληνικάGreek
TürkçeTurkish
ไทยThai
ČeštinaCzech
RomânăRomanian
MagyarHungarian
УкраїнськаUkrainian
IndonesiaIndonesian
DanskDanish
SuomiFinnish
БългарскиBulgarian
עבריתHebrew
NorskNorwegian
HrvatskiCroatian
CatalàCatalan
SlovenčinaSlovak
LietuviųLithuanian
SlovenščinaSlovenian
СрпскиSerbian
EestiEstonian
LatviešuLatvian
فارسیPersian
മലയാളംMalayalam
தமிழ்Tamil
اردوUrdu
Searching...
SoBrief
Fundamentals of Data Engineering

Fundamentals of Data Engineering

Plan and Build Robust Data Systems
Build data systems people can trust, without paying for complexity your users do not need.
by Joe Reis 2022 426 pages
4.16
500+ ratings
Listen
Immersive
V2.3
Amazon Kindle Audible
Try Full Access for 3 Days
Unlock listening & more!
Continue

Key Takeaways

Manage the whole data journey, not a fashionable collection of tools

The work is connective. Data engineering develops and maintains systems that turn raw inputs into dependable information for analysts, data scientists, machine learning engineers, and other users. The book organizes this work as a lifecycle: data is generated in source systems, stored, ingested, transformed, and served. These stages overlap, repeat, and depend on one another. Storage, for example, is not a single stop on the route; it underlies much of the journey.

Foundations unlock analysis. The authors describe moving from data science into engineering after finding that useful models were blocked by unreliable data collection, cleaning, access, and infrastructure. They cite estimates that data scientists spend 70% to 80% of their time on data preparation, while noting that the figure is debated and varies by organization. The central point holds: sophisticated analysis needs reliable groundwork.

Analysis

The lifecycle is a useful antidote to tool-first thinking because it makes dependencies visible. A failure at collection can quietly undermine a polished dashboard or model several steps downstream. This resembles systems engineering, where a component is judged not only by its local performance but by its contribution to the whole system. The framework also keeps the user in view: data has little practical value merely because it moved successfully. Its limitation is that a neat sequence can imply more order than real projects possess. The authors recognize that stages overlap, so the lifecycle works best as a map for diagnosis, not a rigid workflow.

Start with the decision data must improve, not the dataset

Begin with a use. A company can collect huge volumes of information and still have no useful data capability. The authors warn against data vanity projects, such as lakes filled with material nobody can find or use. Begin instead by asking who needs the data, what they will do with it, and what business result would justify the work.

Match the company's maturity. The book describes three broad stages: starting with data, scaling with data, and leading with data. Early teams often need generalists and quick, visible wins, while accepting that speed may create technical debt. Scaling organizations need repeatable practices and systems. Mature organizations can support self-service use, governance, and specialized roles. The authors caution against jumping to machine learning before reliable data foundations exist.

Analysis

This user-first framing connects data engineering to product design: a system earns investment when it helps someone complete a meaningful task. It also prevents a common category error, treating storage capacity or model sophistication as evidence of business value. The maturity stages help leaders set realistic expectations, since a small team cannot safely imitate the operating model of a technology giant. Still, maturity is not a ladder that every organization should climb in the same way. Regulation, product strategy, and the cost of delay can make a focused advanced use case sensible even when other data practices remain immature.

Negotiate with source systems before their changes break your pipeline

Know what creates the data. Source systems include application databases, APIs, files, logs, sensors, and event streams. Data engineers often do not control these systems, so they need to understand how data is produced, how quickly it changes, what its schema looks like, and whether reading it could slow a live application. A schema change or missed update can ripple through every downstream stage.

Choose an ingestion pattern deliberately. Batch processing groups data into scheduled or size-based deliveries. Streaming makes new events available continuously, but adds cost and operational complexity. Change data capture, or CDC, records database changes for downstream use. The authors recommend choosing real-time ingestion only when faster data enables a meaningful action. Clear agreements with source owners can define expected fields, delivery methods, frequency, and contacts.

Analysis

The emphasis on relationships is more than workplace diplomacy. It treats data production as a shared responsibility across organizational boundaries, where a technical interface also needs a human process for communicating change. This is especially important because schema compatibility alone cannot establish what a field means or whether a new value is valid. Formal agreements can reduce ambiguity, but they should not become paperwork detached from engineering practice. A lightweight notification and escalation process may protect a pipeline better than a detailed contract nobody updates.

Keep architecture modular enough to change your mind cheaply

Separate strategy from tools. Architecture describes the systems and choices needed to meet business needs. Technologies are the means of implementing that design. The authors argue that teams get into trouble when they select impressive tools before understanding the problem. A sound design weighs reliability, scale, security, cost, and ease of change rather than chasing a universal best solution.

Preserve options. Common components can make systems easier to share and operate, while loosely connected services let teams change parts independently. Reversible decisions lower the cost of learning: a team can try a service, evaluate it, and replace it if it does not fit. The book also recommends planning for failure and scaling only as demand warrants. Build custom systems when they create a real advantage; otherwise, consider managed or existing tools.

Analysis

Reversibility is a practical response to uncertainty, not an excuse to avoid commitment. Modular boundaries create room to adapt, but every interface adds coordination and failure modes. The right design therefore depends on the cost of change on both sides: a tightly integrated service may be simpler and cheaper for a small, stable workload, while a modular system may protect a fast-changing business from lock-in. The authors' warning against unnecessary complexity is particularly valuable for small teams, whose scarce resource is often attention rather than computing capacity.

Choose storage and models around access patterns, not ideology

Storage has a job. Frequently used hot data needs faster access than archival cold data, which is rarely retrieved. Object storage is economical and well suited to large files and analytics workloads, but it is not a natural fit for constant small updates. Columnar formats help analytical queries scan selected fields efficiently; row-oriented databases are generally better suited to frequent individual lookups and changes.

Models preserve meaning. A data warehouse structures information for analysis. A data lake can hold diverse raw material, but without management it may become difficult to discover or trust. A data lakehouse combines object storage with table management and update capabilities. The authors compare normalized, dimensional, and data vault approaches, each with different trade-offs. Preserve fine-grained records when practical, since summary data cannot recover details already discarded.

Analysis

This section treats storage design as a negotiation between physical performance and organizational meaning. The same dataset can be cheap to retain yet expensive to retrieve, or easy to query while difficult to update. Likewise, a model is not merely a schema: it encodes which distinctions the organization chooses to preserve. That is why the lowest useful level of detail can be valuable. It is not always the right default, however. Retaining everything can increase privacy, compliance, and cost burdens, so detail should be balanced against a defensible purpose and retention policy.

Turn raw records into reusable business logic

Transformations create utility. A transformation changes data into a form useful for reporting, analysis, or machine learning. It may standardize types, remove bad records, join sources, apply business rules, aggregate results, or prepare features for a model. These rules should be consistent and reusable. If every analyst defines profit or customer status differently, the organization can produce several plausible answers to the same question.

Use the simplest effective method. SQL is powerful for many transformations, and modern analytical databases can process large workloads. More general code may be better for specialized tasks that become awkward or hard to maintain in SQL. Query plans and performance monitoring help reveal costly scans, joins, or resource contention. ETL and ELT describe different placements of transformation work, not doctrines that every organization must follow.

Analysis

Reusable business logic is a form of institutional memory. It makes assumptions inspectable rather than leaving them embedded in individual reports or in the recollections of a few analysts. This matters because disagreement about metrics is often treated as a data bug when it is really a disagreement about definitions. The authors' pragmatic view of SQL also avoids language tribalism: declarative code can be concise, while procedural code offers flexibility. Neither guarantees clarity. Reviews, tests, ownership, and documentation still determine whether a transformation remains understandable as the business changes.

Serve data as a trusted product with a named user and action

Design for the user. Serving means making prepared data useful. A dashboard may help leaders understand performance, operational analytics can support immediate intervention, and embedded analytics can show customers activity inside an application. Machine learning needs suitable, discoverable data. Reverse ETL sends processed results back into operational tools. For example, a lead-scoring result can be returned to a CRM so sales staff can use it where they already work.

Trust is part of delivery. Users need data that is accurate, understandable, available, and timely enough for their task. Service expectations can be made explicit through service-level agreements and measurable objectives. Self-service can help users who have the skills and a real need to explore data, but it is not automatically useful for every audience. Feedback from users should shape improvements and help identify products that are no longer needed.

Analysis

The product perspective clarifies that successful serving is not synonymous with publishing a table. Adoption depends on whether the data fits into a user's decision process, while trust depends on whether its definitions and limitations are legible. Returning a score to a CRM, for instance, can remove friction, but it can also make an erroneous score operationally powerful. That raises the importance of monitoring and guardrails. Self-service deserves the same scrutiny: greater access can widen participation, yet poorly defined metrics can scale confusion as easily as insight.

Make security, quality, and cost visible in daily operations

Treat governance as engineering. The book calls security, data management, DataOps, architecture, orchestration, and software engineering cross-cutting undercurrents. They support the entire lifecycle rather than belonging to a final review. Apply least privilege, protect data in transit and at rest, and avoid collecting sensitive information that has no clear use. Metadata, lineage, ownership, and quality checks help people discover data and understand where it came from.

Detect trouble early. DataOps applies operational practices to data systems: automate deployments, monitor both pipelines and data, and respond clearly when something fails. A job can run successfully while producing stale or distorted information, so monitor quality as well as infrastructure. FinOps brings cost awareness into cloud operations. Track spending and usage, because elastic services can scale quickly in both capability and expense.

Analysis

The strongest implication is that privacy, reliability, and cost are properties of system design, not decorations added after delivery. Least privilege can enable broader data use by reducing the consequences of access, while lineage can make correction and deletion more tractable. Monitoring also needs to distinguish a software outage from a change in the underlying population or process; a statistical shift is not automatically a defect. No checklist can settle every ethical question. Teams still need accountable owners who can judge whether a technically permitted use is appropriate.

Build toward live data only where immediate action earns its complexity

Abstraction changes the job. As managed services simplify infrastructure, data engineers can spend less time maintaining clusters and more time on architecture, governance, orchestration, and business usefulness. The authors expect tools to become more interoperable and see a possible shift from the cloud-based modern data stack toward a live data stack, where streaming, analytics, applications, and machine learning work together more directly.

Real time must do real work. Continuous data can support rapid decisions, such as detecting a production defect or responding to changing user behavior. But faster delivery is not inherently valuable. Batch processing remains practical for many reports and model-training tasks, and streaming systems can be harder to operate. The book also points to spreadsheets as widely used, flexible tools that formal data platforms often fail to replace.

Analysis

The live-data vision is persuasive when latency changes the available decision, not merely the timestamp on a dashboard. It suggests a shorter feedback loop between what an application observes and what it does, which can create more responsive products. Yet the prediction should be treated as a direction, not a universal destination. Real-time systems complicate testing, ordering, recovery, and governance, while many business questions tolerate scheduled updates. The spreadsheet observation is a useful reminder that adoption depends on usability and agency, not just scale or architectural elegance.

Analysis

The book's central achievement is to make data engineering legible as an end-to-end discipline rather than a rotating catalogue of technologies. Its lifecycle frame links source behavior, storage, ingestion, transformation, and use, while its cross-cutting practices show why a pipeline cannot be evaluated solely by whether code runs. This shifts the engineer's role from moving bytes to stewarding the conditions under which information can be trusted and acted upon.

Its strongest recurring judgment is economic and organizational: complexity has a cost, and the cost is not limited to cloud invoices. A bespoke system consumes scarce engineering attention; a rigid platform narrows future options; a poorly governed dataset imposes interpretive costs on every downstream user. The authors' emphasis on managed services, modularity, and reversible choices is therefore less a preference for convenience than a strategy for preserving organizational capacity. Their maturity model usefully links technical needs to the ability of a company to consume data, rather than to age or revenue.

The book also resists the familiar temptation to treat machine learning as the natural destination of every data program. Reliable collection, definitions, quality, and access are prerequisites for useful models, and ordinary analytics can expose whether an organization can make sound decisions from data at all. The insistence that data must be used gives the lifecycle a test of purpose.

Two tensions deserve attention. First, the cloud-first perspective is productive for explaining elasticity and managed services, but cloud economics can obscure vendor dependence, egress costs, and the value of existing operational capabilities. Second, the predicted move toward streaming may be right for applications whose decisions benefit from low latency, yet it should not be mistaken for progress by itself. Batch can be simpler, cheaper, and sufficiently timely. The durable lesson is not to choose a favored architecture, but to connect technical choices to real use, explicit risk, and the capacity of the team that must operate them.

Summary in 30 Seconds
Reliable analysis depends on dependable data collection, access, and preparation. Start with the decision a user needs to make, then design the data journey around that use. Choose storage, models, and batch or streaming ingestion to fit access patterns and response times. Modular systems, reusable transformations, quality checks, security, and cost monitoring make data more trustworthy and easier to change. Real-time delivery earns its added complexity only when faster action matters.
Contains spoilers

Last updated:

Report Issue

Review Summary

4.16 out of 5
Average of 500+ ratings from Goodreads and Amazon.

Fundamentals of Data Engineering receives mostly positive reviews for providing a comprehensive overview of data engineering concepts, though some find it repetitive and lacking depth. Readers appreciate its focus on high-level principles and business context rather than specific tools. The book is praised for its structure and coverage of the data engineering lifecycle. It's recommended for those new to the field or seeking to broaden their perspective, but may be less useful for experienced practitioners looking for detailed technical information.

Your rating:
4.55
90 ratings
Want to read the full book?

Glossary

Data engineering lifecycle

End-to-end data flow

The framework for understanding how data moves from its creation in source systems through storage, ingestion, and transformation to use by people or systems. Its stages can overlap or repeat, and storage supports multiple stages. The framework helps engineers evaluate decisions in relation to downstream needs instead of choosing tools in isolation.

Undercurrents

Practices spanning every stage

The cross-cutting foundations that support work throughout the data engineering lifecycle. The book identifies security, data management, DataOps, data architecture, orchestration, and software engineering. They are called undercurrents because they affect multiple stages rather than belonging to a single point in a pipeline.

DataOps

Operations practices for data

A set of cultural and technical practices adapted from Agile, DevOps, and statistical process control to improve the speed, quality, and reliability of data products. Its technical elements include automation, monitoring and observability, and incident response. It also depends on collaboration between data teams and the people who use or produce their data.

FinOps

Collaborative cloud cost management

A financial management practice for cloud technology that brings engineering, finance, and business teams into data-informed spending decisions. In data engineering, it includes tracking usage and costs, evaluating cost against business value, and adjusting resources as workloads change. The goal is not simply lower spending, but effective spending that supports useful outcomes.

Data mesh

Domain-owned data sharing

A decentralized data architecture that assigns domain teams responsibility for making their data usable by others. Its principles include domain-oriented ownership, data treated as a product, shared self-service infrastructure, and federated governance. It aims to reduce dependence on a single centralized data platform or team while retaining organization-wide standards.

Data lakehouse

Managed tables on object storage

An architecture that combines the flexible, scalable storage of a data lake with data-management features associated with a warehouse. A lakehouse typically adds table and schema management, transactional updates and deletes, and data history to data stored in object storage. It aims to support varied tools without leaving raw files unmanaged.

Data contract

Agreement on data delivery

A written agreement between the owner of a source system and the team using its data. It can specify which data is provided, how it is extracted, whether delivery is full or incremental, how often updates arrive, and who the relevant contacts are. Clear contracts help teams communicate expectations and anticipate changes that could break downstream work.

Reverse ETL

Return processed data to sources

The practice of sending processed data from an analytical system back into operational source systems or external services. For example, a model's customer scores might be loaded into a CRM for sales staff. This connects analytical work to the tools where operational decisions happen, but feedback loops require monitoring to prevent errors from triggering harmful actions.

Data vault

History-preserving data model

A data-modeling approach that separates business identifiers, relationships, and descriptive attributes into hubs, links, and satellites. Data is typically loaded in an insert-only manner, preserving source history and accommodating change. Business interpretation is applied when users query the model, and the structure can serve as a foundation for other analytical models.

Live data stack

Real-time application data system

The authors' proposed direction for data architectures that connect applications, streaming pipelines, analytical databases, and machine learning more directly. Rather than treating analytics as a separate batch-oriented system, this approach aims to shorten the time between data generation and action. It is a forward-looking prediction, not a requirement for every organization.

About the Author

Joe Reis is a data engineering expert and author. He co-wrote Fundamentals of Data Engineering to provide a comprehensive overview of the field, drawing on his extensive experience in the industry. Reis focuses on teaching fundamental concepts and best practices rather than specific tools, aiming to create a resource that remains relevant despite rapid technological changes. His approach emphasizes the importance of understanding the broader context of data engineering within business operations. Reis's work is appreciated for its pragmatic tone and focus on real-world applications of data engineering principles.

Download PDF

To save this Fundamentals of Data Engineering summary for later, download the free PDF. You can print it out, or read offline at your convenience.
Download PDF
File size: 0.33 MB     Pages: 16

Download EPUB

To read this Fundamentals of Data Engineering summary on your e-reader device or app, download the free EPUB. The .epub digital book format is ideal for reading ebooks on phones, tablets, and e-readers.
Download EPUB
File size: 1.42 MB     Pages: 20
Want to read the full book?
Follow
Listen2 mins
Home
Swipe
Library
Get App
Try Full Access for 3 Days
Listen, bookmark, and more
Compare Features Free Pro
📖 Read Summaries
Read unlimited summaries. Free users get 3 per month
🎧 Listen to Summaries
Listen to unlimited summaries in 40 languages
—
❤️ Unlimited Bookmarks
Free users are limited to 4
—
📜 Unlimited History
Free users are limited to 4
—
📥 Unlimited Downloads
Free users are limited to 1
—
Risk-Free Timeline
Today: Get Instant Access
Listen to full summaries of 26,000+ books. That's 12,000+ hours of audio!
Day 2: Trial Reminder
We'll send you a notification that your trial is ending soon.
Day 3: Your subscription begins
You'll be charged on Nov 25,
cancel anytime before.
Consume 2.8× More Books
2.8× more books Listening Reading
Our users love us
1M+ readers
Trustpilot Rating
TrustPilot
4.6 Excellent
This site is a total game-changer. I've been flying through book summaries like never before. Highly, highly recommend.
— Dave G
Worth my money and time, and really well made. I've never seen this quality of summaries on other websites. Very helpful!
— Em
Highly recommended!! Fantastic service. Perfect for those that want a little more than a teaser but not all the intricate details of a full audio book.
— Greg M
Save 62%
Yearly
$119.88 $44.99/year/yr
$3.75/mo
Monthly
$9.99/mo
Start a 3-Day Free Trial
3 days free, then $44.99/year. Cancel anytime.
Unlock a world of fiction & nonfiction books
26,000+ books for the price of 2 books
Read any book in 10 minutes
Discover new books like Tinder
Request any book if it's not summarized
Read more books than anyone you know
#1 app for book lovers
Lifelike & immersive summaries
30-day money-back guarantee
Download summaries in EPUBs or PDFs
Cancel anytime in a few clicks

Why one summary?

Scanner
Find a barcode to scan

⭐
⭐
⭐
We have a special gift for you
Open
38% off
DISCOUNT FOR YOU
⭐
⭐
⭐
$79.99
$49.99/year
Only $4.16 per month
Continue
2 taps to start, super easy to cancel
Settings
General
Widget
Loading...