Start free trial
Trending New Genres Authors
EnglishEnglish
EspañolSpanish
简体中文Chinese
繁體中文Chinese (Traditional)
FrançaisFrench
DeutschGerman
日本語Japanese
PortuguêsPortuguese
ItalianoItalian
한국어Korean
РусскийRussian
NederlandsDutch
العربيةArabic
PolskiPolish
हिन्दीHindi
Tiếng ViệtVietnamese
SvenskaSwedish
ΕλληνικάGreek
TürkçeTurkish
ไทยThai
ČeštinaCzech
RomânăRomanian
MagyarHungarian
УкраїнськаUkrainian
IndonesiaIndonesian
DanskDanish
SuomiFinnish
БългарскиBulgarian
עבריתHebrew
NorskNorwegian
HrvatskiCroatian
CatalàCatalan
SlovenčinaSlovak
LietuviųLithuanian
SlovenščinaSlovenian
СрпскиSerbian
EestiEstonian
LatviešuLatvian
فارسیPersian
മലയാളംMalayalam
தமிழ்Tamil
اردوUrdu
Searching...
SoBrief
Why Data Science Projects Fail

Why Data Science Projects Fail

The Harsh Realities of Implementing AI and Analytics, without the Hype
Most data science projects fail, and the models are rarely the reason. Here is what actually breaks.
by Douglas Gray 2024
5.00
19 ratings
Listen
Immersive
V2.3
Amazon Kindle Audible
Try Full Access for 3 Days
Unlock listening & more!
Continue

Key Takeaways

Count success only when models change decisions and produce measurable value.

A model is not the outcome. The book defines an analytics project as successful only when its model reaches production, is used to address a business problem, and generates measurable value. A promising prototype that nobody adopts can still be a failure, even if its mathematics is sound. The authors cite failure estimates near 80%, but the figures vary, partly because studies define failure differently.

Epic’s sepsis model shows why. In one independent study, it missed 67% of sepsis cases and generated alerts for 18% of patients. Other hospitals reported better outcomes after tailoring the model to local settings. The contradiction is instructive: performance depends on patient populations, data, definitions, and clinical workflows, not just the model’s design.

Analysis

What's striking is how this definition shifts attention from model accuracy to the entire path from prediction to action. That is a useful distinction in any high-stakes system: a warning that arrives too late, overwhelms staff, or has no response protocol may be technically informative and practically useless. The failure-rate figures should be treated as warning signals, not universal probabilities. A count that combines abandoned prototypes with deployed systems that underperform can dramatize the problem while obscuring which intervention would help.

Make AI beat its next-best alternative after every cost is counted.

Calculate the incremental case. The relevant question is not whether an AI system could produce savings, but whether those extra benefits exceed its extra costs compared with the best alternative, often the current process. Include setup, operation, maintenance, repair, and the value of resources spent elsewhere. The authors describe this as comparing marginal benefit with marginal cost, using the next-best option as the baseline.

Mike’s farm failed that test. An AI precision-farming system promised up to 80% savings on roughly $100,000 in annual chemical and water costs. That best case equaled about $80,000, while ongoing data costs alone were also $80,000 per year, before a $500,000 setup bill. For a farm earning about $79,000 in profit, the technology could threaten the family’s finances rather than improve them.

Analysis

This is a practical antidote to the tendency to confuse technical possibility with economic value. Opportunity cost matters because a project competes with other uses of the same budget, staff, and attention. The same logic applies outside business: a costly intervention should be compared with what its resources could accomplish elsewhere. Still, not every benefit fits neatly into cash flow. Safety, learning, or strategic resilience may matter, but they should be named explicitly rather than used as vague exemptions from the business case.

Prioritize solutions customers will deploy, not prototypes they praise.

Test willingness to adopt. WayBlazer built AI tools that used text and voice to help travelers plan trips. Major travel brands, including hotel companies, signed on for proof-of-concept projects. Yet turning those pilots into paid production systems required long sales cycles, integration work, and investment. The startup raised $8.7 million but ran out of time and money before enough customers committed.

Rank projects by value and feasibility. The authors recommend comparing a project’s potential business impact with its ability to be implemented. A high-value idea with little chance of deployment may be a moon shot; a modest project that is easy to activate can build capability. Their point is not to avoid ambition, but to account for adoption, integration, cost, and timing before treating a prototype’s applause as evidence of a market.

Analysis

WayBlazer’s story highlights a distinction familiar in product design: users may admire a demonstration without changing habits or budgets for it. Early enthusiasm is weak evidence of sustained demand. A serious prioritization process therefore needs evidence from the people who control deployment, not just those who enjoy the prototype. But a feasibility score can also favor incremental projects by default. Organizations should reserve room for uncertain bets while setting explicit limits on the cost and time they are willing to risk.

Give executives ownership of outcomes, then protect expert execution.

RetailCo lacked alignment. A retailer planned a customer-personalization system with a strong business case, but executives did not agree on the work or resources required. IT expanded the project to include a customer data platform, while leaders replaced an experienced six-person team with two people lacking AI experience. Store teams, worried about short-term targets, reverted to standard emails when early results varied. The initiative lost both scope discipline and the conditions needed to learn.

American Airlines shows the alternative. Its operations-research team built a yield-management system credited with $1.4 billion in additional revenue over three years. Leadership backed the strategic goal and gave specialists room to execute. Senior leaders should set the target, fund the work, and hold teams accountable, rather than micromanage technical choices they do not understand.

Analysis

The contrast is not simply between strong and weak leadership. It is between leadership that creates a stable mandate and leadership that substitutes opinion for expertise. This resembles a principal-agent problem: executives must ensure specialists serve organizational goals, but excessive intervention can destroy the conditions for competent work. Delegation still requires checkpoints, transparent measures, and clear accountability. The useful balance is neither blind trust nor committee control, but informed sponsorship with decisions assigned to people equipped to make them.

Search beyond the supplied dataset when the business story stays unexplained.

Data can be incomplete in context. A logistics company built an optimization tool that users later ignored because important delivery conditions were unknown until the day of service. Inputs changed so often that recommendations shifted impractically, and some inventory data arrived at inconsistent times. Historical data had made the model look workable, but live operations exposed what the dataset had not captured.

A wider search solved another mystery. A healthcare organization’s satisfaction surveys and performance reviews could not explain nurses’ sudden resignations. The team added public economic and housing data and found that departures clustered around Riverside, Phoenix, and Las Vegas, areas hit hard by foreclosures during the financial crisis. Follow-up revealed that financial distress had driven many employees to leave town. The missing explanation was outside the original data sources.

Analysis

These cases show why data work is closer to field research than merely cleaning a table. A dataset records what an organization chose, or was able, to observe; it does not automatically include the conditions that produced the outcome. The investigator must ask what is absent, how measurements were generated, and whether the available data reflect the live decision environment. Broader searches can reveal context, but they also raise questions of privacy, representativeness, and causal interpretation. More data is not automatically better evidence.

Start small only when users can learn from an incomplete first version.

An MVP is not universally safe. A steel producer tested an early optimization tool designed to improve its production network. Planners rejected the first version because it lacked detail they considered essential, while executives wanted proof of value before funding the complete system. The team had captured most, but not all, of the problem’s constraints; the missing requirements made recommendations difficult to trust. The project stalled between an incomplete pilot and a full investment.

Choose the first version deliberately. A minimum viable model can reduce risk and invite feedback when users understand what it can and cannot do. But when an incomplete result undermines confidence or cannot demonstrate useful value, a staged approach may backfire. The team must agree in advance what the pilot will prove, what it omits, and how users will evaluate it.

Analysis

The case complicates the popular idea that every project should begin with the smallest possible product. Iteration is valuable when each version produces interpretable learning, but a crude prototype can damage trust if users mistake its limits for the method’s limits. This is a design problem as much as a modeling one: the test must be small enough to control and realistic enough to matter. Teams should make incompleteness visible, define feedback criteria, and avoid promising that an early-stage tool is ready for operational use.

Embed translators and operators where the decisions actually happen.

Bridge business and analytics. An analytics translator understands both a business domain and analytical methods, and can help operators and technical teams communicate. The role may be filled by an analytically trained person embedded in a business function or by a domain expert trained in analytics. A federated arrangement can connect these embedded staff with a central analytics group, preserving local knowledge while sharing standards.

Adoption is part of delivery. At American Airlines, an aircraft-maintenance planning system worked with analysts rather than simply replacing them. Its interactive optimization let staff inspect schedules, adjust inputs, and use the system to explore options. The analysts were promoted, and the system was associated with substantial maintenance savings. The authors’ broader lesson is that participation, communication, and change management are operational requirements, not optional polish.

Analysis

A translator is valuable because organizational knowledge is often tacit: workers know which constraints matter but may not think to state them in a requirements document. Embedding a bridge-builder near the work helps surface those assumptions before they become defects. The approach also reduces the risk of a central analytics team designing for an abstract workflow. Yet translators need authority and time, not just a new title. Without access to decision-makers and users, the role can become a messenger carrying misunderstandings between silos.

Match the model and experiment to the decision being tested.

Define the target before choosing a technique. In a classroom exercise on airline on-time performance, most teams proposed models to predict the percentage of flights arriving or departing on time. One team chose logistic regression because it interpreted the question as predicting whether an individual flight would be on time. The technique was not inherently wrong; it answered a different question. Regression and gradient-boosted trees better fit the aggregate percentage target described in the case.

Design a fair test. A/B testing can compare two webpage designs on click-through rates, but revenue depends on many other factors, including routes, fares, passenger counts, and purchase timing. A simple click-through experiment cannot establish which design generates more revenue. The metric, population, and experimental design must match the decision, or a clean-looking result may be misleading.

Analysis

This is a reminder that statistical correctness begins with measurement, not computation. A model can be impeccably fitted to a target that stakeholders did not mean, and an experiment can produce a precise estimate of the wrong effect. Clear definitions help, but they are not enough if the test fails to control important differences between groups. The practice resembles clinical trial design: specify the outcome, comparison, and conditions before looking at results. Technical sophistication cannot rescue a poorly framed question.

Treat production as a separate engineering and operating commitment.

A notebook is only a beginning. The authors cite a rule of thumb that deploying and supporting an algorithm can require about $100 for every $1 spent developing it. The true ratio varies, but production may demand data pipelines, system integrations, APIs, testing, monitoring, maintenance, and support. A pilot can therefore be up to 100 times less costly than the operational system it previews.

Complexity has drivers. The effort rises with real-time data, integration points, mission criticality, and model or problem complexity. A planning tool with weekly data and a small user group is different from a system that must operate continuously across an enterprise. Before promising delivery, identify what production requires, who will operate the system, and how failures will be detected and handled.

Analysis

The pilot-to-production gap is a form of systems debt: a demonstration can hide the work required to make a tool reliable, secure, maintainable, and usable at scale. This is why a model’s apparent cheapness can be misleading when its surrounding infrastructure is excluded from the estimate. The $1-to-$100 figure is a heuristic, not a universal law; treating it as a fixed ratio would create its own distortion. The important practice is to estimate the full operating lifecycle, not just the code that produces a prediction.

Monitor changing inputs after launch, because accurate models can drift.

Production changes the evidence. The Epic sepsis model’s performance was affected by local patient populations, definitions, clinical processes, and changed illness codes. A later version was designed to be trained on a hospital’s own data. At FastFoodCo, a forecasting model tested on historical batches failed when a separate team created real-time features that did not exactly match the original definitions. The model was ready, but its production inputs were not equivalent to the pilot data.

Monitor both sides of the model. The authors draw on Booking.com’s experience with many deployed machine-learning models: test performance offline does not guarantee online performance. Teams should track input data as well as model outputs, and use controlled comparisons, such as A/B tests, to establish whether deployment improves outcomes. Mature organizations can still fail when teams work in parallel without shared definitions.

Analysis

A deployed model is part of a changing environment, not a finished artifact. Monitoring only predictions can miss the upstream changes that make those predictions unreliable. Shared feature definitions and clear ownership are therefore governance mechanisms as much as technical safeguards. Controlled deployment also limits the harm of a bad surprise. Still, monitoring is not a substitute for judgment: a signal that data have shifted does not, by itself, tell a team whether to retrain, pause, or change the workflow. People need the authority and expertise to respond.

Analysis

The book’s strongest contribution is its insistence that data science is not a contest in model-building. It is an organizational intervention whose success depends on strategy, data, process, technical infrastructure, and human adoption working together. The recurring stories make this claim concrete: a retailer’s personalization program falters through executive interference and misaligned incentives; a promising logistics optimizer becomes burdensome when its live inputs differ from its assumptions; and a production model can fail even when every team involved is competent in isolation. These are failures at the interfaces between specialties, where responsibility is easiest to diffuse.

The People, Process, Technology framework, expanded to include Strategy, gives readers a useful diagnostic map. Its limitation is that the categories overlap: a data problem may reflect governance, incentives, skills, or leadership, and a technology defect may originate in unspoken user requirements. The framework works best as a way to organize investigation, not as a set of independent causes.

The headline failure estimate deserves similar care. The book acknowledges that estimates range widely and that the term failure can mean anything from nondeployment to weak economic returns. Its further calculations about mature and immature organizations depend on literature mentions and assumptions about relative failure rates. Those calculations are thought-provoking, but they are not equivalent to a representative project census. Readers should use the figures as a warning about recurrent waste, not as precise odds for a specific project.

The practical message is more durable than any single percentage: define the decision and value before building; expose assumptions about data and workflow; involve the people who will use the result; and budget for operational life, not just experimentation. Failure is useful only when teams can discuss it without punishment and convert it into changes in design, governance, and practice. The best antidote to hype is not cynicism. It is disciplined curiosity paired with economic and human realism.

Summary in 30 Seconds
A model is a success only if it reaches production, gets used, and creates measurable value. Judge AI against the next-best alternative, counting all costs. Test whether customers will actually deploy it, not just praise a prototype. Executives should set goals and fund the work, then trust specialists. Embed translators where decisions happen. Production can cost up to 100 times the pilot, and live inputs drift, so monitor data and outcomes.
Contains spoilers

Last updated:

Report Issue
Want to read the full book?

Glossary

ADSAI

Analytics, data science, and AI

The authors’ shorthand for analytics, data science, and artificial intelligence, which they sometimes use interchangeably because projects commonly combine them. The term groups methods and technologies used to analyze data, build models, and support or automate decisions. It does not imply that the fields are technically identical.

Analytically immature organization

Organization lacking analytics foundations

An organization that lacks several conditions needed to deliver analytics projects reliably. These may include high-quality and accessible data, in-house analytical skills, executive support, suitable processes, or experience deploying models. The term describes organizational capability, not the intelligence or effort of particular employees.

Analytics translator

Bridge between business and analytics

A person who connects analytical specialists with the business teams that understand and act on operational problems. The translator combines analytical knowledge with domain understanding, helping clarify requirements, communicate findings, and troubleshoot implementation. The role can be filled by an analytically skilled employee embedded in a business unit or by a domain expert trained in analytics.

Interactive optimization

Human-guided optimization

An approach that combines optimization algorithms with a human user’s iterative input and evaluation. The system generates candidate solutions, while the user reviews results, adjusts information or priorities, and guides further runs. It is intended to combine computational speed with human contextual judgment, rather than treating the model as an autonomous replacement for the user.

Target Zone

Realistic benefit target

The authors’ term for an expectation-setting range that avoids both sandbagging and overpromising. A team estimates a plausible benefit, aims to meet or exceed it, and may set a higher stretch goal. The purpose is to show ambition without presenting an optimistic maximum as a dependable forecast.

Data drift

Changing input data patterns

A change in the data encountered after a model is deployed, compared with the data used to train or test it. Shifts may arise from new populations, altered processes, updated codes, equipment, or other changes in the operating environment. Since changed inputs can degrade model performance, teams should monitor data as well as predictions.

Minimum viable model

Smallest useful model version

An early, limited version of a model intended to produce useful output soon enough for stakeholders to evaluate and provide feedback. It can reduce risk and support iteration, but it is not automatically appropriate. If users cannot judge its value because essential constraints or detail are missing, an incomplete version may undermine confidence rather than build it.

Download PDF

To save this Why Data Science Projects Fail summary for later, download the free PDF. You can print it out, or read offline at your convenience.
Download PDF
File size: 0.34 MB     Pages: 16

Download EPUB

To read this Why Data Science Projects Fail summary on your e-reader device or app, download the free EPUB. The .epub digital book format is ideal for reading ebooks on phones, tablets, and e-readers.
Download EPUB
File size: 1.41 MB     Pages: 19
Want to read the full book?
Follow
Listen
Home
Swipe
Library
Get App
Try Full Access for 3 Days
Listen, bookmark, and more
Compare Features Free Pro
📖 Read Summaries
Read unlimited summaries. Free users get 3 per month
🎧 Listen to Summaries
Listen to unlimited summaries in 40 languages
—
❤️ Unlimited Bookmarks
Free users are limited to 4
—
📜 Unlimited History
Free users are limited to 4
—
📥 Unlimited Downloads
Free users are limited to 1
—
Risk-Free Timeline
Today: Get Instant Access
Listen to full summaries of 26,000+ books. That's 12,000+ hours of audio!
Day 2: Trial Reminder
We'll send you a notification that your trial is ending soon.
Day 3: Your subscription begins
You'll be charged on Nov 24,
cancel anytime before.
Consume 2.8× More Books
2.8× more books Listening Reading
Our users love us
600,000+ readers
Trustpilot Rating
TrustPilot
4.6 Excellent
This site is a total game-changer. I've been flying through book summaries like never before. Highly, highly recommend.
— Dave G
Worth my money and time, and really well made. I've never seen this quality of summaries on other websites. Very helpful!
— Em
Highly recommended!! Fantastic service. Perfect for those that want a little more than a teaser but not all the intricate details of a full audio book.
— Greg M
Save 62%
Yearly
$119.88 $44.99/year/yr
$3.75/mo
Monthly
$9.99/mo
Start a 3-Day Free Trial
3 days free, then $44.99/year. Cancel anytime.
Unlock a world of fiction & nonfiction books
26,000+ books for the price of 2 books
Read any book in 10 minutes
Discover new books like Tinder
Request any book if it's not summarized
Read more books than anyone you know
#1 app for book lovers
Lifelike & immersive summaries
30-day money-back guarantee
Download summaries in EPUBs or PDFs
Cancel anytime in a few clicks

Why one summary?

Scanner
Find a barcode to scan

⭐
⭐
⭐
We have a special gift for you
Open
38% off
DISCOUNT FOR YOU
⭐
⭐
⭐
$79.99
$49.99/year
Only $4.16 per month
Continue
2 taps to start, super easy to cancel
Settings
General
Widget
Loading...