Start free trial
Trending New Genres Authors
EnglishEnglish
EspañolSpanish
简体中文Chinese
繁體中文Chinese (Traditional)
FrançaisFrench
DeutschGerman
日本語Japanese
PortuguêsPortuguese
ItalianoItalian
한국어Korean
РусскийRussian
NederlandsDutch
العربيةArabic
PolskiPolish
हिन्दीHindi
Tiếng ViệtVietnamese
SvenskaSwedish
ΕλληνικάGreek
TürkçeTurkish
ไทยThai
ČeštinaCzech
RomânăRomanian
MagyarHungarian
УкраїнськаUkrainian
IndonesiaIndonesian
DanskDanish
SuomiFinnish
БългарскиBulgarian
עבריתHebrew
NorskNorwegian
HrvatskiCroatian
CatalàCatalan
SlovenčinaSlovak
LietuviųLithuanian
SlovenščinaSlovenian
СрпскиSerbian
EestiEstonian
LatviešuLatvian
فارسیPersian
മലയാളംMalayalam
தமிழ்Tamil
اردوUrdu
Searching...
SoBrief
Standard Deviations

Standard Deviations

Flawed Assumptions, Tortured Data, and Other Ways to Lie with Statistics
A statistic can be accurate and still mislead; the danger lies in how the evidence was made.
by Gary Smith 2014 304 pages
3.95
500+ ratings
Listen
Immersive
V2.3
Amazon Kindle Audible
Try Full Access for 3 Days
Unlock listening & more!
Continue

Key Takeaways

Treat an eye-catching pattern as a question, not a conclusion

Paul was lucky, not psychic. The celebrated octopus chose 12 winners in 14 football predictions, a record that would be unlikely if each guess were a fair coin toss. But most predictions involved Germany, and Paul may have preferred the horizontal stripes of its flag. The apparent pattern could reflect the flag, not an uncanny grasp of soccer.

Success gets the headlines. Many animals were recruited to predict games, but the failures attracted little attention. If enough pets make enough guesses, a few will look astonishingly accurate. The same trap appears when researchers search large datasets for striking relationships. A pattern is a clue to investigate, not evidence that its most appealing explanation is true.

Analysis

Pattern recognition is useful when a signal genuinely predicts danger or opportunity. The trouble is that the mind often notices the memorable hit and forgets the many misses that created the opportunity for one. This resembles a search problem: the more patterns examined, the more impressive coincidences appear. A sound interpretation asks not only whether a pattern exists, but how many alternative patterns were available to find, and whether the proposed explanation predicts new observations.

Count the people and outcomes missing from every comparison

Choices shape the sample. Students who choose a college with a lower graduation rate may differ from those who choose a more selective school. Comparing the schools' overall graduation rates does not reveal what would have happened had the same students attended each one. This is self-selection bias: people’s choices influence which group they enter, so group differences may reflect who chose what, not the effect of the choice.

Survivors are not everyone. The British Royal Air Force initially planned to armor the most bullet-damaged parts of returning planes. Statistician Abraham Wald pointed out that planes hit in vital areas often never returned. The missing aircraft mattered most. The same logic warns against studying only companies that survived, customers who stayed, or patients who lived long enough to enter a study.

Analysis

This lesson reaches beyond statistics into policy and management: a comparison is only meaningful when its population and selection process are understood. Random assignment can isolate effects, but it is often impractical or unethical. Carefully chosen natural experiments, like comparing neighboring London homes supplied by different water companies, can sometimes help. Even then, the result applies most securely to the people and conditions studied, not automatically to everyone.

A significant result means little after a thousand hidden tests

Chance produces false alarms. A result is often called statistically significant when it would occur by chance less than 5% of the time under a specified null hypothesis. Test 100 unrelated ideas, however, and about five may cross that threshold by luck alone. If researchers report only the successes and hide the failures, readers cannot see how many chances were taken.

The published record can mislead. A study cited in the book found that 97% of reported tests in psychology journals were statistically significant, a suspiciously high share. Researchers can also split data into many subgroups or adjust cutoffs until something appears. This practice, called data grubbing, makes the selected result look stronger than it is. Independent replication with fresh data is essential.

Analysis

A p-value answers a narrow question about data under a model; it does not measure the probability that a claim is true. That distinction is easy to lose when journal publication rewards striking results. Preregistering hypotheses and analysis plans makes it harder to quietly change the question after seeing the answers. Replication is not glamorous, but it functions like a second measurement in engineering: it reveals whether a finding survives beyond the circumstances that first made it look exciting.

Check who chose each group before calling one treatment better

Composition can reverse a result. In John Snow’s investigation of cholera, neighboring London homes were supplied by different water companies. One company drew cleaner water from upriver; its customers had far fewer cholera deaths. Because households were close together, the comparison reduced differences in neighborhood conditions and helped challenge the prevailing theory that foul air caused the disease.

Averages can conceal confounders. At Berkeley, women had a lower overall graduate-school admission rate, but women applied more often to departments with lower acceptance rates. Within departments, the apparent overall disadvantage largely disappeared. A confounding factor is a third influence related to both the grouping and the outcome. When aggregated results reverse after groups are separated, the pattern is known as Simpson’s Paradox.

Analysis

The crucial question is not simply whether two outcomes differ, but whether the groups were comparable before the intervention. Snow’s evidence was powerful because it combined a useful natural experiment with a plausible route of infection. The Berkeley example shows why crude averages can mislead, but breaking data into subgroups is not automatically a cure. Subgroups should reflect meaningful causal differences, not arbitrary categories discovered after the fact.

Inspect axes, dates, and units before believing a graph

The picture can overstate the change. An Internet company’s revenue fell only 2%, yet a board chart with a truncated vertical axis made the decline look roughly 40% tall. A graph that omits zero can help reveal small movements, but it can also exaggerate their size. Missing labels and mismatched time intervals make the scale harder to judge.

Omitted observations can be fatal. Before the Challenger launch, a chart showed seven flights with O-ring failures and no clear temperature pattern. It left out 17 flights with no failures. Including all 24 flights showed failures on each of the four flights below 65 degrees. The omitted evidence mattered, and the shuttle broke apart after launch in freezing conditions.

Analysis

Graphs are arguments in visual form. Axis choices, category boundaries, and time windows determine what looks large, small, or unusual. A zero baseline is especially useful for bar charts that encode magnitude, though a clearly labeled truncated axis can be legitimate when the goal is to show fine variation. The general test is transparency: can a reader recover the scale and judge the magnitude without relying on the designer’s visual emphasis?

Use base rates to interpret positive tests and rare events

Accuracy is not the same as certainty. Imagine 1,000 patients, with 1% having a malignant lump. A test detects 80% of malignant cases and correctly identifies 90% of benign ones. It produces about 8 true positives and 99 false positives. Thus, only 8 of 107 positive results indicate cancer, or about 7.5%. The disease is rare, so false positives from the much larger benign group outnumber true positives.

More tests mean more chances to mislead. In an MRI demonstration, a dead salmon appeared to show brain activity when researchers tested a very large number of brain regions. Some apparent signals arose by chance. Before trusting a positive result, ask how common the condition is, how often the test misses it, and how often it flags healthy cases.

Analysis

This is Bayesian reasoning in practical clothing: evidence changes a prior probability, but does not erase it. Natural frequencies, such as eight true cases among 107 positive tests, are often easier to understand than abstract percentages. The appropriate threshold also depends on consequences. A false alarm may justify follow-up testing when treatment is low-risk, but can cause serious harm when it triggers invasive treatment or stigma. A test result is information, not a verdict.

Expect exceptional results to soften, not abilities to vanish

Extreme performances include luck. A person’s observed result is an imperfect measure of underlying ability. Horace Secrist compared the most and least profitable companies in one year, then found that their performance moved closer to average later. Statistician Harold Hotelling showed why this did not prove that competition was turning every company mediocre. The extreme first-year results partly reflected temporary luck, which was unlikely to repeat at the same level.

This is regression toward the mean. It describes the tendency for unusually high or low measurements to be followed by less extreme ones when outcomes contain randomness. A flight instructor may praise an unusually good performance and see the next one worsen, or scold a poor performance and see improvement. Neither result proves that praise harms or scolding helps.

Analysis

Regression is a property of noisy measurements, not a mysterious force pulling people toward mediocrity. It matters whenever a decision follows an unusually good or bad result: school intervention, medical treatment, employee evaluation, or investment advice. The principle does not predict what will happen to a particular person, and ability can change for real reasons. A fair assessment compares repeated measurements and includes an appropriate control group rather than crediting every rebound to the latest intervention.

Do not expect chance to compensate for past bad luck

Independent events have no memory. A fair coin that lands heads several times is not due to land tails. In 1913, roulette players at Monte Carlo kept betting on red after black appeared 26 times in a row, trusting that outcomes had to balance soon. The wheel had no obligation to rescue them. This mistaken belief is often called the law of averages.

People are not coins. Athletic performance can fluctuate, and the book finds evidence of modest hot and cold effects in horseshoes and bowling. But a streak alone does not prove that a player’s odds have changed. In basketball, shot difficulty, defensive pressure, and timing complicate comparisons. Separate genuine changes in performance from streaks that random variation can create.

Analysis

The useful distinction is between independent trials and systems with memory. A coin toss does not change after a previous result; a person can become fatigued, confident, injured, or more closely guarded. Thus, neither “the streak must continue” nor “the streak must end” is a sound default. The next question is whether a plausible mechanism links past performance to future performance, and whether the data measure attempts under comparable conditions.

Pair a plausible mechanism with data it did not generate

Data alone do not explain themselves. In Rwanda, cell-phone movement data appeared to predict cholera outbreaks: people traveled less shortly before cases rose. The apparent signal was not a mysterious behavioral warning. Flash floods washed out roads, limiting travel while also increasing cholera risk. Without that mechanism, officials might have mistaken a local proxy for a general predictor.

Models need contact with reality. The Limits to Growth project used computer simulations to warn that population and resource use would soon overwhelm finite supplies. The book argues that its forecasts relied on untested assumptions and poorly grounded inputs, including a mistaken comparison of living standards. A theory without reliable data is conjecture; data without a credible theory can be a coincidence dressed as prediction.

Analysis

Models are valuable precisely because they simplify complicated systems, but every simplification creates a question: which relationships are supported, and which are assumed? Calibration against historical evidence is not enough if the model cannot explain how its variables interact or anticipate new cases. Conversely, a plausible mechanism is not proof that an effect is large. Strong inference asks for both an account of why something should happen and a test that could show the account is wrong.

Never let leverage turn a historical correlation into a law

A past range is not a promise. Investors were urged to trade the gold-silver ratio because it had often returned to roughly 34 to 38. But the ratio rose far above that range and did not fall below 38 again until decades later. A leveraged futures position could lose more than its initial margin long before the hoped-for convergence arrived.

Long-Term Capital Management made the risk larger. The firm used heavy leverage to bet that historically related prices would move back together. After Russia defaulted in 1998, risk premiums rose across markets that had seemed unrelated. Losses mounted too quickly for the fund to wait for its trades to recover. A statistical relationship is not a guarantee, especially when many investors rely on the same model.

Analysis

The danger is not mathematics itself, but mathematical confidence detached from market structure and survival constraints. A trade can be right over a long horizon and still fail if financing runs out first. Historical correlations often weaken during stress, precisely when portfolios need them most. Responsible risk analysis therefore asks what happens when relationships break, liquidity disappears, or many traders crowd into the same position, rather than relying only on average historical behavior.

Analysis

The book’s central achievement is to turn statistical skepticism into a practical habit of questioning how evidence was produced. Its examples range from Paul the octopus to cholera, the Challenger disaster, and Long-Term Capital Management, showing that data failures are not confined to amateur superstition. They arise from selection, missing observations, multiple testing, coding mistakes, distorted displays, and theories that outrun their evidence. The recurring question is not whether numbers are impressive, but what process could have produced them.

The strongest cases pair a memorable story with a diagnostic principle. Wald’s aircraft insight makes survivor bias visible; Snow’s cholera investigation shows how a natural experiment can weaken confounding; and the dead-salmon MRI makes false positives hard to dismiss as abstract mathematics. These stories give readers a usable skepticism rather than a demand to distrust every statistic.

There is a risk in the book’s prosecutorial style: exposing a flawed study can make skepticism feel like a universal solvent. It is not. Observational data can support credible causal conclusions when researchers use strong designs, transparent assumptions, and independent checks. Nor does an error automatically invalidate every related finding. The right response is calibrated confidence, not reflexive dismissal.

The book’s enduring method is a sequence of questions: Who entered the sample, who disappeared, what else could explain the result, how many analyses were tried, and does a plausible mechanism connect the variables? Then ask whether the explanation survives new evidence. This approach is useful in medicine, business, public policy, and personal decisions because it replaces both blind faith in computation and vague suspicion with accountable reasoning. Numbers become persuasive not through sheer volume or prestigious authorship, but when measurement, theory, comparison, and replication reinforce one another.

Summary in 30 Seconds
A striking pattern is a prompt to investigate, not proof of its proposed cause. Missing cases, self-selection, and confounding can distort comparisons, while testing 100 unrelated ideas can produce about five false alarms at a 5% threshold. Rare conditions make positive tests less reliable: with a 1% prevalence, only about 8 of 107 positive results may be true. Extreme results often move closer to average, and historical correlations offer no guarantee of future performance. Check the sample, alternatives, mechanisms, and independent replications.
Contains spoilers

Last updated:

Report Issue

Review Summary

3.95 out of 5
Average of 500+ ratings from Goodreads and Amazon.

Standard Deviations is praised for its clear explanations of statistical fallacies and misuses of data. Readers appreciate the engaging examples and humorous tone. The book covers topics like survivorship bias, regression to the mean, and data mining without theory. Many found it eye-opening and useful for critical thinking. Some criticisms include lack of originality, repetitiveness in later chapters, and that it's not ideal as an audiobook. Overall, reviewers recommend it as an accessible introduction to statistical pitfalls for a general audience.

Your rating:
4.47
112 ratings
Want to read the full book?

Glossary

self-selection bias

Choices shape group comparisons

A distortion that occurs when people choose which group, treatment, school, or activity they enter. The resulting groups may differ before the outcome is measured. Their later differences therefore cannot automatically be attributed to the choice itself. For example, students selecting different colleges may differ in preparation, motivation, or expectations.

survivor bias

Missing failures skew conclusions

A bias caused by analyzing only people, organizations, or objects that remain observable after a period of risk. Those that failed, died, left, or were destroyed are absent from the sample. Studying only returning aircraft, successful companies, or current customers can therefore produce a misleading picture of what predicts survival or success.

data grubbing

Searching data for significance

The practice of examining data in many ways, including trying subgroups, cutoffs, or outcome measures, until a striking pattern or statistically significant result appears. If the search process is not disclosed and only the successful analysis is reported, ordinary chance findings can look like planned tests of a meaningful theory.

confounding factor

A third influence distorts comparisons

A factor associated with both the comparison being studied and its outcome, making it difficult to identify what caused the observed difference. For example, a population’s age distribution can affect overall death rates, while the population’s location may be the comparison of interest. Researchers must consider whether such factors explain or alter a result.

Simpson’s Paradox

Aggregated patterns reverse in subgroups

A reversal that occurs when a relationship seen in combined data changes direction after the data are separated into meaningful groups. The reversal can arise because the groups differ in size or composition and a confounding factor affects the outcome. It warns that overall averages may conceal important structure, though subgroup patterns also require sensible interpretation.

regression toward the mean

Extreme measurements become less extreme

The tendency for an unusually high or low measurement to be followed, on average, by a result closer to the typical level when outcomes include random variation. It does not mean that ability or quality is declining. A standout performance may combine genuine skill with unusually favorable circumstances that are unlikely to recur at the same level.

Texas sharpshooter

A target is drawn after results

A fallacy in which someone searches outcomes for a cluster or pattern, then treats the discovered pattern as if it had been predicted in advance. Like a shooter painting a bullseye around a cluster of bullet holes, the analyst selects the apparent target after seeing where the data landed. Independent tests are needed to assess the claim.

false-positive problem

Positive tests include healthy cases

The risk of treating a positive test as proof of a condition without considering how common that condition is and how often the test flags people who do not have it. When a condition is rare, false positives among the much larger healthy population can outnumber true positives, even when the test detects many genuine cases.

Greater Fool Theory

Buy high, hoping to resell higher

A speculative view that an asset can be purchased at an inflated price because another buyer will pay an even higher price later. The purchaser relies on resale expectations rather than the asset’s underlying capacity to generate value. The strategy can appear successful while prices rise, but becomes fragile when buyers stop expecting further increases.

convergence trade

Bet that prices will reunite

An investment position that profits if two prices or financial measures move back toward a historical relationship. Traders may buy the relatively cheap asset and sell the relatively expensive one. The approach is risky when the historical relationship has no strong economic reason to persist, or when losses and financing demands arrive before convergence.

About the Author

Gary Smith is an author and economist known for writing about statistics and data analysis for a general audience. He has authored multiple books on the topic, including "Standard Deviations" and "What the Luck?". Smith's work focuses on exposing common misuses and misinterpretations of statistics in various fields. He uses real-world examples and a conversational style to explain complex concepts. Smith's background is in economics, and he has taught at Pomona College. His books aim to help readers become more critical consumers of statistical information and understand the limitations of data-driven conclusions.

Download PDF

To save this Standard Deviations summary for later, download the free PDF. You can print it out, or read offline at your convenience.
Download PDF
File size: 0.35 MB     Pages: 18

Download EPUB

To read this Standard Deviations summary on your e-reader device or app, download the free EPUB. The .epub digital book format is ideal for reading ebooks on phones, tablets, and e-readers.
Download EPUB
File size: 1.40 MB     Pages: 19
Want to read the full book?
Follow
Listen
Home
Swipe
Library
Get App
Try Full Access for 3 Days
Listen, bookmark, and more
Compare Features Free Pro
📖 Read Summaries
Read unlimited summaries. Free users get 3 per month
🎧 Listen to Summaries
Listen to unlimited summaries in 40 languages
—
❤️ Unlimited Bookmarks
Free users are limited to 4
—
📜 Unlimited History
Free users are limited to 4
—
📥 Unlimited Downloads
Free users are limited to 1
—
Risk-Free Timeline
Today: Get Instant Access
Listen to full summaries of 26,000+ books. That's 12,000+ hours of audio!
Day 2: Trial Reminder
We'll send you a notification that your trial is ending soon.
Day 3: Your subscription begins
You'll be charged on Nov 24,
cancel anytime before.
Consume 2.8× More Books
2.8× more books Listening Reading
Our users love us
1M+ readers
Trustpilot Rating
TrustPilot
4.6 Excellent
This site is a total game-changer. I've been flying through book summaries like never before. Highly, highly recommend.
— Dave G
Worth my money and time, and really well made. I've never seen this quality of summaries on other websites. Very helpful!
— Em
Highly recommended!! Fantastic service. Perfect for those that want a little more than a teaser but not all the intricate details of a full audio book.
— Greg M
Save 62%
Yearly
$119.88 $44.99/year/yr
$3.75/mo
Monthly
$9.99/mo
Start a 3-Day Free Trial
3 days free, then $44.99/year. Cancel anytime.
Unlock a world of fiction & nonfiction books
26,000+ books for the price of 2 books
Read any book in 10 minutes
Discover new books like Tinder
Request any book if it's not summarized
Read more books than anyone you know
#1 app for book lovers
Lifelike & immersive summaries
30-day money-back guarantee
Download summaries in EPUBs or PDFs
Cancel anytime in a few clicks

Why one summary?

Scanner
Find a barcode to scan

⭐
⭐
⭐
We have a special gift for you
Open
38% off
DISCOUNT FOR YOU
⭐
⭐
⭐
$79.99
$49.99/year
Only $4.16 per month
Continue
2 taps to start, super easy to cancel
Settings
General
Widget
Loading...