Key Takeaways
Inspect what disappeared before trusting the evidence that remains
The missing cases matter. During World War II, officers studied bullet holes on returning planes and proposed reinforcing the most damaged areas. Statistician Abraham Wald noticed that the engines had fewer holes. Planes struck there were less likely to return, so the surviving aircraft concealed the most dangerous hits. The armor belonged on the engines, where the evidence was absent because the vulnerable planes were gone.
Selection distorts the picture. Mutual fund rankings can make surviving funds look better than the original pool because poorly performing funds often close. A study cited in the book found that including failed funds reduced reported annual returns from 10.8% to 8.9%. Whenever inclusion depends on success, survival, or visibility, ask who never made it into the dataset.
What's striking is that Wald's insight is not a clever correction to a statistic. It is a correction to the process that produced the statistic. The same question applies to online reviews, published research, and company case studies: which failures were filtered out before the evidence was presented? This is a useful discipline for readers and decision-makers alike. Yet missing data do not always tell a single story. A plane can vanish for reasons unrelated to engine damage, and failed funds may differ in more than performance. The lesson is to investigate selection, not to assume its exact effect.
Check whether the curve bends before extending a trend
Local patterns mislead at a distance. A smooth curve looks nearly straight when viewed close up, which is why a line can describe a short stretch of motion while failing badly over a longer one. The book's example is a missile: a straight-line projection fits a few early observations, then predicts endless ascent, although gravity eventually brings the missile down.
Extremes invite bad forecasts. A study extrapolated rising U.S. weight trends and predicted every American would be overweight by 2048. But a proportion cannot keep rising linearly beyond 100%, and the increase had already begun to slow. The Laffer curve makes the broader point: too little or too much taxation can both reduce revenue. Direction depends on starting position, and percentages of totals that can turn negative can produce absurd claims.
What's useful here is the distinction between interpolation and extrapolation. A line can be a good local approximation without being a trustworthy map of what happens next. This matters in business forecasts, climate projections, and personal metrics, where recent momentum is often mistaken for a permanent law. A counterpoint is that curved models can be just as misleading when their shape is chosen to fit expectations. The responsible move is not simply to reject straight lines, but to state the range where a model has evidence behind it and what would make its forecast fail.
The more patterns you test, the less surprising a hit becomes
Chance produces striking results. The Baltimore stockbroker appears brilliant after correctly predicting ten market moves. But the broker secretly sent different predictions to 10,240 people, eliminating recipients who got bad calls. Ten people were bound to see a perfect streak. The same selection effect appears when researchers test many possibilities but publish only the promising result.
A small p-value is limited evidence. A p-value estimates how often results at least this extreme would arise if a specified null hypothesis were true. It does not give the probability that the hypothesis itself is true. Testing thousands of brain regions, as in fMRI, creates many chances for noise to look meaningful. A famous joke scanned a dead salmon and found apparent responses to human emotion, illustrating why researchers must correct for multiple comparisons. Replication tests whether the signal persists.
The key issue is the size of the search, not merely the improbability of the winning result. A result that seems extraordinary after one planned test may be routine after thousands of hidden attempts. This insight reaches beyond laboratories to product experiments, investment records, and viral anecdotes. Modern research practices such as preregistration make the search plan visible before results arrive, reducing the temptation to narrate a lucky outcome as a discovery. Still, stricter thresholds alone are no cure: they can hide small real effects when studies have too few participants. Evidence quality depends on design, search scope, and follow-up.
Update beliefs with base rates, not surprise alone
Evidence needs a starting point. Bayesian inference describes how beliefs should change when new evidence arrives. It combines a prior, the initial plausibility of a hypothesis, with the likelihood of seeing the evidence if that hypothesis were true. The Bible-code researchers reported unusual matches between rabbi names and dates in the Torah, but choices about how to spell and identify the rabbis created wiggle room. Other choices made War and Peace appear similarly predictive.
Rare events stay rare. In the book's hypothetical Facebook screen, an algorithm flags 100,000 of 200 million users as twice as likely as average to be terrorists. Even if its ranking works as claimed, only about ten flagged people would be terrorists, leaving nearly all flagged users innocent. The chance of a flag given innocence is not the chance of guilt given a flag.
Base rates explain why a test can be accurate in one sense and still produce mostly false alarms when the target condition is rare. The distinction is crucial in medical screening, fraud detection, and security systems, where a modest false-positive rate can overwhelm a small pool of actual cases. Bayesian updating also makes assumptions visible: prior beliefs affect the conclusion, and those priors can be disputed. That is not a defect unique to Bayesian reasoning. It is a reminder that conclusions depend both on observed data and on background knowledge, which should be stated rather than smuggled in.
Choose by expected utility, not by the size of the prize
Average payoff is not a promise. Expected value is calculated by multiplying each possible outcome by its probability, then adding the results. A Powerball ticket can have a large advertised jackpot and still be worth less than its price on average, especially when many winners may share the prize. The Massachusetts Cash WinFall game briefly offered a different calculation: when accumulated jackpot money rolled into smaller prizes, high-volume players could buy enough tickets to make the average return favorable.
Risk changes the decision. Even a positive expected dollar return may be unacceptable to someone who cannot absorb a large loss. Utility measures how much an outcome matters to a decision-maker, rather than just its cash value. The Cash WinFall syndicates also used carefully arranged ticket combinations to reduce variation in payouts. Their profits depended on the rules and volume, not on any single ticket being a sure winner.
Expected value is powerful when outcomes and probabilities are reasonably known, as in a lottery with published rules. It is less decisive when probabilities are uncertain, losses threaten basic security, or people value outcomes differently. That is why a wealthy investor and a household with little savings can rationally reject the same gamble for different reasons. The framework also reveals a social dimension: a favorable average for a large syndicate can coexist with a poor deal for ordinary players. Arithmetic evaluates a choice, but it cannot decide whose welfare should count or what level of risk is fair.
Blame luck before declaring a cure, curse, or genius
Extreme performers tend to move back toward ordinary. Regression to the mean is the tendency for unusually high or low results to be followed, on average, by less extreme ones when performance includes chance. Francis Galton saw this in family heights: tall parents tended to have tall children, but those children were usually closer to average height. The same logic undermines Horace Secrist's claim that businesses naturally decay into mediocrity. Firms selected for outstanding results were likely both skilled and lucky; luck need not repeat.
Timing creates false cures. People often try a diet when their weight is unusually high, so some decline may happen without the diet. Baseball players selected for the Home Run Derby have often had exceptional first halves; their later totals tend to fall, which can look like a curse. A comparison group helps separate treatment effects from ordinary fluctuation.
Regression to the mean is a quiet antidote to stories that reward or punish people for a single extreme result. It is especially important in education, medicine, sports, and management, where interventions are often applied to those doing worst or celebrated after a peak. The strongest test compares similar groups over the same period, ideally with random assignment. But regression does not prove an intervention ineffective. A real treatment can coexist with natural movement toward average. The mistake is treating improvement after selection as proof that the intervention caused it.
Treat correlation as a clue, not a causal verdict
Association has several possible explanations. Galton's correlation measures how variables move together, not whether one causes the other. People with high HDL cholesterol tend to have fewer heart attacks, but a drug that raises HDL does not necessarily prevent them. HDL may be a marker of another process rather than the mechanism that protects the heart. This is the surrogate endpoint problem: an easy-to-measure indicator is used in place of the outcome that actually matters.
Causal claims need converging evidence. Early studies found smoking associated with lung cancer, but researchers considered alternative explanations, including shared causes and the possibility that illness influenced smoking. Over time, evidence from heavier smoking, disease patterns, former smokers, and biological research strengthened the causal case. The book also describes Berkson's fallacy: examining only hospital patients can create a misleading association because multiple illnesses independently increase the chance of hospitalization.
The distinction between prediction and intervention is central to modern science. A variable can predict an outcome well while being useless to manipulate, as a barometer predicts rain without causing it. Randomized experiments help answer intervention questions, while causal diagrams and natural experiments can clarify pathways when randomization is impossible. The smoking example also shows that skepticism has two sides: premature certainty can mislead, but endless demands for perfect proof can delay action. Good causal reasoning weighs alternative explanations against the accumulated pattern of evidence, not against a fantasy of certainty.
A crowd can hold rational views that produce no coherent majority
Public opinion is not always a single answer. Voters can favor spending cuts in general while opposing cuts to every named program because they disagree about which programs deserve protection. In a three-candidate election, a majority may prefer A to B, another majority prefer B to C, and a third prefer C to A. This cycle, called the Condorcet paradox, means there may be no candidate who defeats every rival head to head.
Voting rules shape the result. In Florida in 2000, Ralph Nader could not win, but his presence changed the contest between Al Gore and George W. Bush. Alternative systems, including ranked ballots and instant runoff, use more preference information but can still produce counterintuitive outcomes. There is no procedure that perfectly converts every set of rankings into one unambiguous collective will.
This is a reminder that aggregation is not neutral bookkeeping. A voting rule defines which preferences count and how they combine, much as a measurement instrument defines what it can observe. That makes institutional design a substantive political choice, not a technical afterthought. Ranked ballots can reduce some forms of vote splitting, but they do not erase every paradox; no single method is best under all criteria. The useful question is not whether a system perfectly reveals what the public wants, but which tradeoffs it makes visible, which it hides, and whether those tradeoffs are defensible.
Use mathematics to discipline common sense, not manufacture certainty
Math extends ordinary reasoning. The book's central claim is that mathematical thinking is not merely calculation. It supplies definitions, models, and checks that help intuition reach beyond what it can manage alone. Perspective geometry explains why parallel rails seem to meet at a vanishing point. Error-correcting codes use carefully spaced patterns so a receiver can reconstruct a damaged message. Both examples show abstract structures becoming practical tools.
Models have boundaries. A model built from uncertain tuberculosis studies could not reliably predict infections decades ahead, even though it produced a precise-looking number. Nate Silver's election forecasts offered probabilities rather than pretending to know the winner with certainty. Mathematical discipline means stating assumptions, testing the model's limits, and matching precision to evidence. The goal is not to avoid every error. It is to know what supports a conclusion and how unsure it remains.
The book's strongest contribution is its picture of mathematics as a habit of disciplined attention: define the question, inspect assumptions, test alternative explanations, and keep uncertainty in view. Its constructed examples, from perspective to coding, also show why abstraction is not an escape from reality. It can reveal structure that ordinary perception misses. But formal reasoning cannot settle every dispute. Choices about values, acceptable risk, and fairness require judgment beyond calculation. A mathematical answer is trustworthy only when the question, assumptions, and limits are trustworthy too.
Analysis
This book is a guided tour of mistakes that arise when people treat numbers as self-interpreting. Its narrative method matters: a missing bullet hole, a suspiciously successful stock tip, an apparently psychic salmon, or an election with no stable winner gives a concrete puzzle before the mathematical idea is named. The reader learns not a single master formula, but a set of habits: check how data were selected, distinguish local fit from long-range prediction, ask what a probability is conditional on, and separate association from intervention.
The central tension is productive. Mathematics can make ordinary reasoning more explicit and powerful, yet every model depends on definitions and assumptions that mathematics alone cannot certify. A p-value cannot determine whether a detected effect matters. An expected return cannot decide whose losses count. A voting rule cannot reveal a uniquely coherent public will when preferences cycle. The book therefore treats mathematical literacy as both technical competence and intellectual modesty.
Its stories also expose a recurring institutional problem: incentives shape what evidence becomes visible. Funds disappear, failed studies remain unpublished, and political actors select whichever denominator supports their claims. These are not merely individual cognitive errors. They are systems that reward attractive conclusions and hide inconvenient cases. Better reasoning consequently requires better procedures, including transparent search plans, replication, suitable comparison groups, and honest uncertainty reporting.
The book's broadest lesson is not that every question can be mathematized. It is that quantitative claims deserve scrutiny at the level of problem formulation, not just arithmetic. A calculation can be flawless and still answer the wrong question. Mathematical thinking earns its authority by making structure visible, then acknowledging where evidence ends and judgment begins.
Review Summary
How Not to Be Wrong receives mostly positive reviews for making mathematics accessible and relevant to everyday life. Readers appreciate Ellenberg's engaging writing style, humor, and real-world examples. Many find the book enlightening, particularly regarding statistics and probability. Some criticize its length and occasional complexity. Political commentary in later chapters divides readers. Overall, reviewers recommend it for those interested in applied mathematics and critical thinking, though some suggest it may be challenging for readers without a mathematical background.
People Also Read
Glossary
survivorship bias
Missing failures distort resultsA distortion that occurs when analysis includes only people, organizations, or objects that survived a process. Because survival may depend on performance or vulnerability, the remaining sample is not representative of the full group. Studying only returning aircraft, for example, hides damage patterns associated with planes that did not return.
false linearity
Assuming trends stay straightThe mistake of assuming a relationship continues in one direction at a constant rate. A linear model may describe a limited range of observations but fail when extended beyond that range. It can produce impossible or implausible projections, such as a population share exceeding 100% or a missile that never descends.
Law of Large Numbers
Averages stabilize with repetitionA probability principle stating that repeated observations of a random process tend to bring the observed average closer to its expected value. It does not mean the next outcome compensates for earlier outcomes. A fair coin has the same chance of heads after a long run of heads; additional flips dilute the earlier imbalance.
null hypothesis significance test
Tests data against no effectA statistical procedure that begins by assuming a specified null hypothesis, often that a treatment has no effect. It calculates how likely the observed data, or more extreme data, would be under that assumption. A small p-value challenges the null, but does not directly give the probability that the null is true.
Bayesian inference
Updates beliefs with evidenceA method for revising the relative plausibility of hypotheses as evidence arrives. It combines prior probabilities, which represent starting plausibility, with the likelihood of observing the evidence under each hypothesis. The resulting posterior probabilities depend on both the evidence and the prior assumptions.
expected value
Probability-weighted average payoffA calculation that multiplies each possible outcome by its probability and sums the results. It describes the average payoff across repeated instances, not necessarily the outcome of any single event. A lottery ticket can have a positive expected value while still being very likely to win nothing.
regression to the mean
Extreme results become less extremeThe tendency for an unusually high or low observation to be followed by a result closer to the average when measurements include chance variation. It does not imply that a hidden force punishes excellence or rewards failure. It explains why selecting extreme performers and measuring them again can create apparent improvement or decline without a causal intervention.
Berkson's fallacy
Selection creates false associationA misleading association that can arise when analysis is restricted to a group selected by a shared consequence of two conditions. Hospital patients, for example, may appear to show a relationship between illnesses because either illness can bring someone into the hospital. The relationship in the selected group may not exist in the wider population.
Condorcet paradox
Majority preferences cycleA problem in collective choice where majority preferences can form a cycle. A majority may prefer candidate A over B, another majority prefer B over C, and a third prefer C over A. In such a case, no candidate is the consistent majority favorite across all pairwise comparisons.
Hamming code
Corrects damaged digital messagesAn error-correcting code that adds structured bits to a message so a receiver can identify and correct certain transmission errors. In the example described, each three-bit message is encoded as a seven-bit code word. The geometric spacing between permitted code words makes a one-bit error distinguishable from other valid messages.
formalism
Reasoning from defined rulesA view of mathematics in which symbols and objects are governed by explicit axioms and rules of deduction. On this account, a mathematical claim is established by showing that it follows from the chosen assumptions. The approach clarifies what a system permits, while leaving open whether its definitions capture the intended real-world meaning.
FAQ
What's How Not to Be Wrong about?
- Mathematics in Everyday Life: The book explores how mathematical thinking can enhance decision-making in various areas, from politics to medicine. It emphasizes that math is not just about numbers but about understanding the world.
- Real-World Applications: Ellenberg uses examples like Abraham Wald's bullet hole analysis to show how math can lead to better insights and decisions.
- Accessible to All: The author aims to make complex mathematical concepts understandable for a general audience, demonstrating that everyone can engage with math meaningfully.
Why should I read How Not to Be Wrong?
- Enhance Critical Thinking: The book improves your ability to think critically and analytically about everyday problems, helping you avoid common reasoning pitfalls.
- Engaging and Informative: Filled with anecdotes and humor, Ellenberg's storytelling makes complex ideas relatable and enjoyable.
- Broader Perspective: It encourages seeing connections between math and various fields, fostering a holistic understanding of math's influence on our lives.
What are the key takeaways of How Not to Be Wrong?
- Math is Everywhere: Mathematics is integral to everyday life, influencing decisions in politics, economics, and personal choices.
- Beware of Misleading Statistics: Ellenberg warns about misinterpreting statistical data, such as percentages and averages, using examples like obesity projections.
- Understanding Probability: The book emphasizes the importance of understanding probability, especially in scientific research and decision-making.
What are the best quotes from How Not to Be Wrong and what do they mean?
- "Mathematics is the extension of common sense by other means.": Math enhances our natural reasoning abilities, providing tools to understand complex situations better.
- "The armor goes where the bullet holes aren’t.": This highlights the importance of understanding survivorship bias and making decisions based on comprehensive data.
- "Improbable things happen a lot.": A reminder that unlikely events can occur, emphasizing the need for careful statistical reasoning.
How does Jordan Ellenberg explain the concept of linearity in How Not to Be Wrong?
- Linearity vs. Nonlinearity: Ellenberg contrasts linear thinking, assuming direct relationships, with nonlinear thinking, recognizing complex relationships.
- Real-World Examples: He uses the Laffer curve to show how tax rates and government revenue can have a nonlinear relationship.
- Importance of Context: Understanding context is crucial for informed decisions, as optimal solutions often lie between extremes.
What is Bayesian inference as discussed in How Not to Be Wrong?
- Definition of Bayesian Inference: A statistical method that updates the probability for a hypothesis as more evidence becomes available.
- Prior and Posterior Probabilities: Combines prior beliefs with new data to form a posterior belief, illustrated through examples like roulette outcomes.
- Real-World Applications: Applied in fields like medicine and social sciences to improve decision-making, allowing for a nuanced understanding of uncertainty.
How does How Not to Be Wrong address the misuse of statistics?
- Misleading Percentages: Ellenberg discusses how percentages can be manipulated to create misleading narratives, warning against taking statistics at face value.
- The Importance of Context: Context matters when interpreting statistics, as seen in job growth data where the nontradable sector dominated.
- Critical Evaluation: Encourages critical evaluation of statistical claims, reminding that statistical significance doesn't always mean practical significance.
What is the significance of the story of Abraham Wald in How Not to Be Wrong?
- Survivorship Bias: Wald's story illustrates survivorship bias, where conclusions are drawn based only on visible outcomes, ignoring those that didn't survive.
- Mathematical Insight: Wald's insight about where to place armor on planes during WWII demonstrates how mathematical reasoning can lead to better decision-making.
- Real-World Application: Ellenberg uses this story to show how mathematical thinking can be applied to real-world problems, encouraging critical thinking about data.
How does How Not to Be Wrong explain the concept of correlation?
- Correlation Explained: A statistical measure describing the extent to which two variables change together, indicating predictable relationships.
- Correlation Does Not Imply Causation: Just because two variables are correlated does not mean one causes the other, a common misconception.
- Visual Representation: Scatterplots are used to visualize correlations, revealing the strength and direction of relationships between variables.
What role does regression to the mean play in How Not to Be Wrong?
- Definition of Regression to the Mean: A statistical phenomenon where extreme observations tend to be closer to the average upon subsequent measurements.
- Implications for Performance: High performers are often subject to this phenomenon, leading to a natural decline in performance over time.
- Real-World Examples: Examples from sports, like baseball players' performance, show how regression to the mean can mislead expectations.
How does How Not to Be Wrong illustrate the importance of understanding statistics?
- Statistics in Everyday Life: Statistics play a crucial role in decision-making, from health choices to financial investments.
- Avoiding Misinterpretation: Highlights common pitfalls in interpreting statistical data, encouraging critical evaluation of data.
- Empowerment Through Knowledge: Understanding statistics empowers readers to make informed decisions and avoid being misled by misleading data.
What is the significance of the title How Not to Be Wrong?
- Focus on Avoiding Mistakes: The title reflects the book's theme of using mathematical reasoning to avoid common errors in thinking.
- Emphasis on Critical Thinking: Suggests a proactive approach to thinking critically about information and data.
- Broad Applicability: Implies that the lessons learned can be applied across various domains, from personal finance to scientific inquiry.
Download PDF
Download EPUB
.epub digital book format is ideal for reading ebooks on phones, tablets, and e-readers.