Key Takeaways
Treat every published finding as a claim awaiting independent checks
Science needs witnesses. A study becomes scientific knowledge not simply because someone ran an experiment, but because other researchers can inspect the methods and test whether the result holds up. The book opens with Daryl Bem’s 2011 claim that students could somehow anticipate future events. Stuart Ritchie and colleagues repeated one of Bem’s word-memory experiments three times and found no effect. The journal that had published Bem’s paper refused to consider their replication, citing a policy against publishing repeats.
Replication is a test. It means trying to find the same result with new data; reproducibility usually means checking whether the same data and methods yield the same result. A large project that retested 100 psychology studies found that only 39 percent clearly replicated. That figure is not a census of all psychology, but it shows why a striking result deserves independent scrutiny before it becomes a settled fact.
What's striking is that replication is not a demand for identical results in every lab. It is a demand to learn whether a claim survives reasonable variation in people, settings, and measurement. That is close to how engineering tests a bridge: a design is not trusted because it stood once, but because it performs under multiple conditions. The analogy has limits, since human behavior can be more context-sensitive than a steel beam. Still, journals that publish dramatic first findings while rejecting competent checks create an asymmetric record: novelty gets visibility, while verification struggles to get credit.
Publish null results before silence turns them into false consensus
The missing studies matter. Publication bias occurs when the chance of a study appearing in the literature depends on its result. Positive findings are more likely to be written up and accepted; null or mixed findings often disappear. A review of nearly 2,500 papers found positive results in 91.5 percent of psychology and psychiatry papers, compared with 70.2 percent in space science. Such a lopsided record is hard to explain through good guesses alone.
The file drawer is real. In a study tracking completed social-science projects, 22 percent had null results, but only 9 percent of published articles did. Strong results made up 41 percent of completed projects and 53 percent of published work. When researchers combine only what was published, the apparent effect can grow, and a treatment or theory may look more dependable than it is. The people who took part in unpublished studies also contributed effort without adding their results to shared knowledge.
A missing result is not merely an empty space in a database. It changes the apparent balance of evidence, much as a poll that omits people who declined to answer can misdescribe public opinion. This is why registration and complete reporting matter even when the result seems uninteresting. Yet not every imbalance proves deliberate suppression: a study may be incomplete, poorly designed, or genuinely difficult to publish for sound reasons. The useful question is whether the research record tracks what was done, rather than only what made a persuasive story.
Fund studies large enough to detect the effects they claim
Small samples magnify chance. Statistical power is a study’s chance of detecting an effect if that effect is real. When a study has too few observations, it tends to detect only unusually large results, and those unusually large results may be flukes. Follow-up researchers then plan around an inflated estimate, creating a chain of weak studies that appear to confirm one another.
Genetics learned this painfully. Early candidate-gene studies linked individual DNA variants to complex traits such as depression and cognitive ability, sometimes reporting striking effects. Later genome-wide association studies tested far larger samples across many genetic variants. They did not find the promised large effects. Instead, complex traits appeared to involve thousands of variants, each with a tiny contribution. The earlier literature had mistaken noisy outliers for strong signals. Large-scale replications in psychology also found that successful replications often produced smaller effects than the original studies.
This is a statistical version of selection bias: weak studies are most likely to enter the record when they happen to produce an impressive result. Large samples do not guarantee truth, since biased recruitment or poor measures can distort even enormous datasets. But a study designed with enough power is less dependent on lucky noise and can detect modest effects more realistically. The practical challenge is that large studies cost money and time. That makes underpowered research not merely a personal error, but also a funding and career-design problem.
Do not confuse a prestigious journal with an honest result
Fraud exploits trust. Scientific communities cannot independently witness every experiment, so they rely partly on researchers and institutions to report honestly. Paolo Macchiarini, a prominent surgeon at Sweden’s Karolinska Institute, claimed that stem-cell-seeded synthetic windpipes could save patients. Several patients died or suffered catastrophic complications, while his papers described far more favorable outcomes. An independent inquiry found serious misconduct, but the institute initially defended him and pursued some of the doctors who raised concerns. Multiple papers were eventually retracted.
The harm can escape the lab. Andrew Wakefield’s 1998 paper, based on twelve children, falsely linked the MMR vaccine to autism. Investigations found that medical histories had been misrepresented and that financial interests had not been disclosed. The paper helped fuel a vaccine scare, while subsequent large studies found no vaccine-autism link. Fraud is unusual compared with ordinary error, but its consequences can spread through medicine, public policy, and public trust.
These cases show why institutional accountability matters alongside individual integrity. A prestigious name can lend a weak or dishonest claim extra authority, and the institution may have incentives to protect its reputation rather than investigate promptly. At the same time, skepticism must not become indiscriminate suspicion: a failed result is not evidence of fraud, and allegations need careful investigation. The strongest safeguards therefore combine independent oversight, transparent records, and fair procedures for whistleblowers and accused researchers alike.
Lock analysis plans before the data start telling stories
Flexibility creates false leads. P-hacking means trying multiple analyses, exclusions, outcomes, or stopping points, then emphasizing the version that produces a publishable result. With a single test at the conventional 0.05 threshold, a false positive is possible; repeated tests raise the odds of finding at least one by chance. Five independent tests create a 23 percent chance of at least one false positive, and twenty create a 64 percent chance.
Brian Wansink made the temptation visible. After an initial food study failed to support its hypothesis, the Cornell researcher encouraged a student to keep searching the dataset for something useful. Wansink’s published work later drew scrutiny for extensive statistical and reporting problems. HARKing, or hypothesizing after results are known, is a related practice: researchers discover a pattern, then present it as though they predicted it. Pre-registering hypotheses and analyses makes the distinction between planned tests and exploratory searches visible.
Pre-registration works like a public commitment device. It does not prevent researchers from noticing surprises; it asks them to label discoveries made after looking at the data as exploratory, so another study can test them. That distinction protects creativity without pretending that every intriguing pattern was predicted in advance. The method is not a cure-all: a plan can be poorly designed, and registered studies can still go unreported. Its value is more modest and important: it makes analytic decisions easier to inspect and harder to rewrite as certainty.
Build routine error checks into research, not just good intentions
Ordinary mistakes accumulate. A spreadsheet typo weakened a high-profile economics paper claiming that debt above 90 percent of GDP predicted economic decline. Correcting the omitted countries changed the reported average growth above that threshold from minus 0.1 percent to plus 2.2 percent. A statistical-checking program applied to psychology papers found that nearly half contained at least one numerical inconsistency; 13 percent had a serious error that could change the conclusion.
Negligence can become structural. Cell lines used in biomedical research are sometimes mislabeled or contaminated, causing researchers to study the wrong cells. One review identified more than 32,000 papers that used known misidentified cell lines. In animal research, basic safeguards such as random assignment and blinding were often missing or unreported. These are not merely cosmetic defects: poor methods can waste money, misdirect follow-up work, and make painful animal experiments uninformative.
Routine checking is a form of safety engineering. Aviation reduces repeated failures by treating near misses and small defects as information about the system, not as embarrassing exceptions to hide. Research can use a similar logic through automated checks, authenticated materials, clear protocols, and independent review of data pipelines. Automation also has limits: software can contain bugs, and a clean report does not prove a sound design. Human judgment remains essential, but it should be supported by tools that catch predictable mistakes before publication.
Keep public claims smaller than the evidence beneath them
A press release can amplify uncertainty. NASA presented a study of bacteria from California’s Mono Lake as evidence that life might use arsenic in place of phosphorus. The striking claim attracted worldwide attention, but follow-up experiments found no evidence that arsenic replaced phosphorus in the bacteria’s DNA. The episode showed how a preliminary result can become a sweeping headline before its foundations are secure.
The leap from model to person is especially risky. A study in mice is not evidence that a treatment works in humans, and a correlation does not establish cause. A review of health-science press releases found unwarranted behavioral advice in 40 percent, claims that leapt from animals to humans in 36 percent, and causal language for observational findings in 33 percent. The book also shows how popular accounts can inflate modest findings, such as growth-mindset interventions whose average academic effects were small.
Careful communication is not a retreat from public engagement. It is a way to preserve the reader’s ability to tell a preliminary signal from a useful conclusion. The distinction resembles weather forecasting: a probability of rain can guide decisions without being falsely translated into certainty. There is a real tension, since cautious language can be harder to notice in a crowded media environment. But exaggeration is a short-term attention strategy with long-term costs, especially when people make health or policy decisions based on claims that outrun the evidence.
Reward reliable knowledge, not publication counts or novelty
Researchers respond to the scoreboard. Universities and funders often reward publication volume, prestigious journals, citations, and grant income. These measures can encourage salami-slicing, where one body of work is split into several papers, and hype, selective reporting, or corner-cutting. When researchers are judged by the number of papers rather than their reliability, the system selects for output that looks productive, not necessarily work that improves knowledge.
Change the incentives and the workflow. The book recommends wider access to data and methods, more replication studies, public registration of study plans, and Registered Reports, where a journal agrees to publish a study based on its design before the results are known. It also argues that hiring and funding should recognize careful methods, collaboration, and complete reporting. For readers, a practical check is to ask whether a finding has replicated, how large the effect is, and whether the study’s design supports the headline claim.
The deeper argument is institutional: reliable conduct should be easier to pursue and more rewarding than gaming a metric. This echoes a broader lesson from organizational design: people often adapt to what an institution measures, even when the measure was intended only as a proxy. Replacing one score with another can simply create a new target to manipulate. The most promising reforms combine transparent procedures with plural measures of contribution, including replication, data stewardship, and collaboration, while leaving room for unconventional research that may not pay off immediately.
Analysis
Science Fictions is most persuasive when it shifts the diagnosis from individual villains to an ecosystem that routinely rewards the wrong outputs. Fraud is dramatic, but the more pervasive danger is quieter: selective publication, flexible analysis, weak design, and inflated communication can produce a misleading literature without anyone deciding to deceive. The result is not that science has no authority. It is that authority should attach to bodies of convergent evidence, not automatically to a single paper or a journal’s reputation.
The book’s narrative gains force from cases that travel from laboratory practice into public consequence. Bem’s failed precognition result illustrates how publication without replication can make an extraordinary claim look legitimate. Macchiarini and Wakefield show how institutional prestige can magnify harm. The candidate-gene story demonstrates that a whole research program can be built on effects that larger samples later fail to confirm. Together, these episodes make meta-science, the study of research itself, feel less like administrative housekeeping and more like essential quality control.
A useful nuance is that replication rates depend on what is sampled, what counts as success, and how closely a replication matches the original. The 39 percent result from psychology should not be treated as a universal failure rate for science. Nor does a failed replication automatically prove the original false. The book is strongest when it presents these figures as evidence of uncertainty and weak checking, rather than as a final verdict on every field.
Its reform agenda is compelling because it targets incentives and procedures together: preregistration, open materials, registered reports, larger collaborations, and better recognition for careful work. None is sufficient alone. Transparency can expose mistakes but cannot guarantee good theory; metrics can be replaced and gamed; open results can be misrepresented outside the lab. The durable lesson is epistemic rather than cynical: trust scientific conclusions in proportion to the strength, independence, and openness of the evidence supporting them.
Review Summary
Science Fictions is praised for its insightful examination of fraud, bias, and hype in scientific research. Ritchie offers a comprehensive overview of the replication crisis and proposes solutions to improve scientific practices. While some readers find it eye-opening and essential, others criticize its narrow focus on recent examples and potential oversimplification. The book is lauded for its clear writing and engaging examples, but some argue it may erode trust in science. Overall, it's recommended for both scientists and laypeople interested in understanding the current state of scientific research.
People Also Read
Glossary
Mertonian Norms
ideals for scientific conductFour norms proposed by sociologist Robert Merton to describe ideals for science: universalism, judging claims by their evidence rather than the claimant’s status; disinterestedness, pursuing knowledge rather than private gain; communality, sharing findings; and organised scepticism, withholding acceptance until claims have been examined. Ritchie uses them as a benchmark for comparing scientific ideals with actual publication incentives.
Replication crisis
doubts about repeatabilityThe widespread concern that many published scientific findings do not recur when researchers repeat the studies with new data. The crisis became especially visible in psychology, where large collaborative projects found that many prominent results failed to replicate or appeared smaller on retesting. It raises questions about study design, selective publication, statistical practices, and how confidently single studies should be treated.
Publication bias
results shape publication chancesA distortion that occurs when studies with positive, striking, or statistically significant results are more likely to be written up or published than null or mixed findings. The resulting literature overrepresents apparent successes and can exaggerate the size or existence of effects. The related file-drawer problem describes completed studies that remain unpublished, hidden from researchers trying to assess the full evidence.
p-hacking
searching analyses for significanceThe use of flexible analytic choices to obtain or emphasize a statistically significant result. Researchers may test different outcomes, exclude observations, divide participants into subgroups, or collect more data until a favorable result appears. These choices can increase false positives, particularly when readers are not told how many analyses were attempted or which decisions were made after seeing the data.
HARKing
hypotheses written after resultsAn acronym for hypothesizing after the results are known. A researcher first notices a pattern in collected data, then describes it in the paper as though it had been predicted before the study. Exploratory analysis can generate valuable questions, but presenting a discovery as a prior prediction hides the fact that chance patterns become more likely when many possibilities are examined.
statistical power
chance of detecting a real effectThe probability that a study will identify an effect when that effect truly exists. Power depends partly on sample size, the size of the effect, and the quality of measurement. Underpowered studies often miss modest real effects; when they do report positive results, those estimates can be unusually large and unstable. Planning a study requires deciding what effect it should be able to detect.
Open Science
accessible research materialsAn approach that makes as much of the research process as possible available for inspection and reuse. Depending on the study, this can include data, analysis code, materials, protocols, registrations, and peer reviews. Openness supports scrutiny and collaboration, helps reveal errors, and reduces dependence on unverified trust. Privacy, safety, or legal duties can make some data inappropriate to release publicly.
Registered Reports
publication based on study designA publishing format in which researchers submit their research question, methods, and planned analyses for peer review before collecting or analyzing the data. If the design is accepted, the journal commits to publishing the completed study regardless of whether the results are positive or null, provided the researchers follow the approved plan. This weakens the incentive to select results for publication.
meta-science
research on researchThe systematic study of how scientific research is conducted, evaluated, communicated, and corrected. Meta-scientists examine issues such as replication, reporting errors, publication bias, study power, and research incentives. By applying measurement and testing to scientific practice itself, meta-science can identify where the system produces unreliable knowledge and assess whether proposed reforms improve it.
Goodhart's Law
targets corrupt useful measuresThe principle that a measure can stop being a good measure once people are rewarded for maximizing it. In science, publication counts, citation totals, and journal impact factors may begin as rough indicators of contribution, but become targets that can be inflated through practices such as salami-slicing or coercive citation. The metric then tracks strategic behavior as much as research quality.
FAQ
What's Science Fictions about?
- Exploration of Scientific Integrity: Science Fictions by Stuart Ritchie examines how fraud, bias, negligence, and hype undermine the pursuit of truth in scientific research.
- Focus on Replication Crisis: The book highlights the replication crisis, where many studies fail to reproduce results, questioning the reliability of published research.
- Call for Reform: Ritchie advocates for reforms in scientific practices, emphasizing transparency, open data, and better incentives for researchers.
Why should I read Science Fictions?
- Understanding Scientific Flaws: The book provides insight into systemic flaws in scientific research, encouraging critical evaluation of scientific claims.
- Awareness of Hype: It highlights how sensationalized claims can mislead the public, stressing the importance of humility in science communication.
- Proposed Solutions: Ritchie offers practical solutions for reforming scientific practices, making it valuable for researchers and the general public.
What are the key takeaways of Science Fictions?
- Importance of Replication: Ritchie stresses that credible scientific findings must be replicable, a fundamental aspect often neglected.
- Impact of Bias: The book illustrates how biases can skew research results, leading to a distorted view of scientific truths.
- Need for Transparency: Ritchie calls for greater transparency in research practices, including data sharing and pre-registration of studies.
What are the best quotes from Science Fictions and what do they mean?
- “Science has become home to a dizzying array of incompetence, delusion, lies and self-deception.”: Highlights the serious issues compromising scientific integrity.
- “If it won’t replicate, then it’s hard to describe what you’ve done as scientific at all.”: Emphasizes the critical role of replication in validating research.
- “We owe it the truth.”: Stresses the responsibility of scientists to pursue and communicate accurate findings.
What is the replication crisis discussed in Science Fictions?
- Definition of the Replication Crisis: Refers to the realization that many studies, especially in psychology, cannot be reliably reproduced.
- Impact on Trust: The crisis has eroded public trust in science, highlighting the need for rigorous research practices.
- Broader Implications: Affects various scientific disciplines, undermining public trust and influencing policy and practice.
How does Science Fictions address scientific fraud?
- Case Studies of Fraud: Ritchie examines notable cases like Diederik Stapel's fabricated data, illustrating severe consequences.
- Consequences of Fraud: Discusses how fraud damages reputations and erodes public trust in science.
- Call for Accountability: Advocates for stricter measures to detect and prevent fraud, emphasizing accountability.
What is publication bias, and how does it affect scientific research according to Science Fictions?
- Definition of Publication Bias: Occurs when studies with positive results are more likely to be published than those with null results.
- Consequences for Science: Creates a false narrative about the effectiveness of treatments, misleading practitioners and policymakers.
- Impact on Meta-Analyses: Affects meta-analyses by overestimating effectiveness due to exclusion of null results.
What is p-hacking, and why is it a problem in scientific research according to Science Fictions?
- Definition of P-Hacking: Involves manipulating data or analyses to achieve a desired p-value, often below 0.05.
- Consequences for Validity: Undermines research validity, leading to false positives and exaggerated effect sizes.
- Widespread Issue: Common practice among researchers, often done unconsciously, raising concerns about study reliability.
How does Science Fictions propose to reform scientific practices?
- Advocacy for Transparency: Calls for sharing data and methodologies to allow better scrutiny and replication.
- Encouragement of Replication Studies: Emphasizes the need for more replication studies to validate research findings.
- Reevaluation of Publication Practices: Suggests journals should publish null results and replication studies to combat bias.
What role does bias play in scientific research according to Science Fictions?
- Types of Bias: Includes confirmation bias, funding bias, and ideological bias, distorting research outcomes.
- Impact on Results: Leads to selective reporting, creating a distorted view of evidence.
- Need for Self-Criticism: Emphasizes self-criticism to recognize and address biases, maintaining research integrity.
How does Science Fictions suggest addressing the issue of scientific hype?
- Cautious Communication: Advocates for humility and caution in communicating findings, avoiding sensationalized claims.
- Media Responsibility: Suggests journalists should critically evaluate scientific claims to prevent hype.
- Systemic Changes: Calls for changes in publication processes to reduce pressure for flashy results.
What is the significance of open science as discussed in Science Fictions?
- Transparency and Accountability: Promotes transparency, allowing verification of findings and methodologies.
- Collaboration and Sharing: Facilitates data sharing and collaboration, leading to robust findings.
- Public Access to Research: Discusses ethical implications of making taxpayer-funded research publicly available.
Download PDF
Download EPUB
.epub digital book format is ideal for reading ebooks on phones, tablets, and e-readers.