Start free trial
Trending New Genres Authors
EnglishEnglish
EspañolSpanish
简体中文Chinese
繁體中文Chinese (Traditional)
FrançaisFrench
DeutschGerman
日本語Japanese
PortuguêsPortuguese
ItalianoItalian
한국어Korean
РусскийRussian
NederlandsDutch
العربيةArabic
PolskiPolish
हिन्दीHindi
Tiếng ViệtVietnamese
SvenskaSwedish
ΕλληνικάGreek
TürkçeTurkish
ไทยThai
ČeštinaCzech
RomânăRomanian
MagyarHungarian
УкраїнськаUkrainian
IndonesiaIndonesian
DanskDanish
SuomiFinnish
БългарскиBulgarian
עבריתHebrew
NorskNorwegian
HrvatskiCroatian
CatalàCatalan
SlovenčinaSlovak
LietuviųLithuanian
SlovenščinaSlovenian
СрпскиSerbian
EestiEstonian
LatviešuLatvian
فارسیPersian
മലയാളംMalayalam
தமிழ்Tamil
اردوUrdu
Searching...
SoBrief
Experimentation Works

Experimentation Works

The Surprising Power of Business Experiments
Turn business hunches into fast, controlled tests, and learn which ideas deserve to scale.
by Stefan H. Thomke 2020 256 pages
3.9
362 ratings
Listen
Immersive
V2.3
Amazon Kindle Audible
Try Full Access for 3 Days
Unlock listening & more!
Continue

Key Takeaways

Treat executive certainty as a hypothesis, not a launch plan

Experience can mislead. Ron Johnson brought the successful Apple Store concept to J.C. Penney, then radically changed pricing and store design without first testing it with Penney shoppers. Sales fell, losses rose, and Johnson left after seventeen months. His success at Apple did not establish that the same changes would suit a different retailer and customer base.

Small ideas can surprise. At Microsoft’s Bing, a modest change to ad headlines was initially overlooked. A controlled test found a 12 percent revenue increase, worth more than $100 million annually in the United States, without damaging key user-experience measures. Experiments matter because they reveal how actual customers respond, not how experienced managers expect them to respond.

Analysis

The contrast is a useful reminder that expertise is often local. A leader may understand one customer group, business model, or moment in time extremely well, then mistake that familiarity for a portable law. Testing can challenge status and past success without dismissing either: experience helps generate promising questions, while observed behavior helps decide whether those questions hold in a new setting. The caution is that a test cannot make every strategic choice safe. It can, however, expose avoidable confidence before that confidence becomes an expensive rollout.

Turn uncertain ideas into falsifiable questions before spending at scale

Make the question testable. A business experiment manipulates a proposed cause, such as store hours, and measures an outcome, such as sales, against a comparison group. Kohl’s tested whether opening an hour later would reduce costs without meaningfully reducing sales. A hundred stores provided evidence that it would not.

Set the decision rules first. Before testing, clarify what is being changed, what result would count, whether the test is feasible and reliable, and whether decision-makers will act on the result. Intuit learned why behavior matters more than stated preference: twenty usability-study participants rejected a faster payroll option, but in a live test, 58 percent chose it. The feature later lifted conversion by 14 percent.

Analysis

A clear hypothesis does more than improve statistics. It forces a team to name the belief that connects an action to an outcome, making disagreement productive rather than personal. It also makes it harder to relabel a rollout as an experiment after the decision is already settled. The Intuit result illustrates a broader measurement distinction: people’s reports reveal what they can articulate, while behavior reveals what they choose in a particular context. Neither alone explains motivation, so qualitative inquiry remains useful alongside testing.

Shorten learning loops with cheap models, fast feedback, and spare capacity

Test cheaply before polishing. A model is a simplified stand-in for a product, service, or process. Early models can be rough and inexpensive, then become more realistic as the design matures. Team New Zealand used simulations and scale models to improve its America’s Cup yacht, then tested full-sized boats in real conditions because simulations could miss practical effects.

Protect the speed of learning. Near the race, the team shifted toward frequent changes to the keel and tested them on a roughly 24-hour cycle. Two similar boats allowed side-by-side comparisons, reducing the influence of changing wind and water. More generally, overloading experimental teams creates queues and delays feedback. A little spare capacity can make uncertain work move much faster.

Analysis

This is an operational insight, not merely a call to work faster. The quality of learning depends on how quickly a team can form a question, build a test, observe the result, and revise its next move. The yacht illustrates why models should be treated as useful approximations, not miniature reality. A simulation can accelerate exploration, but high-stakes conclusions need evidence at an appropriate level of realism. Capacity also has a counterintuitive role: resources that appear idle can prevent slow queues from consuming the value of an idea.

Use controls to separate cause from coincidence, not just collect more data

Correlation is a clue, not a verdict. A controlled experiment randomly assigns comparable participants to a current approach or a changed version. The difference in outcomes can then be attributed more credibly to the change. In a Yahoo study, observational analysis suggested display ads increased brand searches by 871 to 1,198 percent. A controlled test found an increase of only 5.4 percent, showing how other changes can distort an apparently persuasive correlation.

Trust depends on design. Small samples, biased group selection, poor data, or several simultaneous changes can make results unreliable. Larger samples help detect small effects, but do not automatically fix clustered data or faulty comparisons. Even a statistically significant result is not proof that an effect is real or will recur; unexpected findings deserve checks and replication.

Analysis

The important distinction is between having more data and having a design that supports a causal conclusion. Data volume can shrink random error, but it cannot repair systematic bias or answer a question the study never isolated. This is especially important when teams search among many metrics or variants, since some apparent wins will arise by chance. A disciplined organization therefore treats statistical output as evidence with conditions attached, not as an oracle. Controls improve inference, while replication and domain knowledge help establish whether the result travels beyond the test.

Let small improvements compound across millions of customer interactions

Scale magnifies modest changes. An A/B test compares a current version with a modified version, assigning users to each so their outcomes can be compared. At Microsoft, opening Hotmail in a new browser tab increased user engagement by 8.9 percent in a United Kingdom test and was later confirmed in the United States. The change required only a small amount of code.

Many small wins can add up. In 2007, Barack Obama’s campaign tested combinations of images, videos, and page elements. The strongest version increased sign-ups by 41 percent over the original, with donations estimated to rise by about $60 million. Not every test produces a winner. The advantage comes from making tests cheap and routine enough to find, validate, and scale useful improvements.

Analysis

The digital setting makes the economics unusually favorable: once a change works, it can reach enormous numbers of users at little additional cost. This helps explain why a minor interface choice can matter more than an elaborate feature with a large development budget. Still, scale is an amplifier, not a guarantee of value. A small lift on a narrow or misleading metric can spread a bad experience just as efficiently as a good one. The right goal is not to maximize the count of changes, but to improve outcomes that matter to customers and the business.

Reward learning from failed tests, while refusing to repeat preventable mistakes

A negative result can be useful. The author distinguishes a failed idea from a failed experiment. A well-designed test that rejects an idea narrows the search and may reveal what to try next. A poorly designed test, or repeating a known error, produces little learning. At Booking.com, roughly nine out of ten experiments had no positive effect on the chosen measures, yet the company kept testing because useful improvements emerged from the larger portfolio.

Make the incentives credible. Employees will not experiment freely if managers praise learning but punish every disappointing result. Research described in the book found that inconsistent signals particularly discouraged lower-status staff. Psychological safety does not mean accepting sloppy work. It means people can surface bad news, question assumptions, and explain what a test taught without fear that an honest result will damage their standing.

Analysis

The distinction between learning and carelessness protects experimentation from becoming a slogan for any unsuccessful initiative. A useful failure has a question, a disciplined method, and information that changes what the team does next. This connects openness to accountability: leaders should protect people who reveal uncertainty while scrutinizing whether the test was competently designed. The asymmetry of status matters because junior employees often bear greater personal risk when challenging a senior belief. A culture that wants evidence must make it safe for evidence to be inconvenient.

Build an experimentation system, not merely a testing tool

Infrastructure is organizational. The book identifies seven system levers: scale, scope, and speed in the process; standards and support in management; and shared values and skills in culture. These reinforce one another. A platform cannot create broad testing if staff lack training, leaders ignore results, or teams have no time to run experiments.

Maturity takes stages. The ABCDE framework describes a progression from awareness that experimentation matters, through belief and commitment, to wider diffusion and deep organizational embeddedness. State Farm moved from one or two tests a month to a faster, broader program after adopting a third-party tool and sharing results in regular business meetings. Leaders’ role shifts from choosing every answer to setting goals, building trustworthy systems, and modeling openness to being contradicted.

Analysis

The system perspective explains why buying software often produces disappointing returns. Tools lower the cost of testing, but complementary changes determine whether the organization can absorb what the tests reveal. This resembles the broader pattern in technology adoption: productivity gains depend on redesigning workflows, responsibilities, and skills, not simply installing equipment. The maturity model is helpful as a diagnostic, although organizations need not move through its stages in a perfectly neat sequence. Different departments may be advanced in infrastructure yet still weak in trust or decision-making discipline.

Give teams autonomy only when shared standards make results trustworthy

Booking.com paired freedom with discipline. Teams could launch tests across its website, apps, partner tools, and internal systems. Its platform automated random assignment and reporting, kept a searchable record of prior tests, and made results visible across the company. In 2017, employees ran more than one thousand concurrent experiments on a typical day. A shared primary measure, bookings per day, gave teams a common reference point, while teams could also track other outcomes.

Metrics need guardrails. Bookings measure immediate conversion, not whether guests remain satisfied or return. The company also drew on user research and customer-service feedback to identify problems and interpret results. Experiments involving scarcity messages raised ethical concerns, even when they improved conversion. Autonomy therefore worked alongside legal limits, open debate, and attention to the full customer experience.

Analysis

Booking shows that decentralization and standardization are not opposites. Central infrastructure can make local initiative safer by giving teams common procedures, comparable measurements, and a shared memory of previous attempts. The harder issue is choosing what success means. Conversion is observable quickly, while trust and loyalty take longer to reveal themselves. When a short-term proxy becomes the target, teams can optimize behavior in ways that customers later regret. Ethical review is not simply a brake on experimentation; it helps define which outcomes deserve to count as improvement.

Keep experiments in their lane: test execution, not every strategic choice

Not every question is testable. Experiments are especially useful when a team can compare plausible alternatives, measure meaningful effects, and change course. They are less suited to decisions such as entering an entirely new market, where no reliable baseline may exist. A/B tests can also favor small improvements to an existing product, so companies need qualitative research and bolder exploration to discover needs that current metrics cannot capture.

Evidence informs judgment. Netflix chose promotional images that included Jane Fonda, despite a test suggesting images featuring only her co-star drew more clicks. The choice reflected contractual and strategic considerations. Similarly, customer experiments require ethical boundaries. Comparing a new option with an untested status quo can attract criticism, but that does not excuse manipulative tests or make the status quo automatically harmless.

Analysis

Experimentation is strongest as a discipline for learning under uncertainty, not as a universal replacement for leadership. A test can estimate what a defined population does under specified conditions; it cannot decide which values should govern a business or what risks society should accept. That boundary matters because measurable outcomes tend to crowd out less immediate concerns. Combining controlled tests with interviews, observation, and strategic reasoning gives decision-makers a wider view. The goal is not to abolish judgment, but to make its assumptions visible and its departures from evidence explicit.

Prepare for AI to multiply experiments, while keeping humans accountable

Automation expands the search. The book anticipates systems that use artificial intelligence to generate hypotheses, design and analyze tests, and recommend actions in a continuous loop. It points to engineering research using evolutionary methods: software generates many design variations, simulations assess their performance, and promising versions are selected for further development. Such methods can explore possibilities too numerous for people to test manually.

More tests need better governance. Automated experimentation could speed product development and make routine decisions more evidence-based. But it also raises questions about who defines success, which customers bear the risks, and how unintended effects are detected. The underlying tools do not decide what a business should value. Even a technically sound system needs human oversight of goals, constraints, and consequences.

Analysis

Automation changes the bottleneck from running tests to defining worthwhile questions and governing the search. When a system can generate variants at scale, it may efficiently optimize a narrow target while overlooking harms that were never encoded as measures. This is a familiar problem in algorithmic decision-making: what is easy to quantify can become a proxy for what matters, even when the proxy is incomplete. Human accountability is therefore not an obstacle to scientific method. It is the mechanism for deciding which experiments are legitimate and when a measurable gain is not a defensible gain.

Analysis

The book’s strongest claim is that experimentation is not a technical specialty but an organizational way to distribute learning. Its examples move from yacht design to search advertising, retail stores, and travel platforms, showing a consistent mechanism: uncertainty becomes manageable when teams can expose assumptions to controlled evidence, interpret the result, and revise action. The competitive advantage does not come from being right more often at the outset. It comes from finding errors earlier, cheaply, and repeatedly, while preserving the capacity to act on what is learned.

The framework is persuasive, but the method has boundaries. Randomized tests identify effects within a defined population and period; they do not automatically establish long-term value, explain why behavior changed, or guarantee that results generalize to another market. Metrics can become targets that displace harder-to-measure goods, such as trust, accessibility, or durable satisfaction. The Booking example makes this tension concrete: conversion is fast to measure, while loyalty and reputational harm may take much longer to emerge. Qualitative research, theory, and strategic judgment are therefore complements, not inferior substitutes.

Large-scale experimentation also redistributes authority. It can weaken hierarchy by letting employee ideas compete on evidence, but it can strengthen platform owners who choose the metrics, allocation rules, and acceptable risks. Democratization is genuine only when people can question those choices, not merely submit variants to an automated system. Ethical safeguards and transparent records are part of experimental quality because they determine whose outcomes count and who bears unintended costs.

The book’s lasting contribution is a model of disciplined organizational humility. It refuses both managerial certainty and naive faith in data. Its most useful question is not simply whether a change won, but what the result teaches, for whom it applies, and what should happen next. That standard remains essential as testing becomes cheaper and automation makes the number of possible experiments vastly larger.

Summary in 30 Seconds
Controlled tests distinguish a change's real effects from coincidence, while customer behavior can differ sharply from stated preferences. Small changes can create substantial gains at scale: one headline test raised revenue by 12 percent, worth over $100 million annually in the United States. Many experiments produce no positive result, but a well-designed negative finding still guides the next decision. Reliable programs need shared standards, trustworthy measures, ethical safeguards, and leaders willing to revise assumptions. Automated testing can expand discovery, but people must set its goals and limits.
Contains spoilers

Last updated:

Report Issue
Want to read the full book?

Glossary

Experimentation wheel

Iterative learning cycle

A recurring process for turning uncertain ideas into tested knowledge. Teams develop testable hypotheses, run disciplined experiments, analyze outcomes, and use what they learn to revise or replace the next hypothesis. The cycle is designed to connect each test to the next decision, rather than treating experimentation as a one-time check at the end of development.

High-velocity incrementalism

Fast cumulative improvement

An innovation approach based on making and testing many relatively small changes quickly. Individual improvements may be modest, but they can accumulate into substantial gains when repeated across a large customer base or operating system. The approach depends on low-cost tests, rapid feedback, and the ability to discard weak ideas without delaying promising ones.

A/B test

Randomized variant comparison

A controlled experiment that randomly assigns participants to a current version, called the control, or a changed version, called the treatment. The experimenter compares outcomes on selected measures to estimate the treatment’s effect. A/B tests are common online because large user populations and automated tracking make it possible to test changes quickly and repeatedly.

Overall evaluation criterion (OEC)

Strategic success measure

A metric, sometimes composed of several measures, used to judge whether an experiment supports a company’s longer-term objectives. The OEC should distinguish meaningful improvement from short-term gains that could damage future performance. For example, a search company must avoid rewarding changes that raise immediate clicks or revenue by making the underlying service less useful.

ABCDE framework

Experimentation maturity stages

A model of organizational progress through five stages: awareness, belief, commitment, diffusion, and embeddedness. It describes a shift from merely recognizing the value of experiments to making disciplined testing a broadly shared and routine part of decision-making. The stages reflect growing leadership support, resources, skills, standards, and employee participation.

System levers

Seven capability-building factors

The coordinated elements used to build an experimentation organization: scale, scope, and speed in the process; standards and support in management; and shared values and skills in the culture. The factors reinforce one another. For example, expanding the number of tests is difficult if employees lack the training, trust, or organizational permission needed to run and interpret them.

HiPPO

Highest-ranking opinion

An acronym for the highest-paid person’s opinion, used to describe decisions driven by seniority or authority rather than evidence. The term highlights how hierarchy can cause organizations to favor a powerful person’s assumptions over information from customers or controlled tests. Experimentation can counter this tendency, although leaders still shape the goals and rules of testing.

Semmelweis reflex

Rejection of contrary evidence

The reflexive dismissal of findings that conflict with established beliefs, interests, or professional norms. The term refers to physician Ignaz Semmelweis, whose evidence that handwashing reduced deadly childbirth infections was rejected by much of the medical establishment. In organizations, this reflex can make people distrust a rigorous result simply because it contradicts intuition or accepted practice.

A/B illusion

Uneven scrutiny of experiments

The tendency to judge a controlled test as ethically suspect while overlooking that the existing practice may have been adopted without comparable evidence. People often focus on the visible risks of assigning different groups to alternatives and treat the status quo as inherently justified. The concept does not remove ethical obligations; it encourages scrutiny of both experimental and routine practices.

Causal density

Many interacting influences

The complexity created when numerous variables and interactions affect a business outcome. Store sales, for example, may depend on location, weather, staffing, local competition, customer income, and promotions, as well as the change being tested. High causal density makes it difficult to isolate cause and effect and increases the need for randomization, matched controls, or careful statistical analysis.

Download PDF

To save this Experimentation Works summary for later, download the free PDF. You can print it out, or read offline at your convenience.
Download PDF
File size: 0.26 MB     Pages: 16

Download EPUB

To read this Experimentation Works summary on your e-reader device or app, download the free EPUB. The .epub digital book format is ideal for reading ebooks on phones, tablets, and e-readers.
Download EPUB
File size: 1.25 MB     Pages: 20
Want to read the full book?
Follow
Listen
Home
Swipe
Library
Get App
Try Full Access for 3 Days
Listen, bookmark, and more
Compare Features Free Pro
📖 Read Summaries
Read unlimited summaries. Free users get 3 per month
🎧 Listen to Summaries
Listen to unlimited summaries in 40 languages
—
❤️ Unlimited Bookmarks
Free users are limited to 4
—
📜 Unlimited History
Free users are limited to 4
—
📥 Unlimited Downloads
Free users are limited to 1
—
Risk-Free Timeline
Today: Get Instant Access
Listen to full summaries of 26,000+ books. That's 12,000+ hours of audio!
Day 2: Trial Reminder
We'll send you a notification that your trial is ending soon.
Day 3: Your subscription begins
You'll be charged on Nov 24,
cancel anytime before.
Consume 2.8× More Books
2.8× more books Listening Reading
Our users love us
1M+ readers
Trustpilot Rating
TrustPilot
4.6 Excellent
This site is a total game-changer. I've been flying through book summaries like never before. Highly, highly recommend.
— Dave G
Worth my money and time, and really well made. I've never seen this quality of summaries on other websites. Very helpful!
— Em
Highly recommended!! Fantastic service. Perfect for those that want a little more than a teaser but not all the intricate details of a full audio book.
— Greg M
Save 62%
Yearly
$119.88 $44.99/year/yr
$3.75/mo
Monthly
$9.99/mo
Start a 3-Day Free Trial
3 days free, then $44.99/year. Cancel anytime.
Unlock a world of fiction & nonfiction books
26,000+ books for the price of 2 books
Read any book in 10 minutes
Discover new books like Tinder
Request any book if it's not summarized
Read more books than anyone you know
#1 app for book lovers
Lifelike & immersive summaries
30-day money-back guarantee
Download summaries in EPUBs or PDFs
Cancel anytime in a few clicks

Why one summary?

Scanner
Find a barcode to scan

⭐
⭐
⭐
We have a special gift for you
Open
38% off
DISCOUNT FOR YOU
⭐
⭐
⭐
$79.99
$49.99/year
Only $4.16 per month
Continue
2 taps to start, super easy to cancel
Settings
General
Widget
Loading...