Key Takeaways
Judge a platform by what it amplifies, not only what it removes.
The feed is a choice. Facebook's public debate often focused on whether particular posts should be deleted. But the platform's ranking systems decided which posts billions of people were most likely to encounter. The book's central argument is that amplification can shape public life as powerfully as removal: a post need not break a rule to gain enormous reach, and a harmful post can spread widely while remaining technically permissible.
A complaint system shows the difference. Former engineer Arturo Bejar found that Facebook had replaced a carefully designed way for users to report bad experiences with a confusing process that could silently discard reports. His daughter had received sexual harassment on Instagram, yet the platform did not treat it as a violation. The problem was not just individual moderation calls. It was a product built to hear some signals and ignore others.
What's striking is how this shifts the question from speech policing to infrastructure. A platform can leave every post technically available while still deciding which voices get a megaphone. That distinction resembles the difference between a library holding a book and a broadcaster scheduling it during prime time. Both involve access, but only one actively concentrates attention. The framing also avoids an easy answer: amplification is not inherently wrong, and ranking can help people find useful material. The harder question is what standards should govern a system that distributes attention at industrial scale.
Engagement metrics can turn a product goal into a social hazard.
The metric changed the feed. In 2018, Facebook announced that it would prioritize Meaningful Social Interactions, a score built from measurable actions such as comments, shares, and reactions. The aim sounded humane: encourage connection rather than passive scrolling. In practice, the system rewarded interactions without judging their tone. An angry response counted as engagement, just as a supportive one did.
Outrage traveled well. Facebook's own research found that negative reactions helped posts spread. Political strategists in Poland and Spain told researchers they were shifting toward more attacks because aggressive posts performed better. One party reported moving from roughly half positive posts to four-fifths negative. Facebook said the change reduced daily usage by 50 million hours, but that sacrifice did not establish that the measure improved users' experience.
This is a vivid instance of Goodhart's law: once a measure becomes a target, people adapt to the measure rather than the underlying purpose. Comments can indicate connection, but they can also register fury, mockery, or conflict. The distinction matters because a platform's metric does not merely describe behavior. It changes the rewards that shape behavior. A fair qualification is that no single ranking score can capture the meaning of every interaction. The failure was treating an easy-to-count signal as if it reliably represented a difficult human good.
Design for adversaries, not just ordinary users.
Bad actors learn the rules. Facebook's systems were built around patterns of ordinary use, while opportunists learned which signals earned reach. Former data scientist Jeff Allen called one tactic manufactured virality: take popular material from an existing community, repost it through a large page, and let the platform's own recommendations distribute it. Original reporting was often riskier and slower than recycling a post that had already proved popular.
The scale was striking. In one review, 33,000 pages, about 0.175 percent of pages, accounted for a quarter of page views while producing only 0.14 percent of Facebook revenue. A separate sample found that 40 percent of large publishers relied on stolen, aggregated, or minimally altered material. The lesson was not that every repost is malicious. It was that predictable, low-effort success can reveal a system's exploitable incentives.
Security engineering assumes that systems will be probed by people who do not share their designers' intentions. Social platforms need the same adversarial mindset. If repeated copying earns distribution, a platform should ask whether it is rewarding curation, spam, or both. The challenge is that copying and remixing are also ordinary cultural practices, so blunt rules can punish legitimate creators. Better defenses examine patterns across behavior, ownership, and distribution rather than treating any single repost as proof of abuse.
Give integrity teams decision rights, not just better dashboards.
Evidence lacked authority. Facebook hired researchers and engineers to study misinformation, polarization, and other harms. Their findings often generated proposed fixes, but proposals could be weakened or blocked when they threatened growth or political relationships. Carlos Gomez-Uribe's Sparing Sharing proposal aimed to reduce the influence of hyperactive accounts that disproportionately drove recommendations. Mark Zuckerberg approved a version with only one-fifth of the intended weighting change.
The incentives stayed intact. The company treated daily users as a near-sacred measure, while integrity teams were expected to repair harms without materially changing the growth machinery that helped produce them. Public Policy staff also reviewed product decisions for political risk, sometimes with the final say. The result was a recurring pattern: the company could document a problem, discuss solutions, and still leave the central incentives largely untouched.
The organizational lesson is sharper than a simple claim that executives ignored experts. Researchers can be heard, funded, and praised while remaining unable to alter the decisions that matter. In governance terms, voice without authority is not accountability. Separating safety judgments from lobbying interests could reduce conflicts, though it would not remove hard tradeoffs or guarantee wise policy. The crucial test is whether teams responsible for harms can pause a launch, change a ranking objective, or require evidence before growth-oriented changes go live.
Slow viral mechanics before crisis; content filters cannot keep pace.
Friction can prevent spread. Facebook's emergency Break the Glass measures changed how the platform distributed content during high-risk periods. In Myanmar, limiting reshares reduced viral inflammatory posts by 25 percent and viral hoax photos by 49 percent. The change did not reduce users' sessions, though it lowered Meaningful Social Interactions by about 2 percent. Facebook planned to end the intervention after the election.
The emergency brake came late. Ahead of the 2020 US election, the company used some measures to limit group growth and hateful engagement. After the election, many were rolled back. As Stop the Steal groups surged and the January 6 crisis unfolded, Facebook reinstated emergency restrictions, but only after the movement had grown. The book's evidence suggests that slowing forwarding and recommendations can reduce reach without relying entirely on imperfect post-by-post moderation.
This resembles safety engineering in fields where preventing a hazard is more dependable than detecting every failure after it begins. Slowing a distribution channel can be less intrusive than deleting individual speech, and it may protect legitimate discussion from being drowned out by coordinated surges. Yet emergency controls raise genuine questions about who declares a crisis, how long restrictions last, and whether users can challenge them. The Myanmar example also shows that a modest engagement cost can coexist with substantial reductions in harmful spread.
A translated app is not a locally governed platform.
Language is part of safety. Facebook expanded rapidly across countries without matching that growth with enough local moderators, researchers, or language-specific detection systems. Automated tools that could identify some visual content did not readily transfer to culturally specific hate speech or incitement. In Myanmar, one early Burmese-language reviewer was expected to cover a country of more than 50 million people, while users encountered anti-Rohingya propaganda and calls for violence.
The blind spots had consequences. A test account in India was quickly recommended nationalist material, graphic images, and false stories during a period of military tension. Facebook also failed to address human trafficking on Instagram consistently until Apple threatened to remove its apps. The company's strongest responses often followed public or commercial pressure, even when the human harm was already visible to local employees and advocates.
A service can be globally available while its protection is effectively local only in wealthy, well-resourced markets. That is a design and governance problem, not just a translation problem: language, political context, and local power structures shape what counts as a threat. Automated systems can help, but they cannot substitute for contextual knowledge. At the same time, local enforcement carries risks of its own, including government pressure and inconsistent standards. Fairness requires both local capacity and safeguards against political capture.
Count what users experience, including harms your rules do not recognize.
Policy violations are not the whole story. Instagram's internal researchers found that the app could worsen body-image concerns for some teenage girls, especially those already dissatisfied with their appearance or social standing. In one survey, 40 percent of users who felt unattractive traced that feeling to Instagram. The platform's visual comparisons could hurt without any post violating a rule.
Bejar measured the missing signal. His Bad Emotional Experience Feedback survey asked users about harmful experiences, not merely reportable content. In one presentation, 51 percent of Instagram users said they had a bad or harmful experience in the previous week, yet only 1 percent reported the content; the company acted on only a small fraction of those reports. Among users aged 13 to 15, 24 percent reported receiving unwanted advances weekly.
This distinction echoes patient-reported outcomes in medicine: laboratory measures matter, but they do not replace the person's account of pain, function, or quality of life. A platform's rulebook can be enforced accurately while the experience remains unsafe or degrading. User surveys also have limits, including recall errors and difficulty establishing causation. Still, dismissing subjective harm because it is hard to measure creates a systematic blind spot. The useful approach is to combine reports, surveys, behavioral evidence, and careful testing.
VIP exceptions turn neutral rules into power protection.
High status changed enforcement. Facebook's XCheck system sent some prominent accounts for extra review before ordinary penalties took effect. Such review can prevent costly mistakes involving public figures, but the system expanded to millions of accounts, lacked consistent oversight, and sometimes delayed action on serious violations. Internal reviews found that abusive accounts and persistent rule-breakers had received protection.
The consequences were concrete. Brazilian soccer star Neymar posted intimate images of a woman who had accused him of rape. Because his account received special handling, the images remained up for more than a day and reached 56 million users. Facebook's own procedures called for removal, but the company left his accounts active. The system showed how fear of backlash from powerful users could make supposedly universal rules unequal in practice.
Extra review for high-impact cases can be defensible: mistakes involving elected leaders or human-rights activists may carry unusually serious consequences. The danger is allowing review to become immunity. Procedural protections need transparent criteria, staffed queues, deadlines, and independent audits, or they quietly become privileges. This episode also illustrates a broader institutional asymmetry: organizations often fear visible mistakes against elites more than diffuse harms to ordinary users. Equal treatment requires more than publishing one rule for everyone.
Treat internal research as oversight, not a liability to bury.
Insiders documented the risks. Frances Haugen joined Facebook hoping to help address disinformation and democracy. She and other employees found internal evidence of product harms, proposed changes, and repeated instances where action stalled. Haugen ultimately provided journalists with thousands of documents, including research on teen mental health and internal debates about ranking and integrity. Her reporting helped make visible a gap between what the company knew and what it told the public.
The response narrowed scrutiny. After the documents became public, Facebook restricted access to internal research and issued guidance urging researchers to limit audiences, hedge conclusions, and avoid linking platform activity to offline harm. The company said it still valued research. But the new controls made it harder for employees to test, share, and challenge assumptions that shaped products used by billions.
Internal research is an accountability mechanism only when findings can reach decision-makers and survive disagreement. Psychological safety, clear document retention, and independent review help organizations learn from bad news instead of treating it as disloyalty. At the same time, leaks can expose employees and vulnerable communities, so transparency needs privacy protections and careful redaction. The goal is not maximal disclosure of every working note. It is a system in which inconvenient findings can be examined without depending on one person's extraordinary personal risk.
Safety work can improve engagement after its short-term costs fade.
The assumed tradeoff was incomplete. Facebook teams often treated integrity changes as losses to growth because early experiments showed fewer interactions or visits. A later internal study tracked the cumulative effects of integrity interventions against a comparison group. It found that usage initially dipped, but the difference faded after about six months. After roughly a year, the interventions were associated with a modest, statistically significant usage gain, especially among younger and less frequent users.
The finding challenged company orthodoxy. Other internal analysis estimated that integrity work increased daily use among new users by 0.9 percent. These findings do not prove every safety measure will pay for itself, but they undermine the assumption that stronger integrity necessarily damages the business. Facebook delayed wider circulation of the research because its implications were uncomfortable.
The time horizon is central. A short experiment can capture the disruption of a product change while missing whether people return, trust the service more, or invite others. This is a familiar problem in public health and infrastructure: prevention costs appear immediately, while avoided harms and durable benefits emerge slowly. The evidence should not be oversold as proof that every moderation intervention boosts growth. It does show that companies need long-term tests and user-welfare measures before declaring safety and engagement opposing goals.
Analysis
Broken Code is investigative reporting structured as a slow revelation. It begins with Arturo Bejar, an engineer who once believed Facebook could improve social interaction, discovering that the feedback system he designed had been turned into a barrier. The narrative widens through researchers, product managers, lobbyists, and whistleblower Frances Haugen, then returns to the consequences of design choices during elections, political violence, and teen use. That arc makes the book's most important claim concrete: the platform's harms were not simply a collection of bad posts. They emerged from recommendation systems, product incentives, understaffed safety work, and decisions about whose complaints deserved attention.
The book is strongest when it joins specific mechanisms to organizational incentives. A metric that rewards comments, a recommendation system that treats repeated sharing as interest, and a business culture that prizes daily users can interact to amplify material no executive explicitly set out to promote. This is a more useful account than either technological determinism or the claim that a platform is neutral. Design does not dictate every outcome, but it changes which behaviors become efficient and visible.
The evidence also has limits worth keeping in view. Internal documents reveal what employees measured and feared, but some studies are correlational, and platform-scale causal effects are difficult to isolate. The company disputes aspects of the reporting, and the whistleblower's archive is a selected record rather than a complete institutional history. Those cautions do not erase the repeated pattern: documented risks, proposed interventions, and inconsistent follow-through. They do counsel against claiming that Facebook alone caused elections, polarization, or violence.
The larger contribution is a governance question. When a private platform becomes a major channel for news, social contact, and political organizing, its ranking choices act like public infrastructure while remaining accountable to corporate leadership. The most durable remedy is not a perfect classifier. It is governance that gives independent scrutiny access to evidence, gives safety teams authority to change mechanics, and evaluates success over time rather than by immediate engagement alone.
Review Summary
Broken Code by Jeff Horwitz offers a detailed look into Facebook's inner workings, revealing how the company prioritized growth and engagement over user safety and content integrity. Readers found the book informative, eye-opening, and alarming, praising Horwitz's thorough reporting and clear explanations of complex issues. Many were disturbed by Facebook's repeated decisions to ignore harmful consequences of its platform. While some found the writing dry at times, most readers considered it an essential read for understanding social media's impact on society and politics.
People Also Read
Glossary
Meaningful Social Interactions
Weighted social engagementA Facebook ranking objective introduced to favor interactions thought to bring people closer together. The system assigned greater weight to actions such as comments and reshares than to likes, but it did not distinguish supportive exchanges from hostile ones. The metric therefore treated measurable interaction as a proxy for social value, even when the resulting content encouraged conflict.
Sparing Sharing
Limits hyperactive users' influenceA Facebook News Feed intervention designed to reduce the disproportionate effect of highly active accounts on content recommendations. Its premise was that a small group of users generated an outsized share of engagement, including engagement with misinformation and extreme material. The intervention dampened distribution of content favored by these users, but leadership approved a substantially weakened version.
Break the Glass
Emergency platform safeguardsA collection of preplanned interventions Facebook could activate during heightened risk, such as elections or threats of violence. Measures included stricter ranking penalties, limits on reshares or group growth, and disabling features with known integrity problems. These actions could change platform mechanics quickly, often relying on reduced virality rather than waiting for automated systems to identify every harmful post.
XCheck
Extra review for prominent accountsFacebook's cross-check system routed some content from prominent people and pages through additional review before enforcement. The stated purpose was to avoid mistakes with high-impact accounts. In practice, the system grew inconsistently, reached millions of accounts, and sometimes delayed or prevented consequences for users who had violated the platform's rules.
CORGI
Detects coordinated account behaviorAn internal Facebook tool for identifying accounts that appeared to act in coordination or under central control. It looked for signals such as unusually similar activity, synchronized account creation, shared technical traces, and repeated wording. The system helped detect coordinated inauthentic behavior and was considered for use against both foreign influence operations and domestic networks.
Information Corridor
Pathways carrying harmful narrativesA model for tracing how messages move through connected users, accounts, and Facebook features. Analysis of Stop the Steal supporters grouped participants by roles such as originators, amplifiers, and people connecting different communities. The approach focused on disrupting the routes that spread a movement's messages, not only removing individual posts after they had circulated.
Bad Emotional Experience Feedback
Survey of harmful user experiencesA survey method developed by Arturo Bejar to capture unpleasant experiences users had on Instagram, including harassment and exposure to distressing material. It aimed to measure what ordinary reporting systems missed because many upsetting experiences did not clearly violate formal rules. The results could help the company understand users' lived experience rather than relying only on removals and engagement metrics.
FB Learner
Template for building machine learningAn internal Facebook system that packaged machine-learning methods into templates engineers could use without being specialists in the field. It helped teams run experiments and build AI-powered features more quickly. The tool accelerated the spread of machine learning across products, including content recommendations and moderation, while making it easier for teams to deploy models whose behavior they might not fully understand.
FAQ
What's Broken Code: Inside Facebook and the Fight to Expose Its Harmful Secrets about?
- Focus on Facebook's Impact: The book explores Facebook's influence on political discourse, mental health, and social interactions, revealing the consequences of its algorithms on society.
- Exposing Harmful Practices: It highlights Facebook's struggles with misinformation, hate speech, and system manipulation, emphasizing the tension between growth and integrity.
- Personal Stories and Whistleblower Insights: Jeff Horwitz shares accounts from former employees and whistleblower Frances Haugen, providing a human perspective on Facebook's internal culture and ethical dilemmas.
Why should I read Broken Code by Jeff Horwitz?
- Understanding Social Media's Role: The book is crucial for grasping social media's profound effects on democracy and public discourse, particularly through platforms like Facebook.
- Insight into Corporate Ethics: It offers a deep dive into the ethical challenges tech companies face, encouraging reflection on the responsibilities of social media platforms.
- Engaging Narrative: Horwitz combines investigative journalism with storytelling, making complex issues accessible and engaging through personal employee stories.
What are the key takeaways of Broken Code?
- Growth vs. Integrity: The book highlights the conflict between Facebook's growth objectives and ethical integrity, showing how engagement decisions often compromise user safety.
- Algorithmic Influence: It discusses how Facebook's algorithms prioritize sensational content, leading to polarization and misinformation, and calls for responsible content moderation.
- Call for Accountability: The narrative stresses the need for tech companies to be accountable for their societal impact, advocating for transparency and ethical technology development.
How does Broken Code address misinformation on Facebook?
- Misinformation as a Central Theme: The book details how misinformation spread on Facebook, especially during elections, and the platform's failure to manage it effectively.
- Algorithmic Challenges: Horwitz explains that Facebook's algorithms favored sensational content, contributing to misinformation's virality and highlighting the need for content prioritization reevaluation.
- Efforts to Combat Misinformation: It critiques Facebook's initiatives to combat misinformation as often too slow and insufficient, emphasizing the need for more robust measures.
What role do former employees and Frances Haugen play in Broken Code?
- Personal Accounts and Whistleblower Insights: Former employees and whistleblower Frances Haugen provide firsthand insights into Facebook's culture and decision-making processes.
- Critique of Corporate Practices: They offer critical perspectives on Facebook's practices, particularly regarding user safety and content moderation, highlighting internal conflicts.
- Humanizing the Narrative: Their stories make the issues more relatable, emphasizing the real-world consequences of corporate decisions and the need for transparency.
How does Broken Code explore the concept of "virality"?
- Virality as a Double-Edged Sword: The book discusses how Facebook's design encourages virality, often at the expense of content quality, leading to harmful or misleading information amplification.
- Impact on User Behavior: It explains that the pursuit of virality influences user interactions, prioritizing sensationalism over accuracy, affecting public discourse and societal norms.
- Need for Responsible Design: The narrative calls for reevaluating how virality is incentivized, advocating for a balance between engagement and promoting accurate content.
What are the ethical dilemmas highlighted in Broken Code?
- Balancing Growth and Responsibility: The book illustrates the tension between Facebook's growth objectives and ethical responsibilities, questioning the consequences of prioritizing profit over safety.
- Manipulation of Algorithms: It discusses how algorithms can be exploited by malicious actors, spreading misinformation and hate speech, posing significant ethical challenges.
- Corporate Accountability: The narrative emphasizes the need for tech companies to be accountable for their societal impact, advocating for transparency and ethical technology development.
How does Broken Code depict the internal culture at Facebook?
- "Assume Good Intent" Philosophy: The book describes a culture of trust among employees, critiqued as overly simplistic in the face of complex challenges.
- Pressure for Growth: It highlights the intense pressure on employees to prioritize growth metrics, often at the expense of ethical considerations.
- Resistance to Change: The narrative reveals reluctance within the company to address systemic issues, hindering progress toward ethical practices.
What solutions does Broken Code propose for Facebook's challenges?
- Reevaluation of Algorithms: The book advocates for prioritizing quality over engagement in algorithms to mitigate misinformation and promote healthier discourse.
- Increased Transparency: Horwitz calls for greater transparency in Facebook's operations, particularly in content moderation and algorithmic decision-making.
- Accountability Measures: It emphasizes the importance of holding tech companies accountable for their societal impact, encouraging responsible practices.
How does Broken Code address the mental health implications of social media?
- Teen Mental Health Focus: The book highlights Instagram's negative effects on teenage girls, emphasizing the need for platforms to address these concerns.
- Internal Research Findings: It reveals Facebook's studies indicating that social media design exacerbates feelings of inadequacy among users.
- Call for Responsible Design: The narrative advocates for redesigning social media platforms to prioritize user well-being and mitigate harmful effects.
What is the significance of the XCheck program mentioned in Broken Code?
- Preferential Treatment for VIPs: The XCheck program allowed high-profile users to evade standard content moderation rules, creating a two-tier system.
- Internal Acknowledgment of Flaws: The book reveals internal reviews finding XCheck as a betrayal of Facebook's commitment to fairness.
- Impact on Public Trust: XCheck contributes to skepticism about Facebook's commitment to user safety and equality, raising questions about accountability.
How does Broken Code suggest social media platforms can improve user well-being?
- Redesign for Safety: The book advocates for redesigning platforms to prioritize user well-being over engagement metrics, rethinking harmful content algorithms.
- Transparency and Accountability: It emphasizes the need for transparency in content management and user interactions, holding companies accountable for their impact.
- User-Centric Approaches: The narrative suggests adopting approaches that consider mental health implications, actively working to mitigate negative experiences.
Download PDF
Download EPUB
.epub digital book format is ideal for reading ebooks on phones, tablets, and e-readers.