Selection bias is the distortion that occurs when the cases you study are not representative of the group you want to draw conclusions about, because of how they were chosen or chose themselves. Economist James Heckman shared the 2000 Nobel prize in economics for methods to analyse selective samples. Survivorship bias is a famous form, linked to Abraham Wald‘s Second World War work estimating aircraft vulnerability from the damage on planes that came back.
Be explicit about who or what the conclusion is meant to cover.
02
Trace the filter
Ask how cases got into the data: who survived, who answered, who was published, who could be measured.
03
Compare sample and population
Check whether the filter is related to the outcome you care about; if it is, the result is biased.
04
Correct or redesign
Reweight, model the selection, or better, change how the sample is drawn.
Where you see it today
AI leaderboards
A study of Chatbot Arena found that some providers privately tested many model variants and could publish only the best score, which biases rankings through selective disclosure (Singh et al., 2025).
Training data that misses people
Two face-analysis benchmarks were found to be mostly lighter-skinned, and commercial gender classifiers performed worst on darker-skinned women (Buolamwini and Gebru, 2018). A model learns the sample it is given, not the world.
What gets published
In a set of social science experiments, null results were far less likely to be published than strong results (Franco, Malhotra and Simonovits, 2014). The same filter plausibly shapes AI case studies: successes get written up, failed pilots rarely do.
Judging AI by its showcases
Demos, launch videos and customer stories are chosen to impress. Judging what AI does on average from those cases overstates it; the useful comparison is across all attempts in a comparable setting.
Common forms of selection bias
Survivorship bias
The filterOnly survivors are observed
ExampleStudying returning planes, or only successful startups
Self-selection
The filterPeople choose whether to take part
ExampleOnline polls answered by the most motivated
Sampling bias
The filterThe method misses part of the population
ExampleA training set that under-represents some groups
Publication bias
The filterOnly certain results are published
ExamplePositive trials in journals, null ones in drawers
Selective disclosure
The filterThe reporter picks which results to show
ExamplePublishing the best of many private benchmark runs
How to spot and avoid it
Before trusting a finding, ask who is missing from the data and why.
Count the failures as well as the successes: pilots dropped, products shut, models not released.
Compare the sample's make-up with the population you want to describe.
Pre-register what you will measure, and report everything you ran.
Test AI tools on your own representative cases, not on the vendor's showcase.
The Business Engineer lens
Much of what is said about AI at work comes from a selected sample of wins. The Business Engineer’s FRED test is a reality check against that bias: judge AI on the full set of tasks and outcomes in your business, not on the stories that made it into a keynote.
It is the distortion that occurs when the sample studied is not representative of the population you want to draw conclusions about, because of how cases were chosen or chose themselves.
What is the difference between selection bias and survivorship bias?
Survivorship bias is one form of selection bias, where only the cases that survived a process are observed.
How do you reduce selection bias?
Sample randomly where possible, compare the sample with the population, count failures as well as successes and model the selection when it cannot be avoided.
How does selection bias affect AI?
Models trained on unrepresentative data perform worse for under-represented groups, and AI benchmarks or case studies can mislead when only the best results are reported.
The complete explainer, cleaned up and reformatted for mobile.
Selection Bias refers to the distortion of research or decision-making outcomes due to non-random sample selection or participant self-selection. Addressing bias improves accuracy and decision-making reliability, but challenges arise in data collection, interpretation, and publication. Examples include biased clinical trials, political polls, and user surveys on biased platforms.
Understanding Selection Bias
What is Selection Bias?
Selection bias is a systematic error that occurs in research studies when the individuals or items included in the sample are not representative of the population being studied. This bias can distort study results, leading to inaccurate conclusions and potentially influencing decision-making processes.
Key Elements of Selection Bias
Non-Random Selection: Selection bias arises when the process of selecting study participants or data points is not random but influenced by certain factors or conditions.
Sampling Error: The bias can result from an improper sampling technique, leading to the underrepresentation or overrepresentation of specific subgroups within the population.
Distorted Associations: Selection bias can lead to incorrect associations between variables, making it challenging to establish cause-and-effect relationships.
Why Selection Bias Matters
Understanding selection bias is crucial for researchers, statisticians, policymakers, and anyone involved in data analysis and decision-making. Recognizing the implications of selection bias informs strategies for improving study design, data collection, and research validity.
The Impact of Selection Bias
Flawed Research: Selection bias can undermine the validity of research studies, making their findings less reliable or even entirely invalid.
Misleading Conclusions: It can lead to erroneous conclusions that may influence policy decisions, medical treatments, or business strategies.
Benefits of Understanding Selection Bias
Improved Study Design: Knowledge of selection bias allows researchers to design studies that minimize its effects, resulting in more robust and accurate findings.
Informed Decision-Making: Policymakers and decision-makers can make more informed choices by considering the potential impact of selection bias in research studies.
Challenges of Understanding Selection Bias
Detection and Mitigation: Identifying and mitigating selection bias can be complex, as it requires a thorough understanding of the study context and data collection processes.
Complexity of Bias Types: There are various types of selection bias, each with its unique challenges and implications.
Challenges in Understanding Selection Bias
Understanding the limitations and challenges associated with selection bias is essential for individuals seeking to conduct rigorous research and make informed decisions.
Detection and Mitigation
Transparent Reporting: Researchers should transparently report their study methods, including how participants were recruited and data collected, to facilitate the detection of potential selection bias.
Statistical Techniques: Statisticians employ various statistical techniques to identify and control for selection bias when analyzing data.
Complexity of Bias Types
Types of Selection Bias: Familiarity with different types of selection bias, such as sampling bias, volunteer bias, and healthy user bias, is essential for recognizing their specific characteristics.
Study Context: The context of each study may introduce unique challenges related to selection bias, necessitating tailored approaches for detection and mitigation.
Selection Bias in Action
To understand selection bias better, let’s explore how it operates in real-life scenarios and what it reveals about its impact on research findings and decision-making.
Scenario: A clinical trial for a new medication recruits participants from a specific demographic group, mainly young and healthy individuals.
Selection Bias in Action:
Sampling Bias: The study’s design inadvertently excludes older individuals or those with underlying health conditions.
Flawed Conclusions: The trial may conclude that the medication is highly effective, but this conclusion may not apply to the broader population, including older adults and those with health issues.
Implications: The medication may be approved for widespread use based on potentially biased results, potentially harming those for whom it is less effective or safe.
Political Polling
Scenario: A political polling organization conducts surveys only via telephone calls during daytime hours.
Selection Bias in Action:
Volunteer Bias: The survey results predominantly represent individuals who are available to answer phone calls during the day.
Misleading Predictions: The polling data may suggest a clear lead for a particular candidate, but this may not accurately reflect the views of the entire population, including those who work during the day or prefer not to answer unsolicited calls.
Impact: Political decisions and campaign strategies may be based on skewed polling data, potentially leading to inaccurate predictions and outcomes.
Educational Research
Scenario: A study on the effectiveness of online education recruits participants through online advertisements.
Selection Bias in Action:
Self-Selection Bias: Individuals who respond to the online advertisements and participate in the study may have a higher predisposition for online learning.
Exaggerated Results: The study may overstate the benefits of online education, as it primarily includes participants who are already inclined towards this mode of learning.
Policy Implications: Educational policies based on such research may allocate resources and prioritize online learning without considering the needs and preferences of a broader student population.
Selection bias is prevalent in various real-life situations:
1. Clinical Trials
In medical research, selection bias can occur when patients are not randomly assigned to treatment and control groups. Instead, certain patients may be chosen based on specific characteristics, potentially affecting the study’s outcomes.
2. Political Polls
Selection bias can impact political polls, especially when survey samples are not representative of the entire population. For example, if a poll relies solely on landline telephone surveys, it may underrepresent younger individuals who primarily use mobile phones.
3. User Surveys
Online surveys conducted on specific platforms may suffer from selection bias. For instance, a survey posted on a gaming website may primarily attract responses from gamers, excluding individuals who do not visit the site.
Selection Bias: Key Highlights
Definition and Significance: Selection Bias is the distortion of research or decision outcomes due to non-random sample selection or participant self-selection. Addressing this bias is essential for improving accuracy and reliability in decision-making processes.
Characteristics:
Sample Selection: Non-random sample selection leads to results that do not represent the entire population accurately.
Volunteer Bias: Self-selection by participants results in skewed outcomes.
Survivorship Bias: Focusing only on surviving subjects, overlooking relevant data.
Publication Bias: Preference for publishing positive or significant results, affecting the overall picture.
Applicability:
Medical Studies: Non-random patient selection in clinical trials affects research outcomes.
Survey Sampling: Surveys on biased platforms attract specific demographic groups, influencing results.
Recruitment Processes: Biased hiring practices lead to a lack of diversity in the workforce.
Benefits:
Efficiency: Addressing bias streamlines research efforts, saving time and resources.
Improved Accuracy: Mitigating bias leads to more trustworthy and reliable decision outcomes.
Challenges:
Data Collection: Gathering unbiased data from diverse sources poses a challenge.
Data Interpretation: Analyzing results while considering potential biases is essential.
Publication Practices: Addressing bias in publication decisions is crucial in academic research.
Examples:
Clinical Trials: Non-randomized patient selection can skew outcomes of medical trials.
Political Polls: Biased sampling methods impact the accuracy of election predictions.
User Surveys: Surveys conducted on biased platforms can lead to distorted survey responses.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
