Simpson’s paradox occurs when an association between two variables appears, disappears or reverses once data are split into subgroups, usually because a hidden variable is spread unevenly across them. Karl Pearson (1899) and George Yule (1903) noted the effect first; it is named after Edward Simpson, whose 1951 paper led to the label. The best-known example is the 1973 graduate admissions data from the University of California, Berkeley.
Something other than the factor you study, such as which department people applied to, also drives the outcome.
02
Uneven groups
That variable is distributed differently across the groups you compare.
03
Pooling flips the picture
Adding the subgroups together mixes those different bases and can reverse the trend.
04
The causal question decides
Whether to trust the split or the total depends on how the variables cause each other, not on the numbers alone.
Where you see it today
The Berkeley case
Aggregate data for fall 1973 showed a clear but misleading pattern against female applicants. Department by department, there was little bias, and properly pooled data showed a small bias in favour of women (Bickel, Hammel and O’Connell, 1975).
AI evaluation by segment
Averages hide subgroup gaps. An audit of commercial gender-classification systems found the highest error rates for darker-skinned women, a gap that overall accuracy figures did not show (Buolamwini and Gebru, 2018). It is a close cousin of the paradox: the aggregate tells a different story from the parts.
Model and A/B comparisons
A new model can beat the old one on an overall benchmark while doing worse in each task category, if the mix of tasks differs between test sets. The same applies to A/B tests when traffic mix shifts between variants.
Which table to trust
Philosophers and statisticians still debate the explanation. One influential view, from Judea Pearl, says the paradox comes from confusing causal and probabilistic reasoning, so the right answer depends on the causal model (Stanford Encyclopedia of Philosophy, 2021).
Simpson's paradox vs related traps
Simpson's paradox
What goes wrongA trend reverses when groups are combined
ExampleAdmissions look biased overall but not by department
Berkson's paradox
What goes wrongSelecting on an outcome creates a false correlation
ExampleAmong hired candidates, skills and connections look negatively related
Confounding
What goes wrongA third variable drives both cause and effect
ExampleIce-cream sales and drownings both rise in summer
Ecological fallacy
What goes wrongGroup-level patterns applied to individuals
ExampleAssuming every user in a high-spend region spends a lot
Base-rate fallacy
What goes wrongIgnoring how common something is overall
ExampleOver-trusting a positive test for a rare condition
How to spot and avoid it
Break every headline metric down by the main segments: customer type, channel, region, task.
Check whether group sizes differ between the things you compare.
Ask what else could drive the outcome and whether it differs across groups.
Draw a simple causal diagram before choosing whether to report the split or the total.
For AI models, publish results per task, language and user group, not only the average.
The Business Engineer lens
AI products are judged by aggregate scores: benchmark averages, overall win rates, blended conversion. The Business Engineer’s rule is to read every headline metric by segment before acting, because the mix of users and tasks can shift faster than the models do.
A trend that appears in subgroups of data disappears or reverses when the groups are combined, usually because of a hidden variable distributed unevenly across them.
What is the most famous example?
The 1973 UC Berkeley graduate admissions data, where an apparent bias against women overall did not hold department by department.
Who discovered Simpson's paradox?
Karl Pearson and George Yule noted the effect around 1900; it is named after Edward Simpson, who wrote about it in 1951.
How do you avoid it?
Break results down by relevant segments and think about what causes what before deciding whether the combined or the split figures answer your question.
The complete explainer, cleaned up and reformatted for mobile.
In statistics, the Simpson Paradox happens when a trend clearly shows up in clusters/brackets of data. But it disappears or, at worse it reverses when the data is grouped and combined. In short, the Simpson paradox shows that when the data moves from clusters to combined data, it hides several distributions, which end up creating a biased overall effect.
Definition
ExplanationSimpson’s Paradox, named after British statistician Edward H. Simpson, is a statistical paradox where a trend or association observed within subgroups of data can reverse or disappear when the subgroups are combined. In other words, what seems true for individual parts of the data may not hold when the data is analyzed as a whole.
Key Concepts
Explanation1. Subgroup vs. Aggregate: The paradox revolves around the distinction between examining data within subgroups (i.e., disaggregated data) and analyzing the data as a whole (i.e., aggregated data).
Explanation2. Causality vs. Association: Simpson’s Paradox highlights the difference between a causal relationship and a statistical association. An apparent association between variables may not imply causality when considering the entire dataset.
Causes
Explanation1. Heterogeneous Subgroups: Simpson’s Paradox often occurs when subgroups within the dataset have significantly different characteristics or sample sizes. These differences can lead to skewed results when aggregated.
Explanation2. Hidden Variables: Sometimes, there are unobserved or unaccounted-for variables that influence both the grouping and the outcome, resulting in the paradoxical reversal of trends.
Explanation3. Weighted Averages: Aggregating data with unequal sample sizes can give disproportionate weight to certain subgroups, affecting the overall trend.
Examples
Explanation1. Medical Studies: Simpson’s Paradox is commonly encountered in medical research. A treatment that appears to be less effective in a subgroup may be more effective when considering the entire patient population.
Explanation2. Educational Outcomes: Test scores within different schools or districts may suggest that one school performs better, but when considering all schools together, a different conclusion may emerge.
Explanation3. Sports Statistics: A baseball player may have a higher batting average in different seasons or against different teams, but the overall average for all seasons may be lower.
Consequences
Explanation1. Misleading Interpretations: Failing to recognize Simpson’s Paradox can lead to incorrect conclusions and potentially poor decision-making based on aggregated data.
Explanation2. Inaccurate Policies: In areas like healthcare or education, misinterpreting data can result in the implementation of policies that are ineffective or even detrimental.
Explanation3. Loss of Insights: If analysts focus solely on aggregated data, they may overlook valuable insights that exist within subgroups.
Mitigation Strategies
Explanation1. Data Disaggregation: Consider analyzing and reporting data at both the subgroup and aggregate levels to gain a comprehensive understanding.
Explanation2. Identifying Confounding Variables: Carefully examine potential confounding variables that might influence the relationship between the variables under study.
Explanation3. Transparent Reporting: When presenting data, clearly communicate the presence of Simpson’s Paradox, especially if it could impact decision-making.
Explanation4. Expert Consultation: Seek input from statistical experts or data analysts to ensure the validity of your interpretations, especially when working with complex datasets.
Conclusion
ExplanationSimpson’s Paradox serves as a reminder of the nuances and potential pitfalls in statistical analysis. It underscores the importance of considering data from multiple angles and being cautious when drawing conclusions based on aggregated information. By understanding and addressing the paradox, analysts and decision-makers can make more informed choices and avoid misinterpretations.
The Simpson paradox origin story
As Tom Grigg explained exceptionally well, the Simpson paradox took its name from Edward Hugh Simpson thanks to a technical paper in 1951.
Yet it was made famous when another statistician, Peter Bickel, was called – in 1971 – to analyze the admission data at UC Berkleyโs suspected gender bias.
As the story goes, the university feared a lawsuit, so they had the data analyzed by Bickel.
When the data were combined, it really gave the impression that more males had been selected over women.
In fact, of the total male applicants, 44% were selected, and of the total female applicants, 35% were selected.
Yet when the data were analyzed by the department, it showed something completely different.
The admissions were biased toward women in four of the six departments analyzed.
But, as women applied to departments where fewer applicants were selected when the data combined, it gave an impression of bias toward male applicants.
Understanding the Simpson paradox
A good example is Nassim Taleb’s video on the topic.
While this is related to vaccine data, it can be easily translated into business, as we’ll see.
As Taleb explained about the vaccine data.
When the data are grouped under the same umbrella, after having been analyzed in clusters and homogeneous groups, it suddenly gives an opposite effect.
It’s like the data not only doesn’t give the same result when analyzed in brackets, but it gives the reverse effect.
This is what happens when the Simpson paradox messes up the statistics data.
Why?
Intuitively, when data, before compared under brackets, get combined, it disperses, thus making that worthless for the initial scope.
In the case of the vaccine, because many people over 60s were vaccinated, and a few people under 20s were vaccinated, when the data gets combined, it’s skewed toward the mortality of people over 60s, thus creating a bias and.
Beware of the Lurking variable
To keep things short, hidden variables in the combined spurs the overall analysis, making it worthless.
This is known as a “lurking variable” or a variable that affects the data at the point of creating a “spurious association” (in short, the cause-effect relationship ceases).
The Simpson paradox in business
The Simpson paradox can hide in many of the business and marketing analyses, as when the data is combined, it’s easy to mistake a correlation for causation.
Take the case of, as explained by adexchanger.com, for instance, when deciding on a programmatic campaign, when looking at the data for gender only, it shows how the male budget has seemingly more conversions, thus skewing the data toward males.
Yet from an age analysis, you figure that females between 18-24 have higher conversion rates.
If you don’t understand this bias, it’s easy to overspend on an overrepresented audience, not because it’s more aligned with your audience but because you’re misreading the data.
And as you can imagine, this can have substantial consequences on your bottom line (money wasted on ineffective campaigns and lost revenues as you’re not targeting the right audience).
Quantitative vs. Qualitative Research
Dealing with data is extremely hard.
It’s one of the hardest things in business.
And as most businesses now have a lot of data available, it’s easy to fall into the trapping of misusing it.
For that, it’s critical to establish project business processes, whereas it gets clear to the internal team when to use quantitative vs. qualitative data or both.
The characteristics of quantitative research contribute to methods that use statistics to make generalizations about something. These generalizations are constructed from data used to find patterns and averages and causal test relationships.
Companies like Amazon have learned how to balance that with qualitative research.
Qualitative research is performed by businesses that acknowledge the human condition and want to learn more about it. Some of the key characteristics of qualitative research that enable practitioners to perform qualitative inquiry comprise small data, the absence of definitive truth, the importance of context, the researcher’s skills and are of interests.
Indeed, quantitative data is extremely helpful to improve business processes.
However, it’s critical to know when human judgment needs to kick in, when some qualitative data is available that completely flips things upside down.
Indeed, if Amazon was going to look into these endeavors with a quantitative mindset, it would have never undertaken them as they did not make sense from a quantitative standpoint.
Yet, the intuitive understanding of how those things that might seem negative from a first-order effect standpoint (losing profits in the short-term) might make complete sense from a second-order effect standpoint (becoming way more successful in the long run).
Second-order thinking is a means of assessing the implications of our decisions by considering future consequences. Second-order thinking is a mental model that considers all future possibilities. It encourages individuals to think outside of the box so that they can prepare for every and any eventuality. It also discourages the tendency for individuals to default to the most obvious choice.
Understanding the implications of second-order effects is something that qualitative understanding and human judgment together can do.
Whereas quantitative data can be extremely useful to improve, in the short-term, business processes to make them way more efficient, which also, in the long-term, if properly used can create a competitive moat for the business.
For instance, going back to Amazon’s example, the company processes like inventory management and order fulfillment are part of its core strategic advantage, and they are driven by quantitative data!
Case Studies
Healthcare:
Scenario: A hospital wants to determine which treatment is more effective for a certain illness. At first glance, Treatment A seems to have a higher recovery rate than Treatment B. However, when data is broken down by severity of illness, Treatment B is more effective for severe cases, while Treatment A is more effective for mild cases.
Simpson’s Paradox: The aggregate data suggests Treatment A is better, but a more detailed analysis shows that Treatment B is better for severe cases.
Sports:
Scenario: A baseball player, Player X, has a higher batting average than Player Y in both the first and second half of a season. However, when combining the two halves, Player Y has a higher overall batting average.
Simpson’s Paradox: Individual performance in each half of the season does not necessarily predict overall performance.
Economics:
Scenario: A country’s unemployment rate decreases both this year and the previous year. However, when looking at the two-year period as a whole, the unemployment rate has increased.
Simpson’s Paradox: Annual data may show positive trends, but longer-term trends might reveal a different story.
Education:
Scenario: Students from School A score higher on math tests than students from School B in both 9th and 10th grades. However, when combining scores from both grades, students from School B have a higher average.
Scenario: City A has seen a decline in house prices in both the east and west sectors. However, overall, the city’s house prices have increased.
Simpson’s Paradox: Individual sectors of the city might show a decline, but the overall city might see an increase due to factors in smaller unexamined areas.
Environment:
Scenario: Factory A reduces its carbon emissions in both 2020 and 2021. Factory B increases its emissions in both years. However, when the total emissions of both years are combined, Factory A has a larger increase in emissions than Factory B.
Simpson’s Paradox: Individual yearly reductions can be overshadowed by larger overall increases when data is combined.
Transportation:
Scenario: Car model X has fewer accidents than car model Y in both urban and rural settings. However, when combining the data, car model Y has fewer accidents in total.
The Simpson paradox is an effect that in statistics and probability can create biased analyses. In fact, when present the data combined from an analysis gives a reverse effect compared to the data analyzed in buckets.
The Simpson paradox can create biased analyses also in business and marketing creating overspending toward the wrong audience.
The Simpson paradox also makes it much harder to make decisions in business when doing statistical analysis.
Key highlights
Definition of the Simpson Paradox: The Simpson Paradox is an effect in statistics and probability where a trend appears in clusters of data but disappears or reverses when the data is combined, leading to biased overall effects.
Origin and Famous Case: The paradox is named after Edward Hugh Simpson and gained fame when statistician Peter Bickel analyzed UC Berkeley’s admission data, revealing biases in gender representation.
Occurrence in Business and Marketing: The Simpson Paradox can hide in business and marketing analyses, leading to mistaken correlations for causation and overspending on misinterpreted data.
Impact of Hidden Variables: Hidden variables, known as “lurking variables,” affect combined data, causing spurious associations and disrupting cause-effect relationships.
Importance of Proper Data Analysis: Proper data analysis and understanding when to use quantitative and qualitative research can mitigate the effects of the Simpson Paradox in business decision-making.
Balancing Quantitative and Qualitative Research: Companies like Amazon have demonstrated the importance of balancing quantitative data with qualitative understanding and human judgment for more effective decision-making.
Strategic Implications: Understanding the implications of second-order effects and combining qualitative understanding with quantitative data can create competitive advantages and long-term success in business.
Simpson’s Paradox
DescriptionSimpson’s Paradox is a statistical phenomenon where a trend appears in different groups of data but disappears or reverses when the groups are combined. Simpson’s Paradox occurs when there is a confounding variable that influences the relationship between the variables under study and the groups’ compositions, leading to misleading conclusions if not properly accounted for. Simpson’s Paradox highlights the importance of considering subgroup effects and interaction effects in data analysis to avoid drawing erroneous conclusions from aggregated data.
When to Apply– When analyzing data trends or interpreting statistical relationships in research or decision-making processes. – Particularly in understanding the underlying mechanisms and implications of Simpson’s Paradox, such as confounding variables, subgroup effects, and interaction effects, and in exploring techniques to detect and mitigate the impact of Simpson’s Paradox, such as stratified analysis, sensitivity analysis, and causal inference, to ensure accurate and reliable data interpretation and decision-making in data analysis or research studies.
Confounding Variable
DescriptionA Confounding Variable is an extraneous variable that correlates with both the independent variable and the dependent variable in a study, influencing the observed relationship between them. Confounding variables can lead to spurious correlations or misleading conclusions if not controlled or accounted for in the analysis. Identifying and controlling for confounding variables is essential to ensure the validity and reliability of research findings and statistical analyses.
When to Apply– When designing experiments or conducting observational studies to investigate causal relationships or associations between variables. – Particularly in understanding the role and impact of confounding variables, such as selection bias, lurking variables, and omitted variables, and in exploring techniques to control for confounding variables, such as randomization, matching, and multivariate analysis, to minimize bias and improve the internal validity of research studies or data analyses.
Causal Inference
DescriptionCausal Inference is the process of drawing conclusions about causal relationships between variables based on observational data or experimental evidence. Causal inference aims to determine whether changes in one variable cause changes in another variable, accounting for potential confounding variables and alternative explanations. Causal inference methods include experimental design, regression analysis, and structural equation modeling, among others, to establish causality or infer causal mechanisms from data.
When to Apply– When examining cause-and-effect relationships or evaluating intervention effects in research or policy analysis. – Particularly in understanding the principles and limitations of causal inference methods, such as counterfactual reasoning, causal diagrams, and instrumental variables, and in exploring techniques to strengthen causal inference, such as sensitivity analysis, causal mediation analysis, and propensity score matching, to enhance the validity and reliability of causal conclusions in causal inference or program evaluation studies.
Data Aggregation
DescriptionData Aggregation is the process of combining individual data points or observations into summary statistics or groups for analysis or reporting purposes. Data aggregation can involve averaging, summing, or categorizing data to derive meaningful insights or trends from large datasets. However, data aggregation can obscure underlying patterns or relationships, such as Simpson’s Paradox, if not properly disaggregated or analyzed at different levels of granularity. Understanding data aggregation techniques and their implications is crucial for accurate data interpretation and decision-making.
When to Apply– When summarizing data or reporting aggregated statistics to communicate trends or patterns in datasets. – Particularly in understanding the effects and limitations of data aggregation, such as information loss, granularity bias, and aggregation bias, and in exploring techniques to mitigate aggregation-related issues, such as disaggregation analysis, subgroup analysis, and trend analysis, to ensure accurate and reliable data interpretation and decision-making in data analysis or reporting processes.
Spurious Correlation
DescriptionA Spurious Correlation is a statistically significant relationship between two variables that is coincidental or due to chance, rather than representing a true causal relationship or meaningful association. Spurious correlations can arise from confounding variables, sampling variability, or data artifacts, leading to misleading interpretations or false conclusions if not properly investigated or controlled for in the analysis. Detecting and addressing spurious correlations is essential for accurate data interpretation and hypothesis testing.
When to Apply– When identifying correlations or testing hypotheses in data analysis or research studies. – Particularly in understanding the causes and consequences of spurious correlations, such as data mining bias, data dredging, and ecological fallacy, and in exploring techniques to distinguish spurious correlations from meaningful relationships, such as cross-validation, hypothesis testing, and replication studies, to improve the validity and reliability of statistical analyses or research findings in data science or scientific research endeavors.
Interaction Effect
DescriptionAn Interaction Effect occurs when the relationship between two variables is modified by the presence of a third variable, indicating that the effect of one variable on the outcome depends on the level or presence of another variable. Interaction effects can complicate data analysis and interpretation, as they may alter the direction or magnitude of the relationship between variables across different subgroups or conditions. Understanding interaction effects is essential for identifying nuanced relationships and making accurate predictions or inferences in statistical modeling.
When to Apply– When exploring complex relationships or conducting multivariate analysis in statistical modeling or experimental design. – Particularly in understanding the nature and implications of interaction effects, such as moderation, mediation, and conditional effects, and in exploring techniques to detect and interpret interaction effects, such as interaction terms, subgroup analysis, and structural equation modeling, to uncover nuanced relationships and improve the predictive accuracy of statistical models or research studies in data analysis or social science research fields.
Experimental Design
DescriptionExperimental Design is the process of planning and conducting experiments to test hypotheses or evaluate interventions by systematically manipulating independent variables and measuring their effects on dependent variables. Experimental design involves defining research objectives, selecting participants, and controlling experimental conditions to minimize bias and confounding variables and maximize the internal validity of the study. Well-designed experiments allow researchers to establish causal relationships and draw valid conclusions from the data.
When to Apply– When conducting controlled experiments or evaluating treatment effects in scientific research or program evaluation. – Particularly in understanding the principles and considerations of experimental design, such as randomization, blinding, and control groups, and in exploring techniques to optimize experimental designs, such as factorial designs, crossover designs, and quasi-experimental designs, to enhance the validity and reliability of experimental findings in experimental research or intervention studies.
Multivariate Analysis
DescriptionMultivariate Analysis is a statistical technique used to analyze datasets with multiple variables or observations simultaneously, exploring relationships, patterns, and trends across variables. Multivariate analysis encompasses various methods, such as regression analysis, factor analysis, and cluster analysis, to identify underlying structures or dimensions in complex datasets and make inferences or predictions based on the interrelationships between variables. Multivariate analysis allows researchers to uncover hidden patterns or associations that may not be apparent in univariate or bivariate analyses.
When to Apply– When examining relationships or identifying patterns across multiple variables in data analysis or research studies. – Particularly in understanding multivariate analysis techniques and applications, such as principal component analysis, discriminant analysis, and structural equation modeling, and in exploring techniques to interpret and visualize multivariate data, such as heatmaps, factor plots, and biplots, to gain insights and make informed decisions in statistical modeling or exploratory data analysis endeavors.
Statistical Fallacy
DescriptionA Statistical Fallacy is a misconception or error in reasoning that arises from misinterpreting statistical data or drawing invalid conclusions from statistical analyses. Statistical fallacies can result from sampling biases, data artifacts, or logical errors in statistical reasoning, leading to incorrect interpretations or false beliefs about the data or phenomena under study. Detecting and correcting statistical fallacies is essential for ensuring the integrity and reliability of statistical analyses and research findings.
When to Apply– When evaluating statistical claims or interpreting research findings in scientific literature or public discourse. – Particularly in understanding common statistical fallacies and their implications, such as correlation-causation fallacy, base rate fallacy, and survivorship bias, and in exploring techniques to avoid or mitigate statistical fallacies, such as critical thinking, skepticism, and peer review, to promote sound statistical reasoning and evidence-based decision-making in statistical literacy or research communication efforts.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
Scroll to Top
Discover more from FourWeekMBA
Subscribe now to keep reading and get access to the full archive.
