How Algorithmic Bias Emerges and Distorts AI Decision-Making

A Healthcare Algorithm Used by Millions of Americans Systematically Underestimated Black Patients' Medical Needs

In 2019, Obermeyer and colleagues published a study in Science that analyzed a commercial healthcare algorithm used to identify patients for high-cost care management programs — an intervention designed to help the sickest patients. The algorithm assigned risk scores to 200 million Americans annually. The researchers found that Black patients, at the same algorithmic risk score as white patients, were demonstrably sicker by objective measures like chronic illness burden. The algorithm used healthcare spending as a proxy for health need. Because structural barriers meant Black patients historically had less healthcare access and lower spending — not because they were healthier — the proxy variable encoded historical discrimination into future care allocation. The study estimated the bias reduced the fraction of Black patients identified for the program by approximately 50%. This was not a fringe application: it was a system embedded in standard care delivery across thousands of hospitals.

Sources of Bias: Where It Enters the Pipeline

Algorithmic bias is not a single problem with a single fix. It can enter at multiple points in a machine learning pipeline, and each source has different characteristics and mitigation approaches.

1. Training Data Bias

Machine learning systems learn patterns from historical data. When that data reflects historical discrimination, unequal access, or systematic measurement differences across groups, the model learns those patterns. A hiring algorithm trained on historical hiring decisions will replicate the demographic preferences of the hiring managers who made those decisions. A recidivism prediction model trained on arrest records will reflect the patterns of differential policing — where certain communities have historically received disproportionate law enforcement attention — rather than actual offending rates.

2. Representation Gaps

When certain groups are underrepresented in training data, the model's performance is worse for those groups. NIST's comprehensive evaluation of facial recognition algorithms found that multiple systems had error rates 10–100 times higher for Black women compared to white men — partly because training datasets contained fewer images from these demographic groups, limiting the model's exposure to the range of variation in these faces.

3. Proxy Variables

Even when a model does not explicitly use a protected attribute like race, it may use correlated proxy variables that reproduce discriminatory patterns. ZIP code correlates with race due to residential segregation history. Credit history correlates with socioeconomic status, which correlates with race. Healthcare spending correlates with access, which correlates with race and income. Models using these proxies can discriminate indirectly — a pattern that is often not immediately visible in the model's feature list.

4. Feedback Loops

When model outputs influence future data collection, bias can compound over time. Predictive policing algorithms direct more officers to neighborhoods identified as high-risk, generating more arrests there, producing more data that confirms the area's "high risk" label — independent of whether crime actually changed. The algorithm's predictions become self-fulfilling prophecies that continuously reinforce the initial bias.

Documented Cases of Consequential Algorithmic Bias

Domain	System	Documented Harm	Source
Criminal justice	COMPAS recidivism tool	Black defendants twice as likely to be incorrectly labeled higher-risk; white defendants more likely to be incorrectly labeled lower-risk	ProPublica, 2016
Healthcare	Commercial care management algorithm	Same risk score predicted sicker condition for Black patients; 50% underidentification for care management	Obermeyer et al., Science, 2019
Hiring	Amazon resume screening tool	Penalized resumes containing the word "women's" (e.g., women's chess club); system scrapped internally	Reuters, 2018
Facial recognition	Multiple commercial systems	Contributed to wrongful arrests of Robert Williams, Nijeer Parks, Michael Oliver (all Black men)	ACLU, MIT Media Lab, NIST
Financial services	Apple Card credit algorithm	Women assigned substantially lower credit limits than men with comparable financial profiles; NY DFS investigation	NY DFS, 2019
Benefits administration	Arkansas Medicaid allocation algorithm	Home care hours dramatically cut for thousands of patients with cerebral palsy; algorithm errors found on audit	Arkansas district court, 2016

The Competing Definitions of Fairness Problem

A mathematically precise finding from 2016 (Chouldechova; Kleinberg et al.) proved that several intuitive definitions of algorithmic fairness are mutually incompatible when base rates differ between groups. Specifically, you cannot simultaneously satisfy:

Calibration: People assigned the same risk score have the same actual risk rate across groups.
Equal false positive rates: The rate of incorrectly labeling low-risk people as high-risk is equal across groups.
Equal false negative rates: The rate of incorrectly labeling high-risk people as low-risk is equal across groups.

This is not a failure of current technology but a mathematical impossibility. Choosing one definition of fairness requires accepting violations of others. The COMPAS controversy turned partly on this: Northpointe (COMPAS's developer) could demonstrate calibration; ProPublica documented unequal false positive rates. Both were correct. Which fairness criterion a system should optimize is ultimately a value judgment, not a technical one.

Mitigation Approaches

Approach	Method	Limitation
Pre-processing	Reweight or resample training data to correct representation gaps	May not address proxy variable bias; can reduce overall accuracy
In-processing	Add fairness constraints to the training objective function	Technical complexity; still requires choosing a fairness definition
Post-processing	Adjust decision thresholds differently across groups to equalize error rates	Requires knowing group membership; may violate anti-discrimination law in some jurisdictions
Algorithmic auditing	Independent third-party testing for disparate impact before and after deployment	Access to systems and data often restricted; no legal mandate in most jurisdictions
Dataset diversification	Systematic collection of underrepresented group data; diverse annotation teams	Expensive; doesn't address the problem that historical data is already biased

The Accountability Gap

Perhaps the most consequential aspect of algorithmic bias is not its technical origin but its legal and organizational opacity. The Equal Credit Opportunity Act requires creditors to provide specific reasons for adverse decisions. A black-box neural network may not produce reasons that satisfy this requirement. The EU's GDPR creates a limited "right to explanation" for automated decisions, though the legal scope and technical implementation remain contested. The EU AI Act (2024) requires high-risk AI systems — including those used in employment, credit, education, and biometric identification — to undergo conformity assessments and maintain documentation that can be audited. In the United States, comprehensive federal AI governance legislation remains absent as of 2024, leaving enforcement to existing civil rights statutes and agency guidance.