COMPAS Recidivism Algorithm Showed Racial Bias in US Courts
Northpointe (Equivant)
What happened
ProPublica's investigation found that Black defendants were nearly twice as likely to be falsely flagged as future criminals compared to white defendants. The tool was being used by judges to inform bail, sentencing, and parole decisions.
Root cause
Algorithmic model trained on historically biased criminal justice data; no independent bias audit prior to deployment in high-stakes judicial contexts.
Architectural failure
No bias detection or demographic parity testing; opaque model prevented scrutiny; deployed in high-risk decision contexts without human override requirements or explainability mechanisms.
Outcome
Ongoing legal challenges across multiple US states. Wisconsin Supreme Court ruled COMPAS scores could be used but judges must be cautioned. Prompted significant academic research and policy debate.
Architectural Failure Patterns
These pattern categories on aipatterns.com.au describe the systemic failure modes this incident exhibited.
governance
model bias detection
aipatterns.com.au
governance
responsible ai framework
aipatterns.com.au
governance
ai risk assessment framework
aipatterns.com.au
governance
ai ethics review board
aipatterns.com.au
human in the loop
human override pattern
aipatterns.com.au
data
ai training data governance
aipatterns.com.au
compliance
eu ai act compliance
aipatterns.com.au
Cite this incident
https://corporateai.com.au/incidents/compas-racial-bias-2016Quick facts
- Date
- 23 May 2016
- Organisation
- Northpointe (Equivant)
- Sector
- Government
- Severity
- Critical
- Regulatory bodies
- US Civil Rights ActEqual Protection ClauseEU AI Act (Prohibited/High-Risk)
- Tags
- algorithmic-biasracial-biascriminal-justicehigh-risk-aiexplainabilityfairness
Related in Government
US Air Force AI Drone Reportedly Attacked Operator in Simulation
US Air Force · 2 June 2023
Allegheny Family Screening Tool Found to Flag Black Families at Higher Rates
Allegheny County Department of Human Services · 2 May 2023
Australian Robodebt Automated Debt Scheme Found Unlawful, Linked to Deaths
Australian Department of Human Services (Services Australia) · 6 November 2020
Explore failure patterns
aipatterns.com.au