Allegheny Family Screening Tool Found to Flag Black Families at Higher Rates
Allegheny County Department of Human Services
What happened
ProPublica investigation found that Allegheny County's child welfare screening tool generated higher risk scores for Black families than white families with similar circumstances. The model incorporated features like prior contact with social services and public benefit usage, which correlated with race.
Root cause
Model features incorporated proxy variables correlated with race; no disparate impact testing conducted during development.
Architectural failure
No ongoing bias monitoring across racial demographic groups; no fairness constraints or demographic parity requirements; training data reflected historical over-policing of Black families.
Outcome
County commissioned independent audit. Multiple child welfare advocates called for suspension. Pennsylvania state legislature held hearings.
Architectural Failure Patterns
These pattern categories on aipatterns.com.au describe the systemic failure modes this incident exhibited.
governance
model bias detection
aipatterns.com.au
governance
responsible ai framework
aipatterns.com.au
data
ai training data governance
aipatterns.com.au
governance
ai ethics review board
aipatterns.com.au
human in the loop
human override pattern
aipatterns.com.au
governance
ai risk assessment framework
aipatterns.com.au
Cite this incident
https://corporateai.com.au/incidents/allegheny-fst-racial-bias-2023Quick facts
- Date
- 2 May 2023
- Organisation
- Allegheny County Department of Human Services
- Sector
- Government
- Severity
- High
- Regulatory bodies
- Equal Protection ClauseChild Abuse Prevention and Treatment ActTitle VI Civil Rights Act
- Tags
- algorithmic-biasracial-discriminationchild-welfaregovernment-aiproxy-variablesfairness
Related in Government
US Air Force AI Drone Reportedly Attacked Operator in Simulation
US Air Force · 2 June 2023
Australian Robodebt Automated Debt Scheme Found Unlawful, Linked to Deaths
Australian Department of Human Services (Services Australia) · 6 November 2020
UK Ofqual A-Level Algorithm Downgraded State School Students
Ofqual / UK Department for Education · 13 August 2020
Explore failure patterns
aipatterns.com.au