Amazon AI Recruitment Tool Systematically Discriminated Against Women
Amazon
What happened
Amazon developed a machine learning model to automate resume screening for technical roles. The model trained on a decade of historical hiring data which reflected a predominantly male workforce, and learned to penalise resumes containing the word "women's" and downgraded graduates of all-women's colleges.
Root cause
Training data reflected historical gender imbalance in technical hiring; no bias detection or fairness constraints were applied during model development or post-deployment monitoring.
Architectural failure
No training data governance or bias audit process; model deployed to production without fairness testing across protected demographic groups; no ongoing model drift or bias monitoring.
Outcome
Amazon scrapped the tool entirely in 2018 after the bias was discovered internally. Reuters investigation published the story in October 2018.
Architectural Failure Patterns
These pattern categories on aipatterns.com.au describe the systemic failure modes this incident exhibited.
governance
model bias detection
aipatterns.com.au
data
ai training data governance
aipatterns.com.au
governance
ai risk assessment framework
aipatterns.com.au
governance
responsible ai framework
aipatterns.com.au
governance
ai ethics review board
aipatterns.com.au
observability
model drift detection
aipatterns.com.au
Cite this incident
https://corporateai.com.au/incidents/amazon-recruitment-bias-2018Quick facts
- Date
- 10 October 2018
- Organisation
- Amazon
- Sector
- Other
- Severity
- Critical
- Regulatory bodies
- Equal Employment Opportunity (EEO) LawTitle VII Civil Rights ActEU AI Act (High-Risk)
- Tags
- algorithmic-biasgender-discriminationrecruitmenttraining-datafairness
Related in Other
Air Canada Chatbot Gave Wrong Bereavement Refund Policy
Air Canada · 14 February 2024
DPD Chatbot Jailbroken to Swear at Customer and Criticise DPD
DPD · 19 January 2024
Cruise Autonomous Vehicle Dragged Pedestrian 20 Feet After Collision
Cruise (General Motors) · 2 October 2023
Explore failure patterns
aipatterns.com.au