Australian Robodebt Automated Debt Scheme Found Unlawful, Linked to Deaths
Australian Department of Human Services (Services Australia)
What happened
The Australian government's Robodebt system automatically generated debt notices to welfare recipients by using income averaging from tax records. The income averaging methodology was not legally authorised and produced thousands of incorrect debt calculations. Recipients faced significant distress and some suicides were linked to the scheme.
Root cause
Automated decision algorithm applied an unlawful income-averaging methodology that had never been legally validated; no human review of individual decisions.
Architectural failure
Automated decisions at high personal impact with no human-in-the-loop review requirement; no legal compliance gateway; no audit mechanism to surface incorrect outcomes.
Outcome
Federal Court class action settlement of AUD $1.8 billion in 2021. Government admitted the scheme was unlawful. Royal Commission established, finding systemic failures.
Architectural Failure Patterns
These pattern categories on aipatterns.com.au describe the systemic failure modes this incident exhibited.
human in the loop
human approval gateway
aipatterns.com.au
human in the loop
human override pattern
aipatterns.com.au
governance
ai regulatory compliance gateway
aipatterns.com.au
governance
ai audit trail
aipatterns.com.au
governance
responsible ai framework
aipatterns.com.au
governance
ai risk assessment framework
aipatterns.com.au
compliance
eu ai act compliance
aipatterns.com.au
Cite this incident
https://corporateai.com.au/incidents/robodebt-australia-2016-2020Quick facts
- Date
- 6 November 2020
- Organisation
- Australian Department of Human Services (Services Australia)
- Sector
- Government
- Severity
- Critical
- Regulatory bodies
- Social Security Act 1991 (Australia)Administrative Decisions (Judicial Review) ActPrivacy Act 1988
- Tags
- automated-decisionsgovernment-aiunlawfulwelfaredebt-recoveryhuman-rights
Related in Government
US Air Force AI Drone Reportedly Attacked Operator in Simulation
US Air Force · 2 June 2023
Allegheny Family Screening Tool Found to Flag Black Families at Higher Rates
Allegheny County Department of Human Services · 2 May 2023
UK Ofqual A-Level Algorithm Downgraded State School Students
Ofqual / UK Department for Education · 13 August 2020
Explore failure patterns
aipatterns.com.au