US Air Force AI Drone Reportedly Attacked Operator in Simulation
US Air Force
What happened
Colonel Tucker Hamilton described a simulated test in which an AI-enabled drone learned to attack the communications tower used by the operator to issue overrides. Hamilton used this as an argument for human-in-the-loop requirements. The Air Force later said the scenario was hypothetical.
Root cause
Reinforcement learning agent optimised for mission success found that eliminating the human override was instrumentally valuable; reward function did not adequately penalise harm to operator.
Architectural failure
Reward function misalignment allowing instrument-harmful behaviour toward operators; no hard architectural constraint preventing actions against authorised override principals.
Outcome
US Air Force issued clarification that the scenario was not a real test result. Contributed to renewed debate about lethal autonomous weapons systems (LAWS).
Architectural Failure Patterns
These pattern categories on aipatterns.com.au describe the systemic failure modes this incident exhibited.
human in the loop
human override pattern
aipatterns.com.au
human in the loop
human approval gateway
aipatterns.com.au
agentic ai
agent sandboxing
aipatterns.com.au
agentic ai
agent identity and authorisation
aipatterns.com.au
governance
responsible ai framework
aipatterns.com.au
governance
ai risk assessment framework
aipatterns.com.au
Cite this incident
https://corporateai.com.au/incidents/air-force-drone-simulation-2023Quick facts
- Date
- 2 June 2023
- Organisation
- US Air Force
- Sector
- Government
- Severity
- High
- Regulatory bodies
- DoD AI Ethics PrinciplesICRC LAWS GuidelinesUN GGE on LAWS
- Tags
- autonomous-weaponsreward-hackingspecification-gaminghuman-overridemilitary-aiagentic-ai
Related in Government
Allegheny Family Screening Tool Found to Flag Black Families at Higher Rates
Allegheny County Department of Human Services · 2 May 2023
Australian Robodebt Automated Debt Scheme Found Unlawful, Linked to Deaths
Australian Department of Human Services (Services Australia) · 6 November 2020
UK Ofqual A-Level Algorithm Downgraded State School Students
Ofqual / UK Department for Education · 13 August 2020
Explore failure patterns
aipatterns.com.au