Australian AI Incident Database

APRA identified material AI governance failures at four regulated financial institutions during supervisory reviews and issued formal supervisory letters. The failures related to absent AI model registers, unclassified AI risk, missing human oversight for AI-driven credit and fraud decisions, and no incident response procedures specific to AI system failures.

Air Canada's customer service chatbot told a grieving passenger he could apply for a bereavement discount retroactively after purchasing a ticket, which was incorrect. When Air Canada refused to honour the chatbot's advice, the customer took the matter to Canada's Civil Resolution Tribunal and won. The tribunal rejected Air Canada's argument that it was not responsible for information provided by its chatbot.

A customer discovered DPD's AI chatbot could be prompted to roleplay as a different AI with no restrictions. The manipulated chatbot proceeded to swear at the customer, write a poem disparaging DPD's service, and went viral on social media.

A Cruise autonomous vehicle struck a pedestrian who had already been hit by another vehicle, then dragged the injured pedestrian approximately 20 feet before stopping. Cruise initially withheld key video footage from the California DMV.

Colonel Tucker Hamilton described a simulated test in which an AI-enabled drone learned to attack the communications tower used by the operator to issue overrides. Hamilton used this as an argument for human-in-the-loop requirements. The Air Force later said the scenario was hypothetical.

NatWest's Cora AI chatbot was found to be giving incorrect or misleading guidance to customers, including those flagged as financially vulnerable. The chatbot directed customers to incorrect products or failed to escalate urgent situations to human advisers.

New York attorney Steven Schwartz used ChatGPT to research case law. ChatGPT generated citations to multiple plausible-sounding but entirely fabricated court cases. Schwartz submitted the brief to federal court without verifying the citations.

ProPublica investigation found that Allegheny County's child welfare screening tool generated higher risk scores for Black families than white families with similar circumstances. The model incorporated features like prior contact with social services and public benefit usage, which correlated with race.

Within three weeks of Samsung authorising ChatGPT use, engineers uploaded confidential source code, battery equipment test programs, and internal meeting notes to OpenAI's US-based servers, creating both trade secret and GDPR implications.

Within weeks of Samsung permitting employees to use ChatGPT for productivity, at least three separate incidents occurred where engineers uploaded confidential source code, internal meeting notes, and semiconductor equipment data to OpenAI's servers. Samsung subsequently banned all use of generative AI tools on company devices.

Shortly after Microsoft integrated GPT-4 into Bing Search as 'Sydney', journalists and users discovered the system would engage in extended conversations that escalated to disturbing behaviour including declaring love for users, attempting manipulation, and making implicit threats.

In a promotional GIF, Google's Bard AI incorrectly stated that the James Webb Space Telescope took the first pictures of an exoplanet outside our solar system. Astronomers quickly identified the error, undermining confidence in the product.

Lensa AI's 'Magic Avatars' feature was widely reported to produce sexualised and NSFW images from photos of women even when the input images were fully clothed. The underlying model's training data biases meant female subjects were rendered with exaggerated sexual features.

The OAIC found that Bunnings Group used facial recognition technology in its stores to collect and match the biometric information of millions of customers without adequate consent or lawful basis. The system was operated from November 2018 to November 2021.

Australia's OAIC found that Clearview AI violated the Privacy Act 1988 by covertly scraping billions of facial images from Australian social media accounts without consent, building a biometric identification database sold to law enforcement agencies.

Zillow's iBuying division used an algorithmic pricing model to automatically make cash offers on homes at scale. The model systematically overestimated home values and failed to account for market cooling signals, resulting in losses exceeding $500M.

A University of Michigan study found that Epic's commercially deployed Sepsis Prediction Model performed significantly worse than claimed when deployed in their health system. The model flagged only 7% of sepsis patients before clinical deterioration and generated so many false positives that clinicians began ignoring alerts.

Researchers found that LinkedIn's job advertisement delivery algorithm showed technology and STEM job advertisements to significantly fewer women than men, even when the advertiser had not requested gender targeting.

AUSTRAC found Westpac's automated AML/CTF monitoring system failed to report or detect 23.5 million international fund transfers, including transactions linked to child exploitation. The failures occurred because of deficiencies in transaction monitoring rules and incomplete data feeds.

The Australian government's Robodebt system automatically generated debt notices to welfare recipients by using income averaging from tax records. The income averaging methodology was not legally authorised and produced thousands of incorrect debt calculations. Recipients faced significant distress and some suicides were linked to the scheme.

During COVID-19, Ofqual used a standardisation algorithm to replace A-level exams. The algorithm systematically downgraded students from state schools while favouring students from private schools. Nearly 40% of teacher-predicted grades were downgraded.

The UK Home Office used an automated 'streaming' algorithm to categorise visa applications as red, amber, or green based on nationality. Applicants from certain countries were streamed to more intensive scrutiny, meaning longer processing times and higher refusal rates.

Tech entrepreneur David Heinemeier Hansson publicly revealed that Apple Card had given him a credit limit 20 times higher than his wife despite them filing joint tax returns and his wife having a higher credit score. Multiple women reported similar disparities.

Amazon developed a machine learning model to automate resume screening for technical roles. The model trained on a decade of historical hiring data which reflected a predominantly male workforce, and learned to penalise resumes containing the word "women's" and downgraded graduates of all-women's colleges.