Lensa AI Generated Sexualised Images From Users' Normal Selfies
Prisma Labs (Lensa AI)
What happened
Lensa AI's 'Magic Avatars' feature was widely reported to produce sexualised and NSFW images from photos of women even when the input images were fully clothed. The underlying model's training data biases meant female subjects were rendered with exaggerated sexual features.
Root cause
Base model training data contained disproportionate sexualised imagery of women; no output filtering for NSFW content before delivery to users.
Architectural failure
No output filtering or NSFW classifier applied to generated images before delivery; training data governance did not audit for representational harms; no consent mechanism for sexualised outputs.
Outcome
Widespread media coverage and user complaints. Prisma Labs updated Lensa with stronger NSFW filtering. UK Online Safety Act provisions on NCII subsequently strengthened.
Architectural Failure Patterns
These pattern categories on aipatterns.com.au describe the systemic failure modes this incident exhibited.
Cite this incident
https://corporateai.com.au/incidents/lensa-ai-nsfw-images-2022Quick facts
- Date
- 10 December 2022
- Organisation
- Prisma Labs (Lensa AI)
- Sector
- Other
- Severity
- High
- Regulatory bodies
- UK Online Safety Act 2023GDPRUS SHIELD ActApp Store Guidelines
- Tags
- generative-ainsfwgender-biasimage-generationnon-consensual-imagerytraining-data-bias
Related in Other
Air Canada Chatbot Gave Wrong Bereavement Refund Policy
Air Canada · 14 February 2024
DPD Chatbot Jailbroken to Swear at Customer and Criticise DPD
DPD · 19 January 2024
Cruise Autonomous Vehicle Dragged Pedestrian 20 Feet After Collision
Cruise (General Motors) · 2 October 2023
Explore failure patterns
aipatterns.com.au