
OpenAI announced on Monday a new tool powered by artificial intelligence. The tool is designed to detect child sexual abuse material, known as CSAM, including AI-generated images. According to the company, this initiative is part of broader efforts to combat child exploitation online.
The image classifier, as OpenAI describes it, can identify signs of abuse in previously unseen images. It helps safety teams distinguish potentially harmful content for further review. The company calls it the first widely available CSAM classifier.
The tool is able to indicate whether abusive material may have been generated by AI. The tool was developed with the help of the US-based National Center for Missing and Exploited Children. It is being offered free of charge to verified organizations through OpenAI’s application programming interface.
The company’s internal tests showed that the new tool can reduce the number of images flagged for review by 48%. This does not significantly affect its ability to detect abusive content. According to OpenAI, tests conducted by the technology company fal.ai confirmed that 72% of images flagged as highly suspicious contained previously unidentified abusive material.
Separate tests by SafetyKit showed that the tool identified 90% of the tested cases. These cases included sexualized depictions of children in drawings and animations. OpenAI also announced new safeguards for ChatGPT, including warnings for users who attempt to access abusive material and the removal of flagged conversations.
Such violations may result in a 24-hour suspension. Repeated violations may lead to longer restrictions, account closures, or reports to authorities. The company also published guidelines to help developers detect and report abusive material.
The guidelines come together with artificial intelligence tools and training resources for law enforcement agencies investigating cases of child exploitation.
