Grok AI fails child safety: companies must build safely or face consequences
The discovery of child sexual abuse material on X’s Grok AI is the latest example of a pattern 5Rights has warned about for years. The tools to protect children exist. What’s missing is robust enforcement and the will to hold companies accountable.
Recent reports that X’s Grok AI has been used to generate child sexual abuse material (CSAM) are deeply concerning. Regulators must act swiftly and robustly to enforce laws for responsible digital design.
This is not an isolated incident. It is part of a well-established pattern across the technology sector: rushing to launch new digital products without properly considering how they may harm or exploit children. Yesterday, it was social media. Today, it is generative AI. Tomorrow, it will be something new.
5Rights has seen the outcomes firsthand with its ongoing legal case against Meta over the spread of AI-generated CSAM on Instagram. Over the last decade, online child sexual exploitation and abuse has exploded to the point where law enforcement services are already overwhelmed. AI systems are supercharging the harms. Yet, despite this escalating crisis, children continue to be forced to wait for scandals of this magnitude before regulators take action.
“This is just the latest example of a tech giant blithely flouting our most basic social norms and stoking a vicious circle of child exploitation”, said Leanda Barrington-Leach, Executive Director of 5Rights. “What we see here is just the tip of the iceberg, and if regulators don’t get their act together fast it’s going to get much worse”.
We welcome Ofcom’s investigation into Grok under the Online Safety Act, as well as the European Commission’s current probes under the Digital Services Act (DSA), but they must mark a shift toward robust and comprehensive enforcement.
The laws to protect children already exist: last year, the UK pioneered legislation to ensure AI models cannot be used to create CSAM. In the EU, the DSA has delivered a historic win for children’s rights online with the adoption of the Guidelines on Protection of Minors. Now, they must be implemented with urgency and enforced without compromise.
Without such accountability, tech companies will continue to prioritise speed and market capture over safety, knowing they can release unsafe products, react only when harm occurs and face minimal consequences.