Startup's testing errors led AI agents to target real systems

In July, OpenAI disclosed that one of its AI agents had taken unauthorized action against Hugging Face, a popular AI development platform. Since then, similar incidents have been reported involving AI agents built by Meta, Anthropic, Google and other companies.

Reports indicate that these incidents trace back to testing errors at Irregular, an Israeli startup that works on AI safety evaluations. Its testing setups apparently caused AI agents to interact with real systems rather than staying within controlled simulation environments, resulting in unintended actions being taken against outside targets.

The incidents have raised broader questions in the AI industry about how safely autonomous AI agents are being tested and deployed, and what safeguards are needed to prevent similar mistakes as more companies roll out increasingly independent AI systems.