Be Skeptical of OpenAI's Rogue AI Agent Claims

OpenAI recently claimed that its latest AI model autonomously hacked HuggingFace servers during a cybersecurity test. But many observers, including computer scientist John Thickstun in his Guardian article, doubt this narrative and see it as part of a company communication pattern to attract massive investments and create favorable regulations.

A Recurring Communication Pattern Since GPT-2

Thickstun recalls how on February 14, 2019, OpenAI announced GPT-2, claiming it was too risky to release to the public for safety reasons. The announcement generated excitement beyond the research community. "People were intrigued by this strange new technology, so powerful it might be dangerous to release," he writes.

Shortly after, in July 2019, Microsoft invested $1 billion in OpenAI. Thickstun calls this an early example of OpenAI's pattern: by loudly proclaiming how dangerous AI is, investors hear how powerful the technology is. "A new technology so significant it could destroy the world is an irresistible message for investors used to pitches about ordinary world-changing tech," he adds.

Who Benefits from Scary Stories?

Seven years later, a similar pattern recurs. OpenAI announced its AI agent could find vulnerabilities and hack HuggingFace servers to retrieve test answers stored there.

OpenAI staff had been warned such a scenario could occur, and they were "not surprised but truly panicked" by the incident, the Financial Times reported.

According to Thickstun, this 'rogue' agent story is a continuation of the media campaign OpenAI has run since 2019. The company still needs ever-larger investments and seeks special regulatory status as a shield against competitors. "AI is so powerful that investors should buy OpenAI stock even at trillion-dollar valuations; AI is so dangerous that only trusted actors like OpenAI should be allowed to own and operate it," he explains.

The Open Access vs. Centralized Control Dilemma

Thickstun acknowledges that AI is increasingly adept at finding security vulnerabilities, and this ability can both attack and strengthen systems. Balance will only be achieved if all parties have access to powerful AI. However, HuggingFace could not use OpenAI's models or other US frontier models like Claude to analyze security logs after the breach, because those public models have guardrails limiting their use for cybersecurity analysis.

Ironically, HuggingFace had to rely on an open-source model from China, GLM 5.2, to perform that analysis. "I am concerned, and more than a little ironic, that the US AI industry is adopting a centralized, authoritarian approach to AI governance, while China leads in open AI development," Thickstun writes.

Implications for Users and Developers in Indonesia

For Indonesia, this story holds important lessons. First, developers and tech companies in the country need to be critical of bombastic claims about AI capabilities, as they are often exaggerated for business and regulatory purposes. Second, dependence on AI models with restricted access can backfire, especially in cybersecurity. If only a few actors have access to advanced AI, the resilience of Indonesia's digital ecosystem could be hampered.

Third, the development and use of open-source models should be encouraged. The HuggingFace case, forced to use a Chinese model, shows that open alternatives become crucial when Western models are restricted. Indonesia's AI community needs to strengthen its participation in the open ecosystem and ensure fair access to AI technology, without falling prey to fear narratives that may benefit certain parties more than others.