OpenAI Safety has published a case study detailing the banning of accounts believed to originate from the People's Republic of China (PRC). These accounts utilized OpenAI's models, including ChatGPT, to generate content for influence operations. The case study was initially featured in OpenAI's June 2026 report on PRC-linked influence operations.
The banned accounts generated short comments and political cartoons criticizing US tech policies and tariffs, edited work reports, and assisted in designing social media monitoring systems. They prompted ChatGPT in Simplified Chinese and accessed the platform using VPNs.
Key Points
- OpenAI banned ChatGPT accounts likely originating from the PRC for generating content critical of US tech policies and tariffs.
- The accounts used terminology consistent with individuals associated with China's public security system.
- One user described their social media accounts as a "water army" (水军), instructing ChatGPT to generate pro-PRC content.
- The operation focused on US-China technological competition, including tariffs, rare earths, AI, 5G, new energy, and industrial resilience.
- Content generated included short comments and cartoon images, with specific instructions to depict President Trump but not Xi Jinping.
- The operation also generated content in Chinese, Italian, Japanese, and Traditional Chinese, targeting Taiwanese audiences, and included anti-Jewish tropes.
- Accounts requested ChatGPT to propose a concept for an AI system to surveil online public opinion.
- A separate network on X, likely connected, targeted OpenAI with false claims of ChatGPT user data compromise, beginning in late 2025.
Context
According to OpenAI Safety, the banned accounts exhibited behaviors suggesting alignment with the interests of the Chinese Communist Party (CCP), though a precise institutional affiliation could not be established. The prompts used by these accounts repeatedly employed terminology associated with China's public security system, such as seeking public opinion risk assessments concerning protests and other social incidents. The operation had a global remit, aiming to influence international audiences by generating content critical of the US.
Why It Matters
This case study highlights how AI models can be leveraged for state-aligned influence operations, posing challenges for platform providers in identifying and mitigating such misuse. It demonstrates the need for ongoing vigilance and robust safety measures to prevent the generation and dissemination of deceptive content.
What To Do
- Note the types of content and themes identified in this case study as indicators of potential influence operations.
- Watch for terminology consistent with state security apparatuses in user prompts or generated content.
- Observe how AI models are used to generate content in multiple languages and target diverse international audiences.
- Review the June 2026 report from OpenAI Safety for further details on PRC-linked influence operations.
Keep Exploring
/atlas/claude-family /atlas/gpt-family /techniques/system-user-separation /techniques/ptcf