OpenAI announced it banned a cluster of ChatGPT accounts in February 2025 that were using its models to assist in developing surveillance tools. The accounts, which OpenAI assessed as likely originating in China, engaged in activities such as analyzing documents, generating sales pitches for social media monitoring tools, editing and debugging code, and researching political actors.
OpenAI dubbed this network “Peer Review” based on its behavior in promoting and reviewing surveillance tooling. The activity patterns of these accounts were consistent with mainland Chinese business hours, and prompts were submitted in Chinese. The volume and variety of usage suggested manual prompting rather than automation.
Key Points
- OpenAI banned a cluster of ChatGPT accounts in February 2025 due to their use in AI-assisted surveillance planning.
- The accounts, likely originating in China, operated during mainland Chinese business hours and prompted models in Chinese.
- Activities included researching publicly available information on think tanks and politicians in the United States, Australia, and Cambodia.
- Operators used models to read, translate, and analyze screenshots of English-language documents, some related to Uyghur rights protests.
- ChatGPT was used to generate comments about Chinese dissident organizations like Falun Gong and US policies, occasionally adopting the persona of an English-speaker named “Thompson.”
- The models assisted in editing and debugging code, and generating promotional materials for an AI-powered social media listening tool, the “Qianyue Overseas Public Opinion AI Assistant.”
- One account debugged code that frequently named Meta’s llama3.1:8b deployed via Ollama as the driver for analysis and generation.
- Another account debugged code for malware analysis and referenced other models such as Qwen (from Alibaba Cloud) and an unspecified model by DeepSeek.
Context
According to OpenAI, the network’s primary tasks included using the model as a basic research tool, similar to how search engines were used previously. This involved gathering information on political entities and topics. Another workstream focused on document analysis, where models translated and analyzed English-language screenshots, some of which depicted announcements of Uyghur rights protests or concerned diplomatic topics in the Indo-Pacific region. OpenAI noted insufficient evidence to determine the authenticity of these documents or their acquisition methods.
A third workstream involved generating comments on Chinese dissident organizations and US politics. A significant portion of the activity, according to OpenAI, appeared to be for capability development, such as code debugging, and internal purposes like image analysis and promotional material creation, rather than for direct online publication.
Why It Matters
This case highlights the dual-use challenge of AI models, demonstrating how general-purpose capabilities can be adapted for activities like surveillance planning and tool development. For builders, it underscores the importance of understanding potential misuse vectors and the need for robust safety and monitoring mechanisms in AI platforms.
What To Do
- Note the types of activities, such as code debugging and promotional material generation, that indicate internal capability development rather than direct content publication.
- Observe the specific models referenced in the debugging activities, including Meta’s llama3.1:8b, Qwen, and DeepSeek.
- Consider the behavioral patterns, such as operating hours and language use, that OpenAI identified as indicators of origin and intent.
- Review OpenAI's February 2025 report for additional details on their safety and security investigations.