OpenAI has identified and banned a Russia-origin influence operation named "Stop News" that utilized AI to produce multilingual articles and social media content. This activity, which appeared consistent with a marketing company's operation, generated content in English, French, and Russian, subsequently posted across various websites and social media platforms. OpenAI initiated its investigation after reviewing off-platform indicators detailed in Meta's 2024 second quarter threat report.
Key Points
- The "Stop News" operation used OpenAI's models to generate short comments, long-form articles, and images.
- Long-form articles in English and French were posted on websites posing as news outlets in Africa and the UK, while short comments appeared on X (formerly Twitter).
- Content in Russian served as promotional material on various websites and Russian-language forums.
- The operation extensively used DALL·E to generate cartoon-style images with bright colors or dramatic tones to attract attention.
- Some UK-focused websites associated with the operation established "information partnerships" with public entities, including a school in Wales and a church in Yorkshire.
- The operation's social media accounts, including those on X, Facebook, and Instagram, generally garnered low engagement, with most X accounts having double-digit follower counts and Meta assets totaling 2,100 followers.
- OpenAI also banned a separate Russia-origin operation, "Helgoland Bite," which generated German-language political content about Ukraine, NATO, and domestic issues, distributed on Telegram and X, with one X account having over 27,000 followers.
Context
According to OpenAI, the "Stop News" operation generated content in English focusing on news, sporting, and cultural events in the UK, with some outlets mixing this with criticism of Ukraine and the UK's support. French content often centered on music and culture in Africa, sometimes criticizing Ukraine and France. Russian content primarily promoted commercial services in north-western Russia. OpenAI also noted that this operation, like others it has disrupted, combined modern AI techniques for content generation with older methods, such as using Generative Adversarial Network (GAN) images for profile pictures, a tactic observed in influence operations since at least 2019.
Why It Matters
This disruption highlights how AI models are being leveraged in sophisticated, multilingual influence operations, posing challenges for platform providers and content consumers. Builders and users of AI systems must remain aware of the potential for misuse in generating deceptive content and the evolving tactics employed by malicious actors.
What To Do
- Note the specific platforms and content types (articles, comments, images) used in these operations.
- Observe the combination of modern AI tools like DALL·E with older techniques like GANs for different aspects of influence campaigns.
- Watch for reports from platform providers like OpenAI and Meta detailing new influence operation tactics and disruptions.
- Consider how low engagement metrics on social media do not always reflect the full scope of an operation's reach, especially when "information partnerships" are established.