Study finds X accounts with AI-generated photos mostly spread political and financial slop

A new study shows that accounts on X (formerly Twitter) that use AI-generated profile pictures make up about 0.052 percent of profiles and often share content about politics, COVID-19, and the war in Ukraine.

Researchers from Ruhr University Bochum, GESIS Leibniz Institute, and CISPA Helmholtz Center analyzed nearly 15 million X accounts and identified 7,723 profiles using AI-generated portraits.

These accounts showed distinct patterns: they had fewer followers, followed fewer accounts themselves, and more than half were created in 2023—sometimes hundreds within hours.

Lead author Jonas Ricker notes that this is "clear indication that they weren’t real users." After nine months, X had suspended more than half of these accounts.

Trump, Corona and Ukraine as main themes

The study's methodology focused on recognizing images generated with the StyleGAN 2 AI model, so images generated by newer and potentially more widespread AI systems were not detected. This means that the study does not provide a comprehensive picture of the proportion of fake AI profile pictures, let alone fake accounts on X.

Example of an AI image generated with StyleGan. | Image: via thispersondoesnotexist.com

What is interesting, however, is the content of the detected accounts with fake AI images on X. A manual content analysis of 1,000 accounts with fake AI portrait images revealed a clear thematic focus on politics, often related to Trump, COVID-19 and vaccinations, and the war in Ukraine. Lotteries and financial topics such as cryptocurrencies were also common.

The analysis of 1000 manually checked active X accounts shows a clear dominance of political and financial topics, while business, sex and various other topics take up less space. | Image: Ricker et al.

The researchers speculate that some of these accounts were created specifically to spread disinformation and political propaganda, with the mass creation and identical metadata suggesting they are part of an organized network.

The visualization shows a distinct clustering of topics in English-language tweets, with politics and finance dominating. | Image: Ricker et al.

The researchers plan to expand the automation of AI fake profile image detection and include newer models in future analyses to better understand the scope and impact of the technology on social media platforms like X.

Although the study did not examine the reach of the fake AI image accounts, their lower average follower count and fewer followed accounts suggest they have limited reach. This aligns with an OpenAI study that found social media accounts spreading AI-generated messages for political propaganda received little response, rendering them largely ineffective.