In late January 2024, X was filled with graphic, intense images of Taylor Swift. While celebrities have long been victims of photo leaks and cyber attacks, this time it was different as it was generated using artificial intelligence.
The photos were immediately reported by “Shake It Off”. After being live on the poster’s profile for less than a day, the singer’s fan count was removed. However, they had plenty of time to go viral despite the platform having policies against non-consensual nudity. A report by disinformation research firm Graphica later found that the images were created on 4chan, where users used famous female celebrities to sexually assault each other in an attempt to circumvent nudity content policies. Encouraged to create charged deep faxes.
Unfortunately, Swift’s experiment is not successful. Marvel actress Xochitl Gomez, who was just 17 at the time of the reporting, said on the podcast “The Squeeze” that she struggled to get her deepfake removed from X and shared the psychological impact it had on her. What did Gomez and Swift are just two of the countless women who have recently become victims of deepfakes that portray them in sexual ways.
“People have always used media to try and denigrate people, that hasn’t changed. What’s changed is how accessible it is now,” said Dr. Professor Sui Liu told The Daily Beast.
Late last year, AI image generation platform CivitAI became popular for its “Bounties” feature, which encouraged users to create deepfakes in exchange for virtual rewards. Almost all of the awards created were for women, reports . 404 media. Some of the women included were not celebrities or public figures but private citizens.
Experts expect it to only get worse—especially as more and more incel communities use these technologies online. AI and deepfake consultant and expert Henry Ajder told The Daily Beast that this has been a growing problem for years, and CivitAI is an example of a platform heavily involved in this type of evolution.
He said that CivitAI has “become a hub for not only artistically created content but erotic content. It’s a niche for getting niche knowledge and people have started using it for porn.” Is.
Ajder also describes the technology on the platform as “agnostic or dual-use,” saying that once it’s there, it can be used in any way, “whereas, others clearly is designed to create obscene material without consent.” These tools have only become popular in insel culture through platforms like Reddit and 4chan.
“Governments are just trying to play catch-up: technology has outpaced their ability to regulate.“
– Belinda Barnett
“There is such a low threshold,” Heera Hussain, founder of China, a non-profit organization helping victims of gender-based violence and trauma, told The Daily Beast. “It’s an accessible method that allows people to fulfill their darkest fantasies. […]They may feel that it is not hunting, but it has huge consequences for these people.
It’s not just deepfakes that have infiltrated the insel culture either. There’s even research that shows AI girlfriends may be making Ancel even more dangerous. This tech allows them to shape and control their own visions of a so-called “ideal woman,” with the risk that they may push those visions. real When women find themselves unable to do so, or when a woman feels unapproachable, as in the case of Swift or Gomez, incels start deep fake campaigns. At least, then, incels can do what they like with these women.
“Governments are just trying to play catch-up: technology has outpaced their ability to regulate,” Belinda Barnett, senior lecturer in media at Swinburne University, told The Daily Beast.
‘A different kind of trauma’
It becomes even more alarming when we look at the global context. Patriarchal norms in various nations often put women who fall victim to such campaigns at greater risk. In many more conservative countries, even a woman’s deep lie can be enough for her family to ostracize her or, in extreme cases, use violence against her. For example, in late 2023, an 18-year-old girl was murdered by her father over a photo of her with a man who police suspected had been treated by a doctor.
It doesn’t matter if the photo is fake. The fact that their image is associated with such depictions is enough for society to ostracize them. “It’s not so much about people believing the images to be real as it is about being pure. Revenge porn is a different kind of trauma,” Ajdar explained.
With the AI generation being more accessible, it also creates an easier barrier of entry for global incels who may struggle with language barriers. In South Asia, where Hussain focuses more on her work, at the social and policy level, countering extremism becomes even more difficult. “They don’t have the strong radicalization that they see in the Anselm community,” he explained.
Policies around free speech and technological access differ around the world, Liu says, and so can have different effects. “In the US, using AI generation tools to create content […]There is freedom of speech — but people can also take advantage of it. This line becomes very difficult to draw. Whereas in China, there are very strict restrictions on the use of this technology, so it is possible but prevents the positive use of the same line of technology.”
Insel culture existed long before AI generation tools became popular. Now that they are mainstream, these communities will be quick to adopt them to cause further harm and trauma. It is certain that the problem will get worse before it gets better.
Ajdar said, “In terms of incel culture, this is another weapon in their twisted arsenal to mistreat women, perpetuate stereotypes, and further their twisted views of women. “