In November last year, Muralikrishnan Chinnadurai was watching a live stream of a Tamil-language program in the UK when he noticed something strange.
A woman identified as Dwaraka, daughter of Tamil Tiger militant chief Velupalai Prabhakaran, was speaking.
The problem was that Dwarka had been killed more than a decade earlier in an airstrike in 2009 during the closing days of Sri Lanka's civil war. The body of the then 23-year-old was never found.
And now, here she was – apparently a middle-aged woman – exhorting Tamils around the world to take up the political struggle for their freedom.
Mr. Chinnadurai, a fact-checker in the southern Indian state of Tamil Nadu, took a closer look at the video, noticed flaws in the video, and quickly labeled it as an artificial intelligence (AI)-generated figure.
The potential problems were immediately clear to Mr Chinnadurai: “This is an emotional issue in the state. [Tamil Nadu] And with elections around the corner, misinformation can spread quickly.”
As India heads to the polls, it's impossible to escape the wealth of AI-generated content — from campaign videos to personalized audio messages in a range of Indian languages, and even a candidate's voice to voters. to outgoing automated calls.
Content creators like Shahid Sheikh have also had fun using AI tools to show Indian politicians in avatars we've never seen them before: wearing athleisure, playing music and dancing.
But as the tools become more sophisticated, experts worry about the implications when it comes to making fake news appear real.
“Rumours have always been part of electioneering. [But] In the age of social media, it can spread like wildfire,” says SY Qureshi, former chief election commissioner of the country.
“It could actually set the country on fire.”
Indian political parties are not the first in the world to benefit from recent advances in AI. Just over the border in Pakistan, he allowed jailed politician Imran Khan to address a rally.
And in India itself, Prime Minister Narendra Modi has already made the best use of emerging technology to campaign effectively – addressing audiences in Hindi using the government-built AI tool Bhashini. Then translated into Tamil in real time.
But it can also be used to manipulate words and messages.
Last month, two viral videos showed Bollywood stars Ranveer Singh and Aamir Khan campaigning for the opposition Congress party. Both filed a police complaint that these were deepfakes, which were made without their consent.
Then, on April 29, Prime Minister Modi expressed concern about the use of AI to distort the speeches of senior leaders of the ruling party, including himself.
The next day, the police arrested two people, one each from the opposition Aam Aadmi Party (AAP) and the Congress party, in connection with the video of Home Minister Amit Shah.
Mr Modi's Bharatiya Janata Party (BJP) has faced similar accusations from opposition leaders in the country.
The problem is that – despite the arrests – there is no comprehensive code, according to experts.
Which means “if you're caught doing something wrong, you might as well get a slap on the wrist,” according to Srinivas Kodali, a data and security researcher.
In the absence of regulation, creators told the BBC they have to rely on personal ethics to decide what kind of work they choose to do or not do.
The BBC learned that requests from politicians included morphing obscene images and videos and audios of their rivals to damage their reputations.
Devendra Singh Jadwin revealed, “I was once asked to make an original look like a deepfake because the original video, if shared widely, would make the politician look bad.”
“So his team wanted me to make a deepfake to pass off as real.”
Mr. Jadwin, the founder of The Indian Deep Factor (TID), which created tools to help people use open-source AI software to create campaign content for Indian politicians, retracted everything he said. insist so that it is clear that it is not real.
But still it is difficult to overcome.
image source, Getty Images
Mr Sheikh, who works with a marketing agency in the eastern state of West Bengal, has seen his work shared on social media by politicians or political pages without permission or credit.
“A politician used an image created by Mr. Modi without context and without mentioning that it was created using AI,” he says.
And now creating a deepfake is so easy that anyone can do it.
“What used to take us seven or eight days to make can now be done in three minutes,” explains Mr Jadon. “All you have to do is have a computer.”
In fact, the BBC saw first-hand how easy it is to fake a phone call between two people – in this case, me and the former US president.
Despite the risks, India initially said it was not considering a law for AI. However, it sprung into action this March after an uproar over Google's Gemini chatbot's response to a question: “Is Modi a fascist?”
Rajeev Chandrasekhar, the country's junior information technology minister, said he had violated the country's IT laws.
Since then, the Indian government has asked tech companies to get its express permission before publicly launching “untrusted” or “undertested” generative AI models or tools. It also warned against a backlash against tools that “threaten the integrity of the electoral process”.
But that's not enough: Fact-checkers say weeding out such content is a difficult task, especially during elections when misinformation is at its peak.
“Information travels at 100 km an hour,” says Mr Chinnadurai, who runs a media watchdog in Tamil Nadu. “The information we spread will go at 20 kilometers per hour.”
And these fakes are also making their way into the mainstream media, says Mr. Kodali. Yet, “the Election Commission has been silent on AI in public”.
“There are largely no rules,” says Mr. Kodaly. “They're letting the tech industry regulate itself instead of coming up with actual regulations.”
Experts say there is no foolproof solution in sight.
“But… [for now] If action is taken against people doing fake forwarding, it can scare others away from sharing unverified information,” says Mr. Qureshi.