Humor and humor are endlessly important and subjective, but Google DeepMind researchers found a consensus among professional humorists: “AI is pretty bad at it.”
This was one of many comments collected during a study conducted with twenty professional comedians and performers during workshops at the Edinburgh Festival Fringe last August 2023 and online. The results showed that large language models (LLMs) captured by chatbots presented significant challenges and raised ethical concerns about the use of AI in generating humorous content.
The research involved a three-hour workshop in which comedians participated in a comedy writing session with popular LLMs such as ChatGPT and Bard. It also evaluated the quality of production through a human-computer interaction questionnaire based on the decades-old Creativity Support Index (CSI), which measures how well a tool supports creativity.
Participants also discussed the motivations, processes and ethical concerns of using AI in comedy in a focus group.
The researchers asked comedians to use AI to write stand-up comedy routines and then have them review the results and share their thoughts. The results were not good.
One participant described the AI-generated content as “the blandest, most boring thing — I stopped reading it. It was so bad.” Another referred to the output as “a vomit draft that I know I'll have to iterate and refine.”
“And I don't want to live in a world where it's better,” said another.
The research found that LLMs were able to produce long routine sketches and pieces, but they clearly lacked the human elements that made something funny. When asked to develop a draft structure, the models “presented a scene that provided a lot of structure,” but when it came to details, “LLMs as a tool to support creativity. Didn't succeed.”
Among the reasons, the authors note, was the “global cultural value alignment of LLMs,” as the tools used in the study generated content based on all the collected material, spanning every possible discipline. It also introduced a form of bigotry, which comedians pointed out.
“Participants noted that LLMs reinforced hegemonic perspectives by erasing minority groups and their perspectives, as a form of censorship, in line with current moderation strategies and guidelines used in protective filtering. But qualified,” the study said.
Popular LLMs are limited, the researchers said, citing the so-called “HHH standard,” which calls for honest, harmless, and helpful output—which is “what most users want from an engaged AI. “
The material was described by one panelist as “1950s cruise ship comedy material, but a little less racist”.
Another participant said, “The broader the appeal of something, the less likely it is.” “If you make something that fits everyone, it might not become anyone's favorite.”
The researchers emphasized the importance of considering the subtle differences between hurtful speech and offensive language used in protest and derision. Comedians meanwhile also complained that AI failed because it didn't understand nuances like sarcasm, dark humor, or irony.
“I could have black bits in a lot of things, and then it wouldn't write me any black stuff, because it thought I was going to commit suicide,” one part reported. “So he stopped giving me anything.”
Research found that the fact that chatbots were based on written content did not help.
“Given that the current widely available LLMs are primarily accessible through text-based chat interfaces, they felt that the utility of these tools was limited to the subset of domains required to develop a complete comic product. is,” the researchers noted.
“Any written text can be a fine text, but a great actor can make it a lot of fun,” said one participant.
The study revealed that the limits of AI in comic writing go beyond simple content creation. Comedians emphasized that point of view and point of view are uniquely human traits, with one comedian noting that humans “add more nuances and emotions and nuances” because of their lived experience and connection to the material. “.
Many described the centrality of personal experience to good comedy, enabling them to draw on memories, acquaintances, and beliefs to construct authentic and engaging narratives. Moreover, comedians emphasized the importance of understanding cultural context and audience.
“The kind of comedy I can do in India will be very different from the kind of comedy I can do in the UK, because my social context will change,” said one of the participants.
Thomas Winters, one of the researchers cited in the study, explains why this is a difficult task for AI to tackle.
“The comic's frame-shifting condition reflects the machine's difficulty in achieving it,” he said. “This heavy reliance on human cognitive insights — memory recall, linguistic abilities for word integration, and global knowledge results — often leads researchers to conclude that humor is an AI-complete problem.”
Addressing AI's threat to human jobs, OpenAI CTO Meera Murthy recently said that “some creative jobs will probably disappear, but they probably shouldn't have been there in the first place.” Given the current capabilities of technology, however, it seems comedians can breathe a sigh of relief.
Edited by Ryan Ozawa.