Oscar Wilde once said that humor is the lowest form of wit, but the highest form of genius. Perhaps this is because of how difficult it is to use and understand. Sarcasm is notoriously difficult to convey through text — even in person, it can be easily misinterpreted. Subtle changes in tone that convey sarcasm often also confuse computer algorithms, limiting virtual assistants and content analysis tools.
Xiyuan Gao, Shekhar Nayak, and Matt Coler of the Speech Technology Lab in Groningen, Campus Fryslân developed a multimodal algorithm for better sarcasm recognition that evaluates multiple aspects of audio recordings for greater accuracy. Gao will present his work on Thursday, May 16 as part of a joint meeting of the Acoustical Society of America and the Canadian Acoustical Association, running May 13-17 at the Shaw Center in Ottawa, Ontario, Canada.
Traditional spoof detection algorithms often rely on a single parameter to generate their results, which is the main reason they often fall short. Gao, Naik, and Koller instead used two complementary methods — sentiment analysis using text and sentiment recognition using audio — for a more complete picture.
“We extracted acoustic parameters such as pitch, speaking rate, and energy from speech, then used automatic speech recognition to transcribe speech into text for sentiment analysis,” Gao said. “Next, we assigned emotional markers to each speech segment reflecting its emotional content. By integrating these multimodal signals into a machine learning algorithm, our approach combined emotional information with emotional information for a comprehensive analysis. takes advantage of the combined strengths of auditory and textual information.”
The team is optimistic about the performance of their algorithm, but they are already looking for ways to make it even better.
“There are many expressions and gestures that people use to highlight ironic elements in speech,” Gao said. “These need to be better integrated into our project. Also, we would like to add more languages and adopt developing spoof detection techniques.”
This approach can be used for more than dry wit identification. The researchers highlight that this technique can be widely applied in many fields.
“The development of sarcasm recognition technology can benefit other research domains using sentiment analysis and emotion recognition,” Gao said. “Traditionally, sentiment analysis has focused primarily on text and has been developed for applications such as online hate speech detection and customer opinion mining. Speech-Based Sentiment Recognition can be applied to healthcare with the help of AI. Mock recognition technology that applies a multimodal approach is insightful to these research domains.”