The HLAB directed by H. Andrew Schwartz, which just moved to Vanderbilt’s new College of Connecting Computing, is running a key part of the annual worldwide artificial intelligence test, SemEval, which will evaluate how well current AI text systems understand human emotion.
Each year, more than a hundred teams from around the world participate in the popular event in the Natural Language Processing research community (the community creating the technology behind LLMs like ChatGPT). The task – “Predicting Variation in Emotional Valence and Arousal over Time from Ecological Essays” – was developed by Schwartz, his Ph.D. student Nikita Soni, and an interdisciplinary team and then chosen from dozens of entries. Evaluated over unique data recently collected by Schwartz’s Human Language Analysis Lab (HLAB), teams compete to develop AI or LLMs to see which approaches work best at this human-centered AI task.
“Requiring a deeper understanding of human psychology, it is a test that current state-of- the-art AI/LLMs are still far from perfect,” Schwartz said.
He added that this evaluation is important to AI development because it tests for two key limitations in how AI text systems understand human emotion: (1) treating emotion as a property of data rather than the people behind the data; and (2) predominantly ignoring time while emotion is time specific.
“This task introduces a rare opportunity to study emotions as they are lived—through self-reported, longitudinal experiences—rather than as they are merely perceived by others, paving the way for more personalized, ecologically valid, and emotionally intelligent NLP systems,” said Soni, a member of Schwartz’s lab and a doctoral student at Stony Brook University.
Schwartz’s overall research focuses on improving the state of the art in AI and investigating language, in all forms, as a window into the human condition – mental health, fundamental human traits, and behavioral motives.
On Oct. 31, his lab will host a “Get to Know HLAB Research” session where lab members will share brief talks about their work. Sign up here: Google Form.