Paper
Reading the Mood Behind Words: Integrating Prosody-Derived Emotional Context into Socially Responsive VR Agents
Authors
SangYeop Jeong, Yeongseo Na, Seung Gyu Jeong, Jin-Woo Jeong, Seong-Eun Kim
Abstract
In VR interactions with embodied conversational agents, users' emotional intent is often conveyed more by how something is said than by what is said. However, most VR agent pipelines rely on speech-to-text processing, discarding prosodic cues and often producing emotionally incongruent responses despite correct semantics. We propose an emotion-context-aware VR interaction pipeline that treats vocal emotion as explicit dialogue context in an LLM-based conversational agent. A real-time speech emotion recognition model infers users' emotional states from prosody, and the resulting emotion labels are injected into the agent's dialogue context to shape response tone and style. Results from a within-subjects VR study (N=30) show significant improvements in dialogue quality, naturalness, engagement, rapport, and human-likeness, with 93.3% of participants preferring the emotion-aware agent.
Metadata
Related papers
Gen-Searcher: Reinforcing Agentic Search for Image Generation
Kaituo Feng, Manyuan Zhang, Shuang Chen, Yunlong Lin, Kaixuan Fan, Yilei Jian... • 2026-03-30
On-the-fly Repulsion in the Contextual Space for Rich Diversity in Diffusion Transformers
Omer Dahary, Benaya Koren, Daniel Garibi, Daniel Cohen-Or • 2026-03-30
Graphilosophy: Graph-Based Digital Humanities Computing with The Four Books
Minh-Thu Do, Quynh-Chau Le-Tran, Duc-Duy Nguyen-Mai, Thien-Trang Nguyen, Khan... • 2026-03-30
ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining
Anuj Diwan, Eunsol Choi, David Harwath • 2026-03-30
RAD-AI: Rethinking Architecture Documentation for AI-Augmented Ecosystems
Oliver Aleksander Larsen, Mahyar T. Moghaddam • 2026-03-30
Raw Data (Debug)
{
"raw_xml": "<entry>\n <id>http://arxiv.org/abs/2603.09324v1</id>\n <title>Reading the Mood Behind Words: Integrating Prosody-Derived Emotional Context into Socially Responsive VR Agents</title>\n <updated>2026-03-10T07:58:06Z</updated>\n <link href='https://arxiv.org/abs/2603.09324v1' rel='alternate' type='text/html'/>\n <link href='https://arxiv.org/pdf/2603.09324v1' rel='related' title='pdf' type='application/pdf'/>\n <summary>In VR interactions with embodied conversational agents, users' emotional intent is often conveyed more by how something is said than by what is said. However, most VR agent pipelines rely on speech-to-text processing, discarding prosodic cues and often producing emotionally incongruent responses despite correct semantics. We propose an emotion-context-aware VR interaction pipeline that treats vocal emotion as explicit dialogue context in an LLM-based conversational agent. A real-time speech emotion recognition model infers users' emotional states from prosody, and the resulting emotion labels are injected into the agent's dialogue context to shape response tone and style. Results from a within-subjects VR study (N=30) show significant improvements in dialogue quality, naturalness, engagement, rapport, and human-likeness, with 93.3% of participants preferring the emotion-aware agent.</summary>\n <category scheme='http://arxiv.org/schemas/atom' term='cs.HC'/>\n <category scheme='http://arxiv.org/schemas/atom' term='cs.AI'/>\n <published>2026-03-10T07:58:06Z</published>\n <arxiv:comment>12 pages, 4 figures, Accepted to CHI EA 2026 (Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems)</arxiv:comment>\n <arxiv:primary_category term='cs.HC'/>\n <arxiv:journal_ref>Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems (CHI EA '26), 2026, 1-12</arxiv:journal_ref>\n <author>\n <name>SangYeop Jeong</name>\n </author>\n <author>\n <name>Yeongseo Na</name>\n </author>\n <author>\n <name>Seung Gyu Jeong</name>\n </author>\n <author>\n <name>Jin-Woo Jeong</name>\n </author>\n <author>\n <name>Seong-Eun Kim</name>\n </author>\n <arxiv:doi>10.1145/3772363.3798643</arxiv:doi>\n <link href='https://doi.org/10.1145/3772363.3798643' rel='related' title='doi'/>\n </entry>"
}