Research articleOpen accessVol. 14, Issue 10, October 2026
- Received
- 4 September 2026
- Accepted
- 18 September 2026
- Published
- 6 October 2026
- Licence
- CC BY 4.0
Abstract
Social robots are increasingly deployed in education, healthcare, and service settings, yet most still respond to what users say without regard for how they feel. This paper presents an emotion-driven interaction framework that infers user sentiment from speech and uses it to adapt a robot’s dialogue and nonverbal behaviour in real time. The framework combines a fine-tuned BERT text encoder with a prosodic audio encoder through cross-modal attention. It maps the predicted sentiment and its confidence to one of four response strategies. The classifier was trained and evaluated on 5,000 annotated utterances from 80 speakers, split by speaker. On the held-out test set of 1,000 utterances, the multimodal model reached 84.2% accuracy and a macro-F1 of 0.843. It outperformed a text-only BERT model (79.1%; McNemar χ²(1) = 15.34, p < 0.001) and an early-fusion baseline (81.4%). A within-subjects study with 48 participants compared the emotion-adaptive robot with a non-adaptive control. The adaptive robot received higher ratings of likeability (d = 0.76), perceived intelligence (d = 0.57), animacy (d = 0.61), interaction satisfaction (d = 0.82), and trust (d = 0.55), and it produced fewer conversational breakdowns (d = −0.68). Perceived safety did not differ. The results indicate that sentiment-aware adaptation improves the quality of human-robot interaction with modest computational cost.
Keywords: human-robot interaction, sentiment analysis, affective computing, multimodal fusion, social robots, emotion-aware dialogue
References
[1] R. W. Picard, Affective Computing. Cambridge, MA, USA: MIT Press, 1997.
[2] C. Breazeal, “Emotion and sociable humanoid robots,” Int. J. Human-Computer Studies, vol. 59, no. 1–2, pp. 119–155, 2003.
[3] T. Fong, I. Nourbakhsh, and K. Dautenhahn, “A survey of socially interactive robots,” Robotics and Autonomous Systems, vol. 42, no. 3–4, pp. 143–166, 2003.
[4] B. Pang and L. Lee, “Opinion mining and sentiment analysis,” Foundations and Trends in Information Retrieval, vol. 2, no. 1–2, pp. 1–135, 2008.
[5] R. Socher et al., “Recursive deep models for semantic compositionality over a sentiment treebank,” in Proc. EMNLP, Seattle, WA, USA, 2013, pp. 1631–1642.
[6] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. NAACL-HLT, Minneapolis, MN, USA, 2019, pp. 4171–4186.
[7] C. Hutto and E. Gilbert, “VADER: A parsimonious rule-based model for sentiment analysis of social media text,” in Proc. Int. AAAI Conf. Web and Social Media (ICWSM), vol. 8, no. 1, 2014, pp. 216–225.
[8] C. Busso et al., “IEMOCAP: Interactive emotional dyadic motion capture database,” Language Resources and Evaluation, vol. 42, no. 4, pp. 335–359, 2008.
[9] A. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, “Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph,” in Proc. ACL, Melbourne, Australia, 2018, pp. 2236–2246.
[10] S. Poria, E. Cambria, R. Bajpai, and A. Hussain, “A review of affective computing: From unimodal analysis to multimodal fusion,” Information Fusion, vol. 37, pp. 98–125, 2017.
[11] P. Ekman, “An argument for basic emotions,” Cognition and Emotion, vol. 6, no. 3–4, pp. 169–200, 1992.
[12] T. Kanda, T. Hirano, D. Eaton, and H. Ishiguro, “Interactive robots as social partners and peer tutors for children: A field trial,” Human-Computer Interaction, vol. 19, no. 1–2, pp. 61–84, 2004.
[13] I. Leite, G. Castellano, A. Pereira, C. Martinho, and A. Paiva, “Empathic robots for long-term interaction,” Int. J. Social Robotics, vol. 6, no. 3, pp. 329–341, 2014.
[14] C. Bartneck, E. Croft, D. Kulić, and S. Zoghbi, “Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots,” Int. J. Social Robotics, vol. 1, no. 1, pp. 71–81, 2009.
[15] A. Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017, pp. 5998–6008.
[16] T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 41, no. 2, pp. 423–443, 2019.
How to cite
Dr. Bimal Kumar Kalita (2026). Emotion-Driven AI for Enhancing Human-Robot Interaction through Sentiment Analysis. International Journal of Research in Modern Engineering & Emerging Technology, 14(10), 6-11.
Licence
This article is open access, published under the CC BY 4.0 licence. Copyright remains with the authors.