Estimation of Confidence in the Dialogue based on Eye Gaze and Head Movement Information
Abstract
In human-robot interaction, human mental states in dialogue have attracted attention to human-friendly robots that support educational use. Although estimating mental states using speech and visual information has been conducted, it is still challenging to estimate mental states more precisely in the educational scene. In this paper, we proposed a method to estimate human mental state based on participants’ eye gaze and head movement information. Estimated participants’ confidence levels in their answers to the miscellaneous knowledge question as a human mental state. The participants’ non-verbal information, such as eye gaze and head movements during dialog with a robot, were collected in our experiment using an eye-tracking device. Then we collect participants’ confidence levels and analyze the relationship between human mental state and non-verbal information. Furthermore, we also applied a machine learning technique to estimate participants’ confidence levels from extracted features of gaze and head movement information. As a result, the performance of a machine learning technique using gaze and head movements information achieved over 80 % accuracy in estimating confidence levels. Our research provides insight into developing a human-friendly robot considering human mental states in the dialogue.
Downloads
References
Noroozi F., Corneanu C. A., Kamińska D., et al., Survey on emotional body gesture recognition[J]. IEEE transactions on affective computing, 2018, 12(2): 505-523. DOI: https://doi.org/10.1109/TAFFC.2018.2874986
Defu, Z., Matsufuji, A., Sato-Shimokawara, E., Yamaguchi, T., Emotion Recognition based on speech data containing personal differences, International Symposium on Computational Intelligence and Industrial Applications, 2018.
Matsufuji, A., Shiozawa, T., Hsieh, W. F., Sato-Shimokawara, E., Yamaguchi, T., and Chen, L. -H., The analysis of nonverbal behavior for detecting awkward situation in communication. In 2017 Conference on Technologies and Applications of Artificial Intelligence (TAAI) (pp. 118-123). IEEE. DOI: https://doi.org/10.1109/TAAI.2017.12
Matsufuji, A., Sato-Shimokawara, E., Yamaguchi, T., A method for estimating speaker’s intention using human gestures and acoustic features in dialogue, Annual Conference of the Robotics Society of Japan, 2017.
Langton S. R. H., The mutual influence of gaze and head orientation in the analysis of social attention direction, Quarterly Journal of Experimental Psychology, A, 53(3), 825–845. DOI: https://doi.org/10.1080/027249800410562
Klyde David H., McCruer Duane T., Myers Thomas T., Unified Pilot- Induced Oscillation Theory, Volume I: PI0 Analysis with Linear and Nonlinear Effective Vehicle Characteristics, Including Rate Limiting, WL - TR - 96 - 3028. Air Force Research Laboratories, Wright - Patterson AFB OH, December 1995.
Karlsson P., Allsop A., Dee-Price B. J., et al. Eye-gaze control technology for children, adolescents and adults with cerebral palsy with significant physical disability: Findings from a systematic review[J]. Developmental neurorehabilitation, 2018, 21(8): 497-505. DOI: https://doi.org/10.1080/17518423.2017.1362057
Kasano, E., Muramatsu, S., Matsufuji, A., Sato-Shimokawara E., and Yamaguchi, T., Estimating Speaker’s Confidence in Dialogue Using Speech and Motion Information, Conference on System Integration of the Society of Instrument and Control Engineers, pp.729-732, 2018.
Jyotsna, C., Amudha J., Eye gaze as an indicator for stress level analysis in students. In 2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pp. 1588-1593, IEEE. DOI: https://doi.org/10.1109/ICACCI.2018.8554715
Emerson A., Cloude E. B., Azevedo R., et al. Multimodal learning analytics for game‐based learning[J]. British Journal of Educational Technology, 2020, 51(5): 1505-1526. DOI: https://doi.org/10.1111/bjet.12992
Graziano M. S. A., Guterstam A., Bio B. J., et al. Toward a standard model of consciousness: Reconciling the attention schema, global workspace, higher-order thought, and illusionist theories[J]. Cognitive Neuropsychology, 2020, 37(3-4): 155-172. DOI: https://doi.org/10.1080/02643294.2019.1670630
Ibrahim B., Ding L., Sequential and simultaneous synthesis problem solving: A comparison of students’ gaze transitions[J]. Physical Review Physics Education Research, 2021, 17(1): 010126. DOI: https://doi.org/10.1103/PhysRevPhysEducRes.17.010126
Kento Y., Ayano O., Hiroki F., Kensuke H., Olivier A., Koichi K., Confidence estimation based on gaze while answering English multiple-choice questions, Shingaku Giho, vol. 116, no. 461, PRMU2016-192, pp. 199-204, 2017.
Chihara T., Kobayashi F., Sakamoto J., Evaluation of mental workload during automobile driving using one-class support vector machine with eye movement data[J]. Applied ergonomics, 2020, 89: 103201. DOI: https://doi.org/10.1016/j.apergo.2020.103201
Stephenson L. J., Edwards S. G., Bayliss A. P., From gaze perception to social cognition: The shared-attention system[J]. Perspectives on Psychological Science, 2021, 16(3): 553-576. DOI: https://doi.org/10.1177/1745691620953773
Asteriadis S., Karpouzis K., Kollias S., The Importance of Eye Gaze and Head Pose to Estimating Levels of Attention. Third International Conference on Games & Virtual Worlds for Serious Applications. IEEE, 2011. DOI: https://doi.org/10.1109/VS-GAMES.2011.38
Torrente M., Guillem, Mobility for the severely disabled: a head-controlled wheelchair, 2017.
Sillero N., A Simple Spatial Method for Identifying Point Clusters by Neighbourhood Relationships[J]. Ecologies, 2021, 2(3): 305-312. DOI: https://doi.org/10.3390/ecologies2030017
Amrouche S., Gollan B., Ferscha A., et al. Activity segmentation and identification based on eye gaze features. Proceedings of the 11th Pervasive Technologies Related to Assistive Environments Conference. 2018: 75-82. DOI: https://doi.org/10.1145/3197768.3197775
Mannaru P., Balasingam B., Pattipati K., et al., On the use of hidden Markov models for gaze pattern modelling. Next-Generation Analyst IV. SPIE, 2016, 9851: 252-258. DOI: https://doi.org/10.1117/12.2224190
Wolf E., Martinez M., Roitberg A., et al., Estimating mental load in passive and active tasks from pupil and gaze changes using bayesian surprise. Proceedings of the Workshop on Modeling Cognitive Processes from Multimodal Data. 2018: 1-6. DOI: https://doi.org/10.1145/3279810.3279852
Sabrina A., Benedikt G., Alois F., Josef H., Activity Segmentation and Identification based on Eye Gaze Features, Proceedings of the 11th Pervasive Technologies Related to Assistive Environments Conference, June PP. 75–82, 2018
Ijuin K., Jokinen K. J., Kato T., et al. Eye-gaze in Social Robot Interactions Grounding of Information and Eye-gaze Patterns[C]. Proceedings of the Annual Conference of JSAI 33rd (2019). The Japanese Society for Artificial Intelligence, 2019: 3J3E402-3J3E402.
Browne J T., Wizard of oz prototyping for machine learning experiences[C]. Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 2019: 1-6. DOI: https://doi.org/10.1145/3290607.3312877
Matsufuji, A., Sato-Shimokawara, E., and Yamaguchi, T., Adaptive personalized multiple machine learning architecture for estimating human emotional states. Journal of Advanced Computational Intelligence and Intelligent Informatics, 2020, 24(5): 668-675. DOI: https://doi.org/10.20965/jaciii.2020.p0668
Copyright (c) 2022 EMITTER International Journal of Engineering Technology
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
The copyright to this article is transferred to Politeknik Elektronika Negeri Surabaya(PENS) if and when the article is accepted for publication. The undersigned hereby transfers any and all rights in and to the paper including without limitation all copyrights to PENS. The undersigned hereby represents and warrants that the paper is original and that he/she is the author of the paper, except for material that is clearly identified as to its original source, with permission notices from the copyright owners where required. The undersigned represents that he/she has the power and authority to make and execute this assignment. The copyright transfer form can be downloaded here .
The corresponding author signs for and accepts responsibility for releasing this material on behalf of any and all co-authors. This agreement is to be signed by at least one of the authors who have obtained the assent of the co-author(s) where applicable. After submission of this agreement signed by the corresponding author, changes of authorship or in the order of the authors listed will not be accepted.
Retained Rights/Terms and Conditions
- Authors retain all proprietary rights in any process, procedure, or article of manufacture described in the Work.
- Authors may reproduce or authorize others to reproduce the work or derivative works for the author’s personal use or company use, provided that the source and the copyright notice of Politeknik Elektronika Negeri Surabaya (PENS) publisher are indicated.
- Authors are allowed to use and reuse their articles under the same CC-BY-NC-SA license as third parties.
- Third-parties are allowed to share and adapt the publication work for all non-commercial purposes and if they remix, transform, or build upon the material, they must distribute under the same license as the original.
Plagiarism Check
To avoid plagiarism activities, the manuscript will be checked twice by the Editorial Board of the EMITTER International Journal of Engineering Technology (EMITTER Journal) using iThenticate Plagiarism Checker and the CrossCheck plagiarism screening service. The similarity score of a manuscript has should be less than 25%. The manuscript that plagiarizes another author’s work or author's own will be rejected by EMITTER Journal.
Authors are expected to comply with EMITTER Journal's plagiarism rules by downloading and signing the plagiarism declaration form here and resubmitting the form, along with the copyright transfer form via online submission.
