Estimation of Confidence in the Dialogue based on Eye Gaze and Head Movement Information

  • Cui Dewen Computer Science Department, Tokyo Metropolitan University, Tokyo, Japan.
  • Matsufuji Akihiro Electric Works Company, Panasonic Corporation, Osaka, Japan.
  • Liu Yi Computer Science Department, Tokyo Metropolitan University, Tokyo, Japan.
  • Eri Sato- Shimokawa Computer Science Department, Tokyo Metropolitan University, Tokyo, Japan.
  • Toru Yamaguchi Computer Science Department, Tokyo Metropolitan University, Tokyo, Japan.
Keywords: human-robot interaction, gaze, confidence level, mental states, machine learning


In human-robot interaction, human mental states in dialogue have attracted attention to human-friendly robots that support educational use. Although estimating mental states using speech and visual information has been conducted, it is still challenging to estimate mental states more precisely in the educational scene. In this paper, we proposed a method to estimate human mental state based on participants’ eye gaze and head movement information. Estimated participants’ confidence levels in their answers to the miscellaneous knowledge question as a human mental state. The participants’ non-verbal information, such as eye gaze and head movements during dialog with a robot, were collected in our experiment using an eye-tracking device. Then we collect participants’ confidence levels and analyze the relationship between human mental state and non-verbal information. Furthermore, we also applied a machine learning technique to estimate participants’ confidence levels from extracted features of gaze and head movement information. As a result, the performance of a machine learning technique using gaze and head movements information achieved over 80 % accuracy in estimating confidence levels. Our research provides insight into developing a human-friendly robot considering human mental states in the dialogue.


Download data is not yet available.


Noroozi F., Corneanu C. A., Kamińska D., et al., Survey on emotional body gesture recognition[J]. IEEE transactions on affective computing, 2018, 12(2): 505-523. DOI:

Defu, Z., Matsufuji, A., Sato-Shimokawara, E., Yamaguchi, T., Emotion Recognition based on speech data containing personal differences, International Symposium on Computational Intelligence and Industrial Applications, 2018.

Matsufuji, A., Shiozawa, T., Hsieh, W. F., Sato-Shimokawara, E., Yamaguchi, T., and Chen, L. -H., The analysis of nonverbal behavior for detecting awkward situation in communication. In 2017 Conference on Technologies and Applications of Artificial Intelligence (TAAI) (pp. 118-123). IEEE. DOI:

Matsufuji, A., Sato-Shimokawara, E., Yamaguchi, T., A method for estimating speaker’s intention using human gestures and acoustic features in dialogue, Annual Conference of the Robotics Society of Japan, 2017.

Langton S. R. H., The mutual influence of gaze and head orientation in the analysis of social attention direction, Quarterly Journal of Experimental Psychology, A, 53(3), 825–845. DOI:

Klyde David H., McCruer Duane T., Myers Thomas T., Unified Pilot- Induced Oscillation Theory, Volume I: PI0 Analysis with Linear and Nonlinear Effective Vehicle Characteristics, Including Rate Limiting, WL - TR - 96 - 3028. Air Force Research Laboratories, Wright - Patterson AFB OH, December 1995.

Karlsson P., Allsop A., Dee-Price B. J., et al. Eye-gaze control technology for children, adolescents and adults with cerebral palsy with significant physical disability: Findings from a systematic review[J]. Developmental neurorehabilitation, 2018, 21(8): 497-505. DOI:

Kasano, E., Muramatsu, S., Matsufuji, A., Sato-Shimokawara E., and Yamaguchi, T., Estimating Speaker’s Confidence in Dialogue Using Speech and Motion Information, Conference on System Integration of the Society of Instrument and Control Engineers, pp.729-732, 2018.

Jyotsna, C., Amudha J., Eye gaze as an indicator for stress level analysis in students. In 2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pp. 1588-1593, IEEE. DOI:

Emerson A., Cloude E. B., Azevedo R., et al. Multimodal learning analytics for game‐based learning[J]. British Journal of Educational Technology, 2020, 51(5): 1505-1526. DOI:

Graziano M. S. A., Guterstam A., Bio B. J., et al. Toward a standard model of consciousness: Reconciling the attention schema, global workspace, higher-order thought, and illusionist theories[J]. Cognitive Neuropsychology, 2020, 37(3-4): 155-172. DOI:

Ibrahim B., Ding L., Sequential and simultaneous synthesis problem solving: A comparison of students’ gaze transitions[J]. Physical Review Physics Education Research, 2021, 17(1): 010126. DOI:

Kento Y., Ayano O., Hiroki F., Kensuke H., Olivier A., Koichi K., Confidence estimation based on gaze while answering English multiple-choice questions, Shingaku Giho, vol. 116, no. 461, PRMU2016-192, pp. 199-204, 2017.

Chihara T., Kobayashi F., Sakamoto J., Evaluation of mental workload during automobile driving using one-class support vector machine with eye movement data[J]. Applied ergonomics, 2020, 89: 103201. DOI:

Stephenson L. J., Edwards S. G., Bayliss A. P., From gaze perception to social cognition: The shared-attention system[J]. Perspectives on Psychological Science, 2021, 16(3): 553-576. DOI:

Asteriadis S., Karpouzis K., Kollias S., The Importance of Eye Gaze and Head Pose to Estimating Levels of Attention. Third International Conference on Games & Virtual Worlds for Serious Applications. IEEE, 2011. DOI:

Torrente M., Guillem, Mobility for the severely disabled: a head-controlled wheelchair, 2017.

Sillero N., A Simple Spatial Method for Identifying Point Clusters by Neighbourhood Relationships[J]. Ecologies, 2021, 2(3): 305-312. DOI:

Amrouche S., Gollan B., Ferscha A., et al. Activity segmentation and identification based on eye gaze features. Proceedings of the 11th Pervasive Technologies Related to Assistive Environments Conference. 2018: 75-82. DOI:

Mannaru P., Balasingam B., Pattipati K., et al., On the use of hidden Markov models for gaze pattern modelling. Next-Generation Analyst IV. SPIE, 2016, 9851: 252-258. DOI:

Wolf E., Martinez M., Roitberg A., et al., Estimating mental load in passive and active tasks from pupil and gaze changes using bayesian surprise. Proceedings of the Workshop on Modeling Cognitive Processes from Multimodal Data. 2018: 1-6. DOI:

Sabrina A., Benedikt G., Alois F., Josef H., Activity Segmentation and Identification based on Eye Gaze Features, Proceedings of the 11th Pervasive Technologies Related to Assistive Environments Conference, June PP. 75–82, 2018

Ijuin K., Jokinen K. J., Kato T., et al. Eye-gaze in Social Robot Interactions Grounding of Information and Eye-gaze Patterns[C]. Proceedings of the Annual Conference of JSAI 33rd (2019). The Japanese Society for Artificial Intelligence, 2019: 3J3E402-3J3E402.

Browne J T., Wizard of oz prototyping for machine learning experiences[C]. Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 2019: 1-6. DOI:

Matsufuji, A., Sato-Shimokawara, E., and Yamaguchi, T., Adaptive personalized multiple machine learning architecture for estimating human emotional states. Journal of Advanced Computational Intelligence and Intelligent Informatics, 2020, 24(5): 668-675. DOI:

How to Cite
Dewen, C., Akihiro, M., Yi, L., Shimokawa, E. S.-, & Yamaguchi, T. (2022). Estimation of Confidence in the Dialogue based on Eye Gaze and Head Movement Information. EMITTER International Journal of Engineering Technology, 10(2), 338-350.