비언어적 신호 분석 기반의 사용자 만족도 평가 방안 연구

초록

Objective: This study explores the feasibility and responsiveness of a multimodal approach to estimate user satisfaction by analysing physiological and non-verbal behavioural signals collected in natural, unguided interaction conditions. Background: Traditional satisfaction assessments have primarily relied on self-report questionnaires; however, this method fails to capture subtle emotional shifts or realtime satisfaction changes occurring during interactions with AI systems. Furthermore, it suffers from limitations in fully reflecting users' actual emotional states due to the influence of recall bias and self-censorship. Method: This study utilized a multimodal dataset comprising EEG, ECG, PPG, GSR, and facial expression data collected during spontaneous voice-based human–AI interactions. After preprocessing and feature extraction, Support Vector Machine (SVM) models were independently trained under two classification settings. First, user satisfaction was classified into four discrete states defined by the joint combination of stability and favorability. Second, separate binary classification models were trained to predict stability and favorability individually, using the same dataset and crossvalidation protocol but optimized for each classification task. Results: In four-class satisfaction classification, EEG-based models achieved the highest average accuracy (68.8%), followed by facial expression-based models (67.4%) and physiological signal-based models (65.1%). In the independently trained binary classification analyses, substantially higher performance was observed across all modalities. EEG-based models achieved F1-scores of 86.6% for stability and 85.5% for favorability, while facial models achieved F1-scores of 86.2% and 84.4%, respectively. Physiological signals showed comparable performance for stability (85.7%) but lower F1-scores for favorability (81.7%) under natural interaction conditions. Conclusion: Our results demonstrate that user satisfaction can be estimated continuously in real-time from non-invasive signals, and offer real promise for the development of adaptive and emotionally responsive interaction systems. Application: The proposed approach contributes to the advancement of real-time multimodal satisfaction estimation models and provides useful insights for the design of emotionally adaptive human-AI interaction systems.

키워드