Emotion-aware multimodal lightweight framework for adaptive voice interaction on edge device

초록

Voice user interfaces (VUIs) have been a major focus of human-computer interaction research over the past two decades. Advances in voice and emotion recognition have significantly enhanced their capabilities. However, VUIs still struggle to produce responses that fit the user's emotions in various contexts. Meanwhile, the growing demand for on-device intelligence makes efficient operation on resource-constrained edge devices increasingly important. To address these issues, this study presents an emotion-aware VUI framework designed for edge devices. The framework includes a lightweight multimodal emotion recognition model that integrates electroencephalogram signals and facial expressions. Furthermore, to enable on-device efficiency, this paper applies filter pruning to remove redundant filters. It uses cross-layer ranking together with k-reciprocal nearest-filter selection. This improves cross-layer consistency and preserves salient representations while sharply reducing computation. Post-training quantization is then applied to further shrink model size and memory traffic, lowering latency without harming accuracy. Finally, a response-customization module dynamically refines system outputs based on the recognized emotions. The experimental implementation on actual edge devices demonstrates that the proposed system effectively adapts the interaction types to the emotions of the users. In addition, the proposed model achieves low latency and resource-efficient performance.

키워드