인간-로봇 상호작용에서의 작업 계획 수립을 위한연속 수어 인식

초록

In real-world human-robot interaction (HRI) environments, voice-based interfaces often face limitations due to noise, language barriers, and communication with users who have hearing or speech impairments. As a result, gesture-based interaction using sign language is gaining attention as an intuitive and non-verbal alternative. However, conventional systems rely on one-to-one mapping of static gestures to predefined commands or use gestures merely as a supplementary input to voice recognition, limiting contextual understanding and flexibility. This study proposes an end-to-end framework that captures continuous sign language gestures via camera, interprets the user’s intent using a large language model (LLM), and translates this intent into executable robotic actions. The proposed system consists of a bidirectional long short term memory-based sign language recognizer, an LLM-based intent reconstruction module, and the Scene describer that extracts scene elements, including objects and humans, to facilitate plan generation. Experimental evaluations confirm that the proposed framework provides a viable alternative to traditional language-based methods in settings where verbal communication is challenging.

키워드