ST-MoRE: Spatio-Temporal MoE with Refinement for Action Segmentation

초록

This paper proposes ST-MoRE (Spatio-Temporal Mixture of Experts with Refinement Ensemble), a novel multimodal framework for precise temporal action segmentation. Traditional models often suffer from over-segmentation and boundary ambiguity due to their heavy reliance on visual or skeleton data without physical context. To address this, we integrate 3D kinematic data with physical contact information into a 16-channel multimodal tensor using a topological mapping strategy. The ST-MoRE architecture employs separable spatio-temporal attention blocks and a Mixture of Experts (MoE) mechanism to effectively handle the heterogeneous kinematic characteristics across five distinct action phases (Idle, Start, Interaction, End, and Return). Furthermore, a hybrid post-processing pipeline—combining Viterbi decoding and a 1D CNN-based boundary refiner—is introduced to ensure temporal logic and achieve frame-level boundary precision. Evaluated on a custom high-quality dataset of 130,000 frames, ST-MoRE achieved a test accuracy of 96.08%, significantly outperforming existing GCN, RNN, and Transformer-based models. With a real-time inference latency of 1.704 ms and robust generalization performance (a validation-to-test difference of only -0.16%p), ST-MoRE provides a reliable foundation for human-robot collaboration and digital twin systems.

키워드