Enhancing monocular 3D semantic scene graph prediction via cross-modality knowledge distillation

초록

Three-dimensional semantic scene graph (3DSSG) prediction has achieved remarkable performance by integrating heterogeneous data sources, including RGB-D sensor-derived point clouds and multi-view images. However, existing 3DSSG research depends on specialized cameras, precise reconstruction workflows, and extensive manual labeling to construct 3DSSG pipelines based on RGB-D data. Visual simultaneous localization and mapping (SLAM)-based point clouds offer a practical alternative when combined with images and supported by automated labeling tools. In this study, we propose a knowledge distillation framework wherein a teacher model trained with RGB-D point clouds and images transfers learned representations to a student model that relies on visual-SLAM-generated point clouds and images through three targeted distillation modules. Extensive experiments on the 3RScan dataset validated the effectiveness of the proposed design across diverse scenarios, including distribution-based analyses and single-module ablations, offering a comprehensive evaluation of monocular-based semantic scene graph prediction without undermining 3D semantic understanding.

키워드