DEformer: Dual Embedded Transformer for Multivariate Time Series Forecasting

Abstract

Deep learning models have significantly addressed the challenges of multivariate time series forecasting. Recently, Transformer-based models which have primarily focused on either temporal or inter-variate (spatial) dependencies have demonstrated exceptional performance. These models decide whether to embed multivariate time series data temporally or spatially. Hence, we propose the dual embedded transformer (DEformer) which simultaneously considers both temporal and spatial dependencies. Our model enables capturing these dependencies independently through two distinct encoders. The temporal encoder, which processes the entire time series of each variate, is designed to learn temporal dependency. Conversely, the spatial encoder, which processes the multivariate data at each time step, is optimized to capture spatial dependency. Both encoders share an identical architecture, comprising an attention mechanism to model correlations and a feed-forward network to enhance feature representation. Through empirical studies on challenging real-world datasets, we not only demonstrate that our method can outperforms state-of-the-art approaches, but also prove the performance of independently embedding both dependencies through ablation study.