Application of multimodal data fusion and explainable AI for classifying water stress in sweet potatoes

Abstract

Sweet potato (Ipomoea batatas L.) exhibits strong resilience in nutrient-poor soils and contains high levels of dietary fiber and antioxidant compounds. It also is highly tolerant to water stress, which has also contributed to its global distribution, particularly in regions prone to climatic variability. However, frequent abnormal climatic events have recently caused declines in both the quality and yield of sweet potatoes. To address this, machine learning (ML) and deep learning (DL) models based on a Vision Transformer-Convolutional Neural Network (ViT-CNN) were developed to classify water stress levels in sweet potato. RGB-thermal imagery captured from low-altitude platforms and various growth indicators were used to develop the classifier. The K-Nearest Neighbors (KNN) model outperformed other ML models in classifying water stress levels at all growth stages. The DL model simplified the original five-level water stress classification into three levels. This enhanced its sensitivity to extreme stress conditions, improve model performance, and increased its applicability to practical agricultural management strategies. To enhance practical applicability under open-field conditions, several environmental variables were newly defined to calculate the crop water stress index (CWSI). Furthermore, an integrated system was developed using gradient-weighted class activation mapping (Grad-CAM), explainable artificial intelligence (XAI), and a graphical user interface (GUI) to support intuitive interpretation and actionable decision-making. The system will be expanded into an online and fixed-camera platform to enhance its applicability to smart farming in diverse field crops.