LLM-structured expert feedback for knowledge calibration of deep learning model to detect phishing URL

초록

Phishing attacks increasingly exhibit structural variability and intent obfuscation, challenging detection systems that rely on static features or limited data distributions. Deep neural models provide strong representation learning capabilities, yet often misclassify novel URLs that differ from the training distribution. In contrast, human decision-making incorporates external knowledge and structured reasoning beyond observed data. To reduce this gap, we propose a knowledge calibration method that incorporates symbolic expert knowledge from large language model into phishing detectors. An LLM translates expert feedback expressed in natural language into formal first-order logic rules. This translation is guided by five in-context learning strategies, enabling the structured incorporation of symbolic knowledge into the detection framework. These rules are used to generate phishing scores, which are then combined with neural outputs through a discrepancy-aware calibration objective. A triplet-based Transformer encoder is used to learn discriminative URL representations, and the joint loss function aligns neural predictions with symbolic expectations. Extensive experiments across four benchmark datasets show that our method achieves up to 98.85% accuracy and 97.42% recall, significantly outperforming baselines and ensemble models. Moreover, temporal drift evaluations under chronologically split zero-day settings show that the proposed model maintains an accuracy of 0.8153 even under the most severe drift scenario, outperforming the strongest baseline by over 9%p, demonstrating robust generalization of symbolic knowledge to unseen phishing attacks.

키워드