A Study on Applying Large Language Models to Issue Classification

Abstract

Prompt-based large language models (LLMs) have demonstrated their ability to perform tasks with minimal or no additional training data. In the context of issue classification, researchers have actively explored the capabilities of LLMs in classifying issue reports. However, existing studies still face limitations in accuracy. This study replicates an LLM-based issue classification study using GPT-3.5 Turbo and explores variants, such as adopting different models like Llama 3.18 B and GPT-4o. Experimental results show that the classifier fine-tuned with GPT-3.5 Turbo still yields the same accuracy as shown in the original research and that the classifier fine-tuned with Llama 3.18 B(0.8004) yields an F1-score of 0.0535 lower than that of the classifier fine-tuned with GPT-3.5 Turbo (0.8467). On the other hand, the classifier with GPT-4o (0.8639) yields an average F1-score 0. 01 higher than that of the classifier fine-tuned with GPT-3.5 Turbo (0.8467). Additionally, the project-agnostic classifier fine-tuned with GPT-4o yields the highest F1-score of 0.8680. These findings contribute to advancing LLM-based issue classification by providing experimental insights into the accuracy of LLMs in this issue classification task. © 2025 IEEE.