온디바이스 대형언어모델 엑사원 4.0 1.2B가 자동생성한 건축환경문헌 기반 Q&A 데이터셋의 품질평가

초록

This study explores the application of large language models (LLMs) in architectural engineering education by evaluating the quality ofquestion?answer (Q&A) pairs automatically generated from architectural environment literature using Exaone 4.0 1.2B, an on-device LLM. Atotal of 36 papers in the architectural environment domain were collected and preprocessed into text files, which were then used as input forzero-shot prompting. This process generated 1,913 Q&A pairs. Evaluation was conducted using ROUGE-L, containment, and cosine similarity(SBERT), along with a review of formal errors such as incomplete sentences, encoding corruption, typographical or spacing issues,meta-utterances, metadata exposure, residual citations, numerical or unit errors, and answer duplication (A=Q). The results show an averagecontainment score of 0.399 and an average cosine similarity of 0.420. In addition, 272 formal errors were identified, representing 14.2 percentof all generated pairs. These findings provide a baseline assessment of Exaone 4.0 1.2B’s performance in automatic Q&A generation for thearchitectural environment domain. Future research is expected to focus on reducing formal errors and improving semantic quality to enhanceeducational and practical applications.

키워드