20.500.12556/DKUM-93644 Evaluating Proprietary and Open-Weight Large Language Models as Universal Decimal Classification Recommender Systems Manual assignment of Universal Decimal Classification (UDC) codes is time-consuming and inconsistent as digital library collections expand. This study evaluates 17 large language models (LLMs) as UDC classification recommender systems, including ChatGPT variants (GPT-3.5, GPT-4o, and o1-mini), Claude models (3-Haiku and 3.5-Haiku), Gemini series (1.0-Pro, 1.5-Flash, and 2.0-Flash), and Llama, Gemma, Mixtral, and DeepSeek architectures. Models were evaluated zero-shot on 900 English and Slovenian academic theses manually classified by professional librarians. Classification prompts utilized the RISEN framework, with evaluation using Levenshtein and Jaro–Winkler similarity, and a novel adjusted hierarchical similarity metric capturing UDC’s faceted structure. Proprietary systems consistently outperformed open-weight alternatives by 5–10% across metrics. GPT-4o achieved the highest hierarchical alignment, while open-weight models showed progressive improvements but remained behind commercial systems. Performance was comparable between languages, demonstrating robust multilingual capabilities. The results indicate that LLM-powered recommender systems can enhance library classification workflows. Future research incorporating fine-tuning and retrieval-augmented approaches may enable fully automated, high-precision UDC assignment systems. universal decimal classification large language models conversational systems recommender systems prompt engineering zero-shot classification hierarchical similarity univerzalna decimalna klasifikacija jezikovni modeli hierarhična podobnost priporočljivi sistemi true false true MDPI AG Angleški jezik Slovenski jezik https://www.mdpi.com/journal/applsci © 2025 by the authors Članek v reviji 2025-07-09 12:11:32 2025-07-21 14:34:46 2025-07-22 03:10:09 0000-00-00 00:00:00 2025 0 0 7666-7689 št. 14 let. 15 2025 0000-00-00 Zaloznikova Objavljeno apc 2025-06-20 2025-07-07 2025-07-08 004.8 2076-3417 243245571 10.3390/app15147666 2076-3417 applsci-15-07666-v2.pdf applsci-15-07666-v2.pdf 1 72ACD70840709FDBA57806D0FE92EBAF 509c865e567d4ce8feb14b93a6ac1c4b570fb4023d4d312578e5986f5ad432ed 22cab779-5cad-11f0-8f2d-00155d000105 https://dk.um.si/Dokument.php?lang=slv&id=193410 https://www.mdpi.com/2076-3417/15/14/7666/pdf 1 76f2307a-5cad-11f0-8f2d-00155d000105 https://dk.um.si/Dokument.php?lang=slv&id=193411 Fakulteta za elektrotehniko, računalništvo in informatiko 0 0 0