20.500.12556/DKUM-95932 Human-led and artificial intelligence-automated critical appraisal of systematic reviews comparative evaluation Aim To evaluate and compare human-led and artificial intelligence-automated critical appraisal of evidence. Background Critical appraisal is essential in evidence-based practice, yet many nurses lack the skills to perform it. Large language models offer potential support, but their role in critical appraisal remains underexplored. Design We conducted a comparative study to evaluate the performance of five commonly used large language models versus two human reviewers in appraising four systematic reviews on interventions to reduce medication administration errors. Methods We compared large language models and two human reviewers in independently appraising four systematic reviews using the JBI Critical Appraisal Checklist. These models were Perplexity Sonar (Pro), Claude 3.7 Sonnet, Gemini 2.0 Flash, GPT-4.5 and Grok-2. All models received identical full texts and standardized prompts. Responses were analyzed descriptively and agreement was assessed using Cohen’s Kappa. Results Large language models showed full agreement with human reviewers on five of 11 JBI items. Most disagreements occurred in appraising search strategy, inclusion criteria and publication bias. The agreement between human reviewers and large language models ranged from slight to moderate. The highest level of agreement was observed with Claude (κ = 0.732), while the lowest level was observed with Gemini (κ = 0.394). Conclusion Large language models can support aspects of critical appraisal evidence but lack contextual reasoning and methodological insight required for complex judgments. While Claude 3.7 Sonnet aligned most closely with human reviewers, human oversight remains essential. Large language models should serve as adjuncts and not substitutes for evidence-based practice. artificial intelligence in healthcare multimodal large language models nursing evidence-based practice umetna inteligenca v zdravstvu multimodalni veliki jezikovni modeli zdravstvena nega z dokazi podprta praksa true false true Elsevier Angleški jezik Slovenski jezik https://www.sciencedirect.com/journal/nurse-education-in-practice © 2025 The Authors. Published by Elsevier Ltd. Znanstveno delo 2025-11-12 12:23:49 2025-11-12 12:23:50 2026-03-04 03:14:22 0000-00-00 00:00:00 2025 0 0 str. 1-6 št. članka 104614 Letn. 89 2025 0000-00-00 Zaloznikova Objavljeno apc 2025-05-30 2025-10-27 2025-10-28 616-083:004.8 256810243 10.1016/j.nepr.2025.104614 1873-5223 1-s2.0-S1471595325003713-main.pdf 1-s2.0-S1471595325003713-main.pdf 1 4F774DD00C5B15851F78DA6A546E2FDF 6c85cb25cf464531509b5434100df77ade2a2292e697b9fc7a3e07909f1eabc0 9a40407a-16c5-11f1-9bb8-00155d580e1a https://dk.um.si/Dokument.php?lang=slv&id=204161 https://www.sciencedirect.com/science/article/pii/S1471595325003713 1 10a8d066-bfba-11f0-97da-00155d000105 https://dk.um.si/Dokument.php?lang=slv&id=200572 Fakulteta za zdravstvene vede Fakulteta za elektrotehniko, računalništvo in informatiko 0 0 0