| Title: | Human-led and artificial intelligence-automated critical appraisal of systematic reviews : comparative evaluation |
|---|
| Authors: | ID Gosak, Lucija (Author)
ID Štiglic, Gregor (Author)
ID Tam, Wilson (Author)
ID Vrbnjak, Dominika (Author) |
|---|
| Files: |  1-s2.0-S1471595325003713-main.pdf (651,99 KB)
MD5: 4F774DD00C5B15851F78DA6A546E2FDF
 https://www.sciencedirect.com/science/article/pii/S1471595325003713
|
|---|
| Language: | English |
|---|
| Work type: | Scientific work |
|---|
| Typology: | 1.01 - Original Scientific Article |
|---|
| Organization: | FZV - Faculty of Health Sciences
FERI - Faculty of Electrical Engineering and Computer Science
|
|---|
| Abstract: | Aim To evaluate and compare human-led and artificial intelligence-automated critical appraisal of evidence. Background Critical appraisal is essential in evidence-based practice, yet many nurses lack the skills to perform it. Large language models offer potential support, but their role in critical appraisal remains underexplored. Design We conducted a comparative study to evaluate the performance of five commonly used large language models versus two human reviewers in appraising four systematic reviews on interventions to reduce medication administration errors. Methods We compared large language models and two human reviewers in independently appraising four systematic reviews using the JBI Critical Appraisal Checklist. These models were Perplexity Sonar (Pro), Claude 3.7 Sonnet, Gemini 2.0 Flash, GPT-4.5 and Grok-2. All models received identical full texts and standardized prompts. Responses were analyzed descriptively and agreement was assessed using Cohen’s Kappa. Results Large language models showed full agreement with human reviewers on five of 11 JBI items. Most disagreements occurred in appraising search strategy, inclusion criteria and publication bias. The agreement between human reviewers and large language models ranged from slight to moderate. The highest level of agreement was observed with Claude (κ = 0.732), while the lowest level was observed with Gemini (κ = 0.394). Conclusion Large language models can support aspects of critical appraisal evidence but lack contextual reasoning and methodological insight required for complex judgments. While Claude 3.7 Sonnet aligned most closely with human reviewers, human oversight remains essential. Large language models should serve as adjuncts and not substitutes for evidence-based practice. |
|---|
| Keywords: | artificial intelligence in healthcare, multimodal large language models, nursing, evidence-based practice |
|---|
| Publication status: | Published |
|---|
| Publication version: | Version of Record |
|---|
| Submitted for review: | 30.05.2025 |
|---|
| Article acceptance date: | 27.10.2025 |
|---|
| Publication date: | 28.10.2025 |
|---|
| Publisher: | Elsevier |
|---|
| Year of publishing: | 2025 |
|---|
| Number of pages: | str. 1-6 |
|---|
| Numbering: | Letn. 89, št. članka 104614 |
|---|
| PID: | 20.500.12556/DKUM-95932  |
|---|
| UDC: | 616-083:004.8 |
|---|
| ISSN on article: | 1873-5223 |
|---|
| COBISS.SI-ID: | 256810243 |
|---|
| DOI: | 10.1016/j.nepr.2025.104614 |
|---|
| Copyright: | © 2025 The Authors. Published by Elsevier Ltd. |
|---|
| Publication date in DKUM: | 12.11.2025 |
|---|
| Views: | 201 |
|---|
| Downloads: | 9 |
|---|
| Metadata: |    |
|---|
| Categories: | Misc.
|
|---|
|
:
|
Copy citation |
|---|
| | | | Average score: | (0 votes) |
|---|
| Your score: | Voting is allowed only for logged in users. |
|---|
| Share: |  |
|---|
Hover the mouse pointer over a document title to show the abstract or click
on the title to get all document metadata. |
