| Title: | PICOT questions and search strategies formulation: a novel approach using artificial intelligence automation |
|---|
| Authors: | ID Gosak, Lucija (Author)
ID Štiglic, Gregor (Author)
ID Pruinelli, Lisiane (Author)
ID Vrbnjak, Dominika (Author) |
|---|
| Files: |  J_of_Nursing_Scholarship_-_2024_-_Gosak_-_PICOT_questions_and_search_strategies_formulation__A_novel_approach_using.pdf (527,46 KB)
MD5: 8CAAD71B117D326B198DB5D794A9709A
 https://sigmapubs.onlinelibrary.wiley.com/doi/epdf/10.1111/jnu.13036
|
|---|
| Language: | English |
|---|
| Work type: | Scientific work |
|---|
| Typology: | 1.01 - Original Scientific Article |
|---|
| Organization: | FZV - Faculty of Health Sciences
FERI - Faculty of Electrical Engineering and Computer Science
|
|---|
| Abstract: | Aim
The aim of this study was to evaluate and compare artificial intelligence (AI)-based large language models (LLMs) (ChatGPT-3.5, Bing, and Bard) with human-based formulations in generating relevant clinical queries, using comprehensive methodological evaluations.
Methods
To interact with the major LLMs ChatGPT-3.5, Bing Chat, and Google Bard, scripts and prompts were designed to formulate PICOT (population, intervention, comparison, outcome, time) clinical questions and search strategies. Quality of the LLMs responses was assessed using a descriptive approach and independent assessment by two researchers. To determine the number of hits, PubMed, Web of Science, Cochrane Library, and CINAHL Ultimate search results were imported separately, without search restrictions, with the search strings generated by the three LLMs and an additional one by the expert. Hits from one of the scenarios were also exported for relevance evaluation. The use of a single scenario was chosen to provide a focused analysis. Cronbach's alpha and intraclass correlation coefficient (ICC) were also calculated.
Results
In five different scenarios, ChatGPT-3.5 generated 11,859 hits, Bing 1,376,854, Bard 16,583, and an expert 5919 hits. We then used the first scenario to assess the relevance of the obtained results. The human expert search approach resulted in 65.22% (56/105) relevant articles. Bing was the most accurate AI-based LLM with 70.79% (63/89), followed by ChatGPT-3.5 with 21.05% (12/45), and Bard with 13.29% (42/316) relevant hits. Based on the assessment of two evaluators, ChatGPT-3.5 received the highest score (M = 48.50; SD = 0.71). Results showed a high level of agreement between the two evaluators. Although ChatGPT-3.5 showed a lower percentage of relevant hits compared to Bing, this reflects the nuanced evaluation criteria, where the subjective evaluation prioritized contextual accuracy and quality over mere relevance.
Conclusion
This study provides valuable insights into the ability of LLMs to formulate PICOT clinical questions and search strategies. AI-based LLMs, such as ChatGPT-3.5, demonstrate significant potential for augmenting clinical workflows, improving clinical query development, and supporting search strategies. However, the findings also highlight limitations that necessitate further refinement and continued human oversight.
Clinical Relevance
AI could assist nurses in formulating PICOT clinical questions and search strategies. AI-based LLMs offer valuable support to healthcare professionals by improving the structure of clinical questions and enhancing search strategies, thereby significantly increasing the efficiency of information retrieval. |
|---|
| Keywords: | PICOT question, search strategies, artificial intelligence |
|---|
| Publication status: | Published |
|---|
| Publication version: | Version of Record |
|---|
| Submitted for review: | 31.01.2024 |
|---|
| Article acceptance date: | 07.11.2024 |
|---|
| Publication date: | 27.01.2025 |
|---|
| Year of publishing: | 2025 |
|---|
| Number of pages: | str. 5-16 |
|---|
| Numbering: | Letn. 57, št. 1 |
|---|
| PID: | 20.500.12556/DKUM-93712  |
|---|
| UDC: | 616-083:004.8 |
|---|
| ISSN on article: | 1547-5069 |
|---|
| COBISS.SI-ID: | 222504195 |
|---|
| DOI: | 10.1111/jnu.13036 |
|---|
| Copyright: | © 2024 The Author(s) |
|---|
| Publication date in DKUM: | 21.07.2025 |
|---|
| Views: | 232 |
|---|
| Downloads: | 11 |
|---|
| Metadata: |    |
|---|
| Categories: | Misc.
|
|---|
|
:
|
Copy citation |
|---|
| | | | Average score: | (0 votes) |
|---|
| Your score: | Voting is allowed only for logged in users. |
|---|
| Share: |  |
|---|
Hover the mouse pointer over a document title to show the abstract or click
on the title to get all document metadata. |
