| | SLO | ENG | Cookies and privacy

Bigger font | Smaller font

Show document Help

Title:Exploration of augmentation strategies in multi-modal retrieval-augmented generation for the biomedical domain : a case study evaluating question answering in glycobiology
Authors:ID Kocbek, Primož (Author)
ID Frkatović-Hodžić, Azra (Author)
ID Lalić, Dora (Author)
ID Hui, Vivian (Author)
ID Lauc, Gordan (Author)
ID Štiglic, Gregor (Author)
ID IEEE (Copyright holder)
Files:.pdf IEEE_BigData25_MMAI_Kocbek_etal.pdf (232,82 KB)
MD5: 33355984EC6223B01395A2EA13F9D69C
 
URL https://ieeexplore.ieee.org/document/11401037
 
Language:English
Work type:Scientific work
Typology:1.08 - Published Scientific Conference Contribution
Organization:FZV - Faculty of Health Sciences
Abstract:Multi-modal retrieval-augmented generation (MMRAG) promises grounded biomedical QA, but it is unclear when to (i) convert figures/tables into text versus (ii) use optical character recognition (OCR)-free visual retrieval that returns page images and leaves interpretation to the generator. We study this trade-off in glycobiology, a visually dense domain. We built a benchmark of 120 multiple-choice questions (MCQs) from 25 papers, stratified by retrieval difficulty (easy text, medium figures/tables, hard cross-evidence). We implemented four augmentations—None, Text RAG, Multi-modal conversion, and late-interaction visual retrieval (ColPali)—using Docling parsing and Qdrant indexing. We evaluated mid-size opensource and frontier proprietary models (e.g., Gemma-3-27BIT, GPT-4o family). Additional testing used the GPT-5 family and multiple visual retrievers (ColPali/ColQwen/ColFlor). Accuracy with Agresti–Coull 95% confidence intervals (CIs) was computed over 5 runs per configuration. With Gemma-3-27BIT, Text and Multi-modal augmentation outperformed OCR-free retrieval (0.722-0.740 vs. 0.510 average accuracy). With GPT-4o, Multi-modal achieved 0.808, with Text 0.782 and ColPali 0.745 close behind; within-model differences were small. In follow-on experiments with the GPT-5 family, the best results with ColPali and ColFlor improved by 2% to 0.828 in both cases. In general across the GPT-5 family, ColPali, ColQwen, and ColFlor were statistically indistinguishable; ColFlor matched ColPali while being far smaller. GPT-5-nano trailed larger GPT-5 variants by roughly 8-10%. Pipeline choice is capacity-dependent: converting visuals to text lowers the reader burden and is more reliable for mid-size models, whereas OCR-free visual retrieval becomes competitive under frontier models. Among retrievers, ColFlor offers parity with heavier options at a smaller footprint, making it an efficient default when strong generators are available.
Keywords:multimodal retrieval-augmentation generation, evaluation, large language models
Publication status:Submitted to the publisher
Year of publishing:2026
Number of pages:Str. 5156-5165
PID:20.500.12556/DKUM-97620 New window
UDC:004.89+37.011.22
COBISS.SI-ID:272777987 New window
DOI:10.1109/BigData66926.2025.11401037 New window
Publication date in DKUM:25.03.2026
Views:222
Downloads:11
Metadata:XML DC-XML DC-RDF
Categories:Misc.
:
Copy citation
  
Average score:(0 votes)
Your score:Voting is allowed only for logged in users.
Share:Bookmark and Share



Hover the mouse pointer over a document title to show the abstract or click on the title to get all document metadata.

Record is a part of a monograph

Title:Proceedings : 2025 IEEE International Conference on Big Data (BigData)
Place of publishing:[Piscataway]
Publisher:IEEE
ISBN:979-8-3315-9447-3
COBISS.SI-ID:272757763 New window

Document is financed by a project

Funder:EC - European Commission
Funding programme:HE
Project number:101159018
Name:Synergy for Healthy Longevity
Acronym:SynHealth

Funder:EC - European Commission
Project number:101101903
Name:Slovenian AI Factory
Acronym:SLAIF

Funder:ARIS - Slovenian Research and Innovation Agency
Project number:GC-0001
Name:Umetna inteligenca za znanost

Licences

License:Other
Description:COPYRIGHT TRANSFER The undersigned hereby assigns to The Institute of Electrical and Electronics Engineers, Incorporated (the "IEEE") all rights under copyright that may exist in and to: (a) the Work, including any revised or expanded derivative works submitted to the IEEE by the undersigned based on the Work; and (b) any associated written or multimedia components or other enhancements accompanying the Work. GENERAL TERMS The undersigned represents that he/she has the power and authority to make and execute this form.1. The undersigned agrees to indemnify and hold harmless the IEEE from any damage or expense that may arise in the event of a breach of any of the warranties set forth above. 2. The undersigned agrees that publication with IEEE is subject to the policies and procedures of the IEEE PSPB Operations Manual.3. In the event the above work is not accepted and published by the IEEE or is withdrawn by the author(s) before acceptance by the IEEE, the foregoing copyright transfer shall be null and void. In this case, IEEE will retain a copy of the manuscript for internal administrative/record-keeping purposes. 4. For jointly authored Works, all joint authors should sign, or one of the authors should sign as authorized agent for the others.5. The author hereby warrants that the Work and Presentation (collectively, the "Materials") are original and that he/she is the author of the Materials. To the extent the Materials incorporate text passages, figures, data or other material from the works of others, the author has obtained any necessary permissions. Where necessary, the author has obtained all third party permissions and consents to grant the license above and has provided copies of such permissions and consents to IEEE 6. You have indicated that you DO wish to have video/audio recordings made of your conference presentation under terms and conditions set forth in "Consent and Release." CONSENT AND RELEASE ln the event the author makes a presentation based upon the Work at a conference hosted or sponsored in whole or in part by the IEEE, the author, in consideration for his/her participation in the conference, hereby grants the IEEE the unlimited, worldwide, irrevocable permission to use, distribute, publish, license, exhibit, record, digitize, broadcast, reproduce and archive, in any format or medium, whether now known or hereafter developed: (a) his/her presentation and comments at the conference; (b) any written materials or multimedia files used in connection with his/her presentation; and (c) any recorded interviews of him/her (collectively, the "Presentation"). The permission granted includes the transcription and reproduction of the Presentation for inclusion in products sold or distributed by IEEE and live or recorded broadcast of the Presentation during or after the conference. In connection with the permission granted in Section 1, the author hereby grants IEEE the unlimited, worldwide, irrevocable right to use his/her name, picture, likeness, voice and biographical information as part of the advertisement, distribution and sale of products incorporating the Work or Presentation, and releases IEEE from any claim based on right of privacy or publicity.
Licensing start date:20.11.2025
Applies to:P. Kocbek, A. Frkatović-Hodžić, D. Lalić, V. Hui, G. Lauc and G. Štiglic, "Exploration of Augmentation Strategies in Multi-Modal Retrieval-Augmented Generation for the Biomedical Domain*: *A Case Study Evaluating Question Answering in Glycobiology," 2025 IEEE International Conference on Big Data (BigData), Macau, China, 2025, pp. 5156-5165, doi: 10.1109/BigData66926.2025.11401037.

Secondary language

Language:Slovenian
Keywords:večmodalno pridobivanje-povečanje generacije, vrednotenje, veliki jezikovni modeli


Comments

Leave comment

You must log in to leave a comment.

Comments (0)
0 - 0 / 0
 
There are no comments!

Back
Logos of partners University of Maribor University of Ljubljana University of Primorska University of Nova Gorica