| Title: | A comprehensive user-centric method for evaluating and comparing XAI explanations |
|---|
| Authors: | ID Brdnik, Saša (Author)
ID Karakatič, Sašo (Author)
ID Šumak, Boštjan (Author) |
|---|
| Files: |  A_Comprehensive_User-Centric_Method_for_Evaluating_and_Comparing_XAI_Explanations.pdf (3,78 MB)
MD5: C42BEFA2F6ECA8ED1AF28605F9B5AD1E
|
|---|
| Language: | English |
|---|
| Work type: | Article |
|---|
| Typology: | 1.01 - Original Scientific Article |
|---|
| Organization: | FERI - Faculty of Electrical Engineering and Computer Science
|
|---|
| Abstract: | Evaluation and selection of explanations in explainable artificial intelligence (XAI) is essential as AI systems grow, increasing transparency needs. Existing methods rarely built on theoretical grounding and failed to distinguish perceived and actual user understanding. This paper proposes a comprehensive user evaluation method for XAI explanations in a structured questionnaire format, enabling efficient data collection and comparison. Core explanation dimensions recognized in literature were integrated into our method: trust, satisfaction, mental models, and mental effort. Reliable instruments analyzed with exploratory factor analysis were used to assess trust and satisfaction. Mental models were assessed through reflection, prediction, and glitch detection tasks. Ten XAI techniques were evaluated: image-based (LIME positive and negative masking, feature occlusion, saliency maps, SHAP, LIME overlay mask) and numeric-based (SHAP bar chart, partial dependence plots, decision tree, counterfactual explanations, ANCHOR). Our method effectively differentiates explanations across multiple dimensions, aiding the selection of the most suitable technique for end-users. |
|---|
| Keywords: | user-centered XAI evaluation, XAI evaluation, user evaluation method, explainable artificial intelligence |
|---|
| Publication status: | Published |
|---|
| Publication version: | Version of Record |
|---|
| Submitted for review: | 21.06.2024 |
|---|
| Article acceptance date: | 05.02.2025 |
|---|
| Publication date: | 12.03.2025 |
|---|
| Publisher: | Taylor & Francis |
|---|
| Year of publishing: | 2025 |
|---|
| Number of pages: | 4 str. |
|---|
| PID: | 20.500.12556/DKUM-97982  |
|---|
| UDC: | 004.8 |
|---|
| ISSN on article: | 1532-7590 |
|---|
| COBISS.SI-ID: | 228953347 |
|---|
| DOI: | 10.1080/10447318.2025.2470278 |
|---|
| Publication date in DKUM: | 06.05.2026 |
|---|
| Views: | 163 |
|---|
| Downloads: | 60 |
|---|
| Metadata: |    |
|---|
| Categories: | Misc.
|
|---|
|
:
|
Copy citation |
|---|
| | | | Average score: | (0 votes) |
|---|
| Your score: | Voting is allowed only for logged in users. |
|---|
| Share: |  |
|---|
Hover the mouse pointer over a document title to show the abstract or click
on the title to get all document metadata. |
