| Title: | Impact of developer queries on the effectiveness of conversational large language models in programming |
|---|
| Authors: | ID Taneski, Viktor (Author)
ID Karakatič, Sašo (Author)
ID Rek, Patrik (Author)
ID Jošt, Gregor (Author) |
|---|
| Files: |  applsci-15-06836-v3.pdf (994,41 KB)
MD5: B03C7CFF080684EE2F63BADBAD2581E7
 https://www.mdpi.com/2076-3417/15/12/6836
|
|---|
| Language: | English |
|---|
| Work type: | Article |
|---|
| Typology: | 1.01 - Original Scientific Article |
|---|
| Organization: | FERI - Faculty of Electrical Engineering and Computer Science
|
|---|
| Abstract: | This study investigates the effects of LLM-based coding assistance on web application development by students using a frontend framework. Rather than comparing different models, it focuses on how students interact with LLM tools to isolate the impact of query type on coding success. To this end, participants were instructed to rely exclusively on LLMs for writing code, based on a given set of specifications, and their queries were categorized into seven types: Error Fixing (EF), Feature Implementation (FI), Code Optimization (CO), Code Understanding (CU), Best Practices (BP), Documentation (DOC), and Concept Clarification (CC). The results reveal that students who queried LLMs for error fixing (EF) were statistically more likely to have runnable code, regardless of prior knowledge. Additionally, students seeking code understanding (CU) and error fixing performed better, even when normalizing for previous coding ability. These findings suggest that the nature of the queries made to LLMs influences the success of programming tasks and provides insights into how AI tools can assist learning in software development. |
|---|
| Keywords: | large language models, LLMs, prompt engineering, query type analysis, AI-assisted programming, educational software development |
|---|
| Publication status: | Published |
|---|
| Publication version: | Version of Record |
|---|
| Submitted for review: | 05.05.2025 |
|---|
| Article acceptance date: | 14.06.2025 |
|---|
| Publication date: | 17.06.2025 |
|---|
| Publisher: | MDPI |
|---|
| Year of publishing: | 2025 |
|---|
| Number of pages: | 22 str. |
|---|
| Numbering: | Vol. 15, iss. 12, [article no.] 6836 |
|---|
| PID: | 20.500.12556/DKUM-93383  |
|---|
| UDC: | 004.43 |
|---|
| ISSN on article: | 2076-3417 |
|---|
| COBISS.SI-ID: | 240210435 |
|---|
| DOI: | 10.3390/app15126836 |
|---|
| Publication date in DKUM: | 23.06.2025 |
|---|
| Views: | 175 |
|---|
| Downloads: | 15 |
|---|
| Metadata: |    |
|---|
| Categories: | Misc.
|
|---|
|
:
|
Copy citation |
|---|
| | | | Average score: | (0 votes) |
|---|
| Your score: | Voting is allowed only for logged in users. |
|---|
| Share: |  |
|---|
Hover the mouse pointer over a document title to show the abstract or click
on the title to get all document metadata. |
