BEGIN:VCALENDAR VERSION:2.0 X-WR-CALNAME:EventsCalendar PRODID:-//hacksw/handcal//NONSGML v1.0//EN CALSCALE:GREGORIAN BEGIN:VTIMEZONE TZID:America/New_York LAST-MODIFIED:20240422T053451Z TZURL:https://www.tzurl.org/zoneinfo-outlook/America/New_York X-LIC-LOCATION:America/New_York BEGIN:DAYLIGHT TZNAME:EDT TZOFFSETFROM:-0500 TZOFFSETTO:-0400 DTSTART:19700308T020000 RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU END:DAYLIGHT BEGIN:STANDARD TZNAME:EST TZOFFSETFROM:-0400 TZOFFSETTO:-0500 DTSTART:19701101T020000 RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU END:STANDARD END:VTIMEZONE BEGIN:VEVENT CATEGORIES:College of Engineering,Thesis/Dissertations DESCRIPTION:Thesis Advisor: Dr. Adnan El-Nasan, Computer and Information Sc ience Committee Members: Dr. Jiawei Yuan, Computer and Information Scienc e Dr. Long Jiao, Computer and Information Science Abstract: Introductory programming courses face challenges in providing scalable feedback on stud ents’ understanding of core programming concepts. Automated grading show s whether the code passes its test cases, but not the specific gaps in the programming concepts that caused the errors. Knowledge Tracing models tar get those underlying concepts, however, they demand extensive historical d ata, machine-learning expertise, and GPU hardware many instructors may lac k access to.This thesis introduces Knowledge Component-Constrained Diagnos tic Prompting (KCDP), a framework that diagnoses programming gaps through prompt design. KCDP directs a generic commercial Large Language Model to t he concepts each problem is designed to test, requires it to reason throug h the code before naming any gap, and maps each root cause to an instructo r's predefined Knowledge Components (KCs). Human experts and the model are restricted to the same KC vocabulary, so their diagnoses are directly com parable, and their agreement can be measured against each other. Given the prompting-oriented implementation, KCDP can potentially be deployed acros s different commercial LLM platforms using standard access and an instruct or’s existing course-defined KC taxonomy. This provides an accessible ap proach for scalable, concept-level diagnosis in introductory programming e ducation without requiring specialized machine learning infrastructure. KC DP was evaluated against two experts using Google's Gemini 2.5 Flash, wher e it reached an F1 of 0.839 against the human agreement ceiling of 0.885 ( 94.8% of human agreement) with a Cohen's κ of 0.557 against a human-human κ of 0.669. KCDP results held up when run on a second, unrelated model ( DeepSeek), suggesting it is the prompt design, not the model, that is doin g the work. The gaps also proved to be genuine in detecting recurring weak nesses. When KCDP flagged a struggling student as weak in a specific conce pt, that student failed the next problem testing the same concept 77% of t he time, against a 26.4% chance rate, while strong students were rarely fl agged. This shows that the output carries real information about concepts students struggle in which could be basis for triage and other downstream recovery tools. For further information please contact Dr. Adnan El-Nasan at aelnasan@umassd.edu.\nEvent page: https://www.umassd.edu/events/cms/7- 23-26-diagnostic-promptingfor-automated-knowledge-gap.php X-ALT-DESC;FMTTYPE=text/html:

Thesis Advisor: Dr. Adnan El-Na san\, Computer and Information Science

\n

Committee Members:

\n\n

Abstract:

\n

Introductory programming courses face challenges in providing scalable feedback on students’ understanding of core programming concepts. Automa ted grading shows whether the code passes its test cases\, but not the spe cific gaps in the programming concepts that caused the errors. Knowledge T racing models target those underlying concepts\, however\, they demand ext ensive historical data\, machine-learning expertise\, and GPU hardware man y instructors may lack access to.
This thesis introduces Knowledge Co mponent-Constrained Diagnostic Prompting (KCDP)\, a framework that diagnos es programming gaps through prompt design. KCDP directs a generic commerci al Large Language Model to the concepts each problem is designed to test\, requires it to reason through the code before naming any gap\, and maps e ach root cause to an instructor's predefined Knowledge Components (KCs). H uman experts and the model are restricted to the same KC vocabulary\, so t heir diagnoses are directly comparable\, and their agreement can be measur ed against each other. Given the prompting-oriented implementation\, KCDP can potentially be deployed across different commercial LLM platforms usin g standard access and an instructor’s existing course-defined KC taxonom y. This provides an accessible approach for scalable\, concept-level diagn osis in introductory programming education without requiring specialized m achine learning infrastructure. KCDP was evaluated against two experts usi ng Google's Gemini 2.5 Flash\, where it reached an F1 of 0.839 against the human agreement ceiling of 0.885 (94.8% of human agreement) with a Cohen' s κ of 0.557 against a human-human κ of 0.669. KCDP results held up when run on a second\, unrelated model (DeepSeek)\, suggesting it is the promp t design\, not the model\, that is doing the work. The gaps also proved to be genuine in detecting recurring weaknesses. When KCDP flagged a struggl ing student as weak in a specific concept\, that student failed the next p roblem testing the same concept 77% of the time\, against a 26.4% chance r ate\, while strong students were rarely flagged. This shows that the outpu t carries real information about concepts students struggle in which could be basis for triage and other downstream recovery tools.
 
For further information please contact Dr. Adnan El-Nasan at aelnasan@umassd. edu.

Event page: https://www.umassd.ed u/events/cms/7-23-26-diagnostic-promptingfor-automated-knowledge-gap.php

DTSTAMP:20260701T165359 DTSTART;TZID=America/New_York:20260723T140000 DTEND;TZID=America/New_York:20260723T150000 LOCATION:Dion 311 SUMMARY;LANGUAGE=en-us:Knowledge Component-Constrained Diagnostic Prompting  for Automated Knowledge Gap Detection UID:de7c8a2e2b9efdc4b41c19ebcde34717@www.umassd.edu END:VEVENT END:VCALENDAR