Automated grading systems provide feedback to students in a variety of ways, but they typically focus on identifying incorrect program behaviors. Such systems provide notices of test case failures or runtime errors, but without debugging skills, students often become frustrated when they don’t know where to start addressing these defects. Borrowing from work in software engineering research related to automated defect location, we leverage previous research describes using statistical fault localization (SFL) techniques to identify the probable locations of defects in student coding assignments. The goal is to use these SFL techniques to provide a scaffold for students, to direct their debugging efforts without giving too much guidance, and thus minimizing the learning associated with investigating the defects. After determining the “suspiciousness” for each line of code involved in the defect, we create a “heat map” visualization overlay onto their source code of the “suspiciousness” scores to visually guide a student’s attention to parts of their code that are most likely to contain problems.
This paper describes the results of an analysis comparing two semesters of our introductory CS1 students, one which had access to the heat map feedback (Fall 2017: n~170), and one that did not (Fall 2015: n~270). The results show that when the heat maps were available, students found it easier to make improvements on their code from submission to submission, as well as spending less time overall achieving the maximum score on the instructor testing assessments. In fact, while we were optimistic in the impact of the heat map feedback, the results we observed were unexpectedly strong. To address this concern, we conducted an additional study, using student submissions from our Spring 2019 CS 1 students (n~230). The results of the second analysis confirmed the results of the first.