Depth of Use: An Empirical Framework to Help Faculty Gauge the Relative Impact of Learning Management System Tools


Learning management system (LMS) tools are increasingly relevant to scaling computing pedagogies. Measuring their utilization and impact at scale, however, remains computationally expensive. We examine the problem of estimating the utilization of a department-wide LMS, and its impact on the design, management and outcomes of Computer Science courses. We introduce ‘depth-of-use’ (DOU): a first-principles, resource-specific metric of LMS utilization. We then hypothesis-test the relationship between DOU and course attributes like modality (course level, mode-of-delivery, third-party app use), participation (enrollment, viewership), logistics (teaching support, digital skills training) and outcomes (average GPA, DFW rate). Experiments on metadata from over 1300 Computer Science courses taught at Virginia Tech between 2015 and 2019 suggest that our framing of DOU helps identify resource-level preferences of micro-cohorts of courses, linked to their content, logistics and pedagogies. We discover that, across the Computer Science department at Virginia Tech, overall LMS use is consistently linked to favorable learning outcomes. We also discover that a complex interaction between the needs for scale, ubiquitous access and interoperability drives strong LMS utilization, with graduate and online-only courses faring highest in their aggregate use of LMS services. Finally, we describe two key applications of our analyses. One, we demonstrate how DOU can help CS faculty identify the relative impact of transition from legacy apps to LMS services. Two, we describe how DOU can help instructional designers evaluate and improve their design interventions.

ITiCSE ‘20: Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education