PhD Thesis Defense: Steven Dang, "Exploring Behavioral Measurement Models of Learner Motivation"

Steven Dang

Wednesday, December 15, 2021 - 10:30am
Newell-Simon Hall 4305 & remote (location to be in email announcements)

Exploring Behavioral Measurement Models of Learner Motivation

(Hybrid event)

Thesis Committee:
Ken Koedinger (HCII, CMU)
John Stamper (HCII, CMU)
Geoff Kaufman (HCII, CMU)
Artur Dubrawski (RI, CMU)

Sidney D’Mello (CS, University of Colorado, Boulder)


No learning happens until students make the choice to engage, regardless of how well-designed and personalized a lesson. While advanced algorithms have been developed to personalize and accelerate learning, similar highly quality models and algorithms aren’t available for intelligent support of student motivation. The greatest challenge lies in the lack of high-quality measurement models to support the administration of motivational interventions. Existing models focus on the observable engagement behaviors of students. These measures are prone to noise from non-learner-specific influences as opposed to reflecting the underlying motivational drivers of engagement. This complicates the task of leveraging these analytics for assessing individual student motivational needs to support greater engagement.


In this dissertation, I address this gap through the development of a model to measure student diligence, their capacity to self-regulate and engage with learning activities. I leverage prior research in psychology and psychometrics to identify behavioral metric candidates. Through secondary analysis of a year-long longitudinal dataset of log data from students learning with intelligent tutoring systems, I evaluate the viability of these behavioral measures to estimate student diligence. I further develop these measures by leveraging theory to account for some cognitive, temporal, and social confounding factors.  My analysis indicates that these behavioral measures, while better indicators of diligence, are still prone to other sources of noise that make the measures unreliable.


To address the unique challenges of measurement with observational data, I explore the viability of diversifying the model inputs by leveraging multiple operationalizations of diligence for estimation. I demonstrate that multi-operational models possess more desirable psychometric properties than any individual measure. Furthermore, I developed the Learner Engagement Simulator (LEnS) to generate data that reflects the challenges of estimating motivational constructs due to unobserved influences from the social and environmental context. My analysis of the simulated data reinforces the findings with real student data that multiple operationalizations of diligence increases the estimator accuracy.


In summary, this thesis makes several contributions. In the learning sciences, this work contributes to the understanding of how student’s capacity to self-regulate interacts with the cognitive, temporal, and social contexts during classroom learning to form patterns of engagement. To the learning analytics community, this work contributes a fine-grained, log-data based model of student diligence that can be leveraged to assess and support students’ engagement. For the educational data mining community, this work demonstrates the value of leveraging multiple-operationalizations when building behavior-based measurement models of motivation. For the human-computer interaction community, this work develops the LEnS Framework for furthering the use of simulations to model how motivational constructs drive the behavior of systems of learners and educational technology within a learning environment.

Queenie Kravitz

Upcoming Events