Thesis Defense: Derek Lomas
When
-
Where
GHC 6115
Description
THESIS DEFENSE
Optimizing Motivation and Learning with Large-Scale Game Design Experiments
Derek Lomas
COMMITTEE
Ken Koedinger (Co-Chair)
Jodi Forlizzi (Co-Chair)
Emma Brunskill
Jesse Schell
ABSTRACT
Online experiments (e.g., A/B tests) can be used to measure the effects of software design on user behavior. This measurement capacity can be used to optimize the design of software and to contribute to basic scientific research. For instance, with millions of students engaged in online learning software, large numbers of experiments can support rapid advancements in the science of learning -- and support the continuous improvement of actual student outcomes.
This dissertation presents new online methods for measuring and optimizing student motivation and learning. These methods were applied to a popular educational game to investigate how player motivation was affected by challenge, ability, learning, novelty and a variety of common game design elements. Several online experiments (involving over 100,000 online players) produced results that conflict with predictions from dominant theories of motivation. In addition to this large-scale basic research in psychology, this dissertation also presents new machine learning methods for automatically optimizing the design space of online software to maximize desired user behaviors. Taken together, this work contributes a model for using online experiments to support the production and continuous improvement of highly-effective online learning systems.
FULL DOCUMENT
Link to Complete Document
DOCUMENT SUMMARY
Student motivation, which is relatively easily measured in online settings, is a critical component of successful educational outcomes. In a review of related literature, I examine the basis of human motivation, as viewed by Plato and modern cognitivist psychology. I show theoretical convergence on “integration” as a unifying process underpinning different types of motivation and then show how this notion extends to contemporary theories of effective learning game design (i.e., “intrinsic integration”). Subsequently, I examine existing taxonomies of motivational design and discuss these in the context of educational game design. In consideration of online methods for measuring motivation, I describe the possibility of using rapid online experiments to systematically test the predictions of these taxonomies.
To illustrate the possibility of a testable taxonomy of motivational design elements, I introduce a set of online experiments that test the hypothesis that “moderate levels of challenge will be optimally motivating.”I investigate this hypothesis using Battleship Numberline, an award-winning educational game designed to help children develop an improved number sense (i.e., the ability to rapidly and accurately approximate the magnitude of numbers). A series of classroom-based controlled experiments showed that learning from the game is measurable in less than 20 minutes and that this learning transfers to traditional paper-and-pencil math items. Furthermore, I show that Battleship Numberline can be used to measure attributes like player ability and learning as well as game challenge and motivation.
Then, I use a large-scale design experiment (>70,000 gameplay sessions) to search for the optimal level of game challenge to maximize motivation and learning across 14,000 different configurations of the design space of Battleship Numberline. The results show that minimal challenge almost always maximizes motivation. This finding conflicts with modern theories of motivation, which predict that moderate levels of challenge should be optimal. I further investigated motivational theory using game design variations of user choice, novelty, goals, achievement rewards and other motivational design elements. Moderate levels of novelty were found to maximize motivation. “Close games” (where a player almost won or lost) also maximized motivation, while moderate challenge did not. However, if players were given a choice of difficulty, they tended to choose moderately challenging games and play them for longer than easier games. Yet, if players were randomly assigned the same game levels, they tended to play easier games longer than moderately difficult games.
Following this exploration of basic research in the psychology of motivation, I then investigate novel applied research methods focused on “design space optimization”. Using a meta-experimental design, I evaluated three different machine learning approaches for automatically optimizing a design space based on user data. The results showed that multi-armed bandit algorithms can rapidly identify high performing design configurations within a design space and also to minimize player exposure to “low-value” design conditions. This has implications on the ethics of large-scale experiments in learning software, where scientific discovery has to be balanced with the needs of students. The results also reveal that automatic optimization has substantial risks and it is recommended that designers be kept “in the loop” to ensure qualitative insight.
To conclude, I review my empirical findings and discuss their implications for a testable taxonomy of motivational design elements. I then discuss several limitations of my research, including its focus on “convenient” metrics. Finally, I highlight how feedback loops between software and online experimentation can accelerate learning science, design science and the production of highly effective learning software.
CONTRIBUTIONS
This dissertation demonstrates that online games can serve as basic and applied research instruments to measure the effects of game design elements on motivation, learning and other psychological constructs. This work contributes to psychology, educational research, and HCI.
Psychology contribution:
This work contributes to psychological models of achievement motivation by showing how player motivation is influenced by the interaction of constructs like ability (initial performance), challenge (probability of success), novelty (variation in activity) and other design features like goals, choice and feedforward.
This work advances the notion of a testable taxonomy of motivational design elements by reviewing and synthesizing existing motivation taxonomies.
Educational Research contribution:
This work contributes to instructional design research by demonstrating the systematic design of a game that can develop relevant skills (fraction number sense) and transfer improved in-game performance to out-of-game performance.
This work contributes to educational research and psychometrics by validating that a game can provide a reliable and valid assessment of number sense and that it can measure ability, learning, challenge and motivation at scale.
HCI contribution:
This work contributes to the interface optimization literature by demonstrating how large-scale design experiments, as a form of crowdsourcing, can be used to optimize the design of a user interface.
This work contributes to the machine learning literature by providing empirical evidence that illustrates how multi-armed bandit algorithms can automatically optimize a design space while minimizing user exposure to sub-optimal design configurations.
This work contributes to the design literature by formulating design as a design space optimization problem and by evaluating new tools that support the participation of designers in a data-driven optimization process.