HCII Ph.D. Thesis Defense: Alexandria K. Vail
When
-
Description
Multimodal and Dyadic Modeling of Client-Therapist Interaction
Thesis Committee:
Dr. Louis-Philippe Morency (Chair, LTI)
Dr. Jeffrey F. Cohn (University of Pittsburgh)
Dr. Robert Kraut (HCII)
Dr. Adam Perer (HCII)
Abstract:
Productive interaction between client and therapist is central to successful therapy, but is often hindered by substantial challenges along the way. During each therapy session, the therapist is constantly assessing the client's symptoms through their behavior. These behaviors may be expressed through multiple channels: verbal spoken language and nonverbal ``body language''. Therefore, the first challenge we focus on is the multimodal aspect of behavior. Another fundamental challenge during therapy is the development and maintenance of a collaborative relationship between the client and the therapist. This relationship develops over the course of several weeks, requiring longitudinal study within and across multiple sessions. Thus, the second challenge we focus on is the social aspect of behavior. Finally, we acknowledge the challenge of modeling such complex behavior over time in a manner that is useful for prediction tasks while remaining interpretable for scientific insights. We explore hybrid modeling: the combination of data-driven methods frequently used in computational modeling, such as neural networks, with hypothesis-driven methods often preferred in psychology and statistics, such as structural equation modeling.
We pursue the challenge of multimodal behavior dynamics through two dimensions: verbal behavior and nonverbal behavior. This work addresses the difficulty of evaluating client symptoms across multiple modalities. The verbal component of behavior conveys information not only through high-level message intent, but also through more detailed aspects of speech, such as word choice and sentence structure. We present a multifaceted analysis of the client's spoken language as it relates to their psychological health, including a detailed consideration of lexical, structural, and disfluency components of their speech. The nonverbal component of behavior includes behaviors such as facial expressions, gestures, or eye gaze patterns. In particular, we study the ever-prevalent nonverbal signal of gaze aversion patterns and how they provide information about the severity of the client's symptoms. We pursue the challenge of social behavior dynamics in two aspects: turn-taking behavior and entrainment behavior. This work investigates the growth and decline of the collaborative relationship between the client and therapist over the course of multiple dyadic interactions. Through turn-taking behavior, interaction participants attempt to maintain the flow of conversation. We recount a detailed analysis of turn-taking behaviors and mirroring of head gestures as they signal the quality of the collaboration between client and therapist. Through entrainment behavior, participants synchronize their behavior patterns, whether consciously or subconsciously. We present a modeling of stylistic and content entrainment over multiple sessions as it relates to the client-therapist relationship.
Finally, we pursue the challenge of modeling these complex behavior patterns using hybrid modeling, combining data-
driven and hypothesis-driven methods for predictive and interpretable computational behavior modeling. We present a unique approach to representation learning: the process of identifying meaningful patterns in data. Our approach utilizes structural equation models to create valuable and interpretable representations for computational models. We further refine this method to support end-to-end learning, including simultaneous training of both data-driven and hypothesis-driven components. We demonstrate that integrating structural equation modeling into a neural network during the training process can often improve the predictive performance of the model.
Document @ Google Drive: https://bit.ly/VailThesis