HCII Ph.D. Defense: Jason Wu

A grand challenge in human-computer interaction (HCI) is constructing user interfaces (UIs) that make computers useful for all users across all contexts. Conventional UI development processes have approached this goal by iteratively converging towards a single “final” UI through prototyping, implementation, and testing. However, even when following best practices, this approach locks in a set of assumptions that often cannot accommodate the diversity of user abilities, usage contexts, or computing technologies, ultimately limiting how we can use computers. For example, UIs designed for one context might not perform well in another, and UIs can be inaccessible to users with different abilities and preferences. In this dissertation, I propose a new approach that uses machine-learning-driven systems that automatically understand and manipulate existing UIs. Using content and functionality inferred from the UI, combined with sensed usage context, a new interface can be synthesized that better meets the immediate needs of individual users.

My work represents the initial technical foundation for this vision. First, I describe approaches for understanding user ability and context (user understanding), which HCI suggests is the basis for building good interfaces. I describe a recommendation system that recommends device settings (e.g., accessibility features) based on sensed usage behaviors and user interaction logs. Results from a user study showed that the majority (74%) of predicted recommendations were rated as helpful. Nevertheless, this approach of adapting interfaces through configuration changes has traditionally been limited, since applications often do not properly expose their semantics to external services. To this end, I describe several projects in the area of UI understanding, which shows that it is possible to overcome this barrier using data-driven ML models that predict interface layout, structure, and functionality from visual information, which is how UIs are generally assumed to be used. These predicted semantics can enable many forms of existing computing infrastructure, such as accessibility and UI agents to work more reliably and robustly. Finally, I combine both user and UI understanding to dynamically generate and adapt UIs that meet the specific needs of users. I describe ML-driven systems that generate UIs by modifying existing application layouts and generating UI code based on personalized user profiles and design objectives. Ultimately, through my work, I show that computational understanding of user interfaces allows UIs to be transformed from static objects into malleable representations that can be dynamically reshaped for new devices, modalities, and users.