HCII Ph.D. Thesis Proposal: Jason Wu
When
-
Description
Computational Understanding of User Interfaces
Committee:
Jeff Bigham (Chair), Carnegie Mellon University
Jodi Forlizzi, Carnegie Mellon University
Sherry Tongshuang Wu, Carnegie Mellon University
Tom Mitchell, Carnegie Mellon University
Jeff Nichols, Apple
Abstract:
A grand challenge in human-computer interaction (HCI) is constructing user interfaces (UIs) that can serve different users in different contexts. It is exceedingly difficult to manually design an optimal UI because of the trade-offs that go into designing for different scenarios and the mismatches between expected and actual usage conditions. As a result of these practical challenges and substantial effort required on the part of designers and developers, most UIs are created with a limited set of usage assumptions "baked in" that are often misaligned with end users. In this dissertation, I develop computational models and approaches that allow machines to understand and ultimately enhance UIs. Improved machine understanding has many benefits for assistive technology, software engineering, and end-user technology. I specifically focus on an application of computational UI understanding that addresses the assumption mismatch problem by understanding existing UIs then transforming them to better match observed usage behaviors. First, I built a recommendation-based system that mapped passively-collected usage behaviors to existing OS features that applied transformations to make existing apps easier to use. However, a drawback of this initial approach was that it does not work on apps that are constructed using inaccessible toolkits that do not expose application semantics and do not respond to OS features or work with most assistive technology. To this end, I collected datasets, built computational models, and developed learning algorithms that predict this required metadata for any existing app from pixel information (i.e., a screenshot). Using these predicted app semantics, I introduce two methods of dynamically generating proxy interfaces that are enhanced based on user-specific information or responsive to OS-level transformations enabled by existing features and customizations.
Link to proposal document: https://jasonwunix.com/proposal.pdf