HCII PhD Thesis Defense - Yi-Hao Peng
When
-
Where
Newell-Simon Hall (NSH) 3305
Description
Thesis Committee:
Jeff Bigham (Co-Chair) - CMU HCII & LTI
Amy Pavel (Co-Chair) - UC Berkeley EECS
Jodi Forlizzi - CMU HCII
Graham Neubig - CMU LTI
Ming-Hsuan Yang - Google DeepMind & UC Merced EECS
Abstract:
A long-standing goal in human-computer interaction is to make interfaces easy to use for everyone. The goal remains especially difficult for blind screen reader users. Screen readers flatten rich visual interfaces into a linear stream, removing the at-a-glance overview that sighted users rely on to discover actions, compare choices, and recover from mistakes. Modern AI agents that operate interfaces from natural language offer a new path to this goal. But at many points in a task, an agent has to choose between acting on its own and bringing the person into the decision. Current agents almost always just act, assuming whatever the request leaves unsaid. My dissertation takes up both sides of that decision. When a choice should stay with the person, I build agents that recognize it, pause, and return it in accessible form instead of committing to defaults, with perception grounded in models trained on synthetic environments. When the person has delegated a choice, I build systems that keep the result inspectable and revisable, and models that learn an individual's taste from a handful of comparisons instead of averaging everyone's preferences. My work contributes computational models and accessible tools that allocate agency decision by decision, with the broader goal of helping people steer, verify, and therefore trust automation in everyday digital work.
Thesis document:
https://drive.google.com/file/d/1lsdzXun9jiZAbdF9h3HcAoT7ezjYJXS7/view?usp=sharing
Remote: Zoom Link