HCII PhD Thesis Proposal: Franklin Mingzhe Li
When
-
Description
Architecting Physical Information Space: AI-Enabled Assistive Technology for Non-Visual Cooking
Franklin Mingzhe Li
HCII PhD Thesis Proposal
Time & Location
Wednesday, Oct 2nd, 2024 - 11AM EST
Newell-Simon Hall (NSH) 3305
Meeting ID: 994 6647 6974
Passcode: 507805
Committee
Patrick Carrington (Chair), Carnegie Mellon University
John Zimmerman, Carnegie Mellon University
Chris Harrison, Carnegie Mellon University
Shaun K. Kane, Google
Gregory Abowd, Northeastern University
Abstract
Many Activities of Daily Living (ADLs) and Instrumental Activities of Daily Living (IADLs), such as cooking, are highly visual and create barriers for people with vision impairments. With over 2.2 billion people globally affected by vision loss, these challenges significantly impact autonomy, health, and overall quality of life. Cooking, in particular, involves navigating the kitchen, identifying ingredients, and following recipes, which often leads to reliance on unhealthy, costly pre-packaged meals. This contributes to a 150% higher obesity rate among the visually impaired. Despite advancements in assistive technologies, there remains a critical gap in providing solutions that address non-visual interaction with the physical information space required for complex tasks like cooking.
This thesis explores this gap by exploring how AI-enabled assistive technologies can support non-visual interaction with physical spaces, specifically through architecting a physical information space for the kitchen. For AI-based systems to be effective through physical activities, we need data about objects' space, environments, and user actions. We need to create such physical information space—comprising structured data about objects space, environments, and user actions—must first be generated. The physical information space is necessary because interactions and user needs are contextualized under different physical spaces, a generalized approach can fail support user's needs, such as not providing the preferred contextual information, while adopting AI-enabled assistive technologies. This process is particularly difficult in physical activities like cooking, where real-time contextual information is key, such as what to track, why they are important, and how to communicate and support interaction. My research explores how to architect this physical information space, including the design, development, and deployment of AI-driven models.
The research is structured into four key stages. First, it examines current cooking practices and challenges faced by visually impaired individuals, using data from content analysis of YouTube videos and interviews with visually impaired cooks. Second, it explores the design of accessible recipe systems, analyzing how visually impaired users access, structure, and interpret recipe information. Third, the thesis investigates how people access contextual information about kitchen objects, such as position, orientation, and dynamically changing rich object status. Finally, I create OSCAR (Object Status Context Awareness for Recipes), a system that uses the generated information space to track and align object context with recipe steps to support non-visual cooking.
OSCAR represents a novel application of AI and computer vision to create and use a physical information space for non-visual cooking. By leveraging the structured data in the physical information space, the system provides real-time feedback, guides users through recipe steps, tracks object status, and provides hands-free contextual support. For instance, OSCAR could inform users if food is fully cooked, monitor the location of ingredients, and confirm the completion of recipe steps. The goal of OSCAR is to demonstrate how an AI-powered platform, built on a robust information space, can transform the cooking experience for visually impaired individuals, offering greater autonomy, safety and enjoyment.
To complete the dissertation, I will continue to develop and refine features of OSCAR mobile interface to function as a versatile and context-sensitive cooking progress tracking assistant for people with vision impairments. I will also investigate and explore additional functionalities with more granular physical information space, such as helping users find task-specific instructions, and providing personalized guidance based on user's kitchen setup and ingredients. I will conduct usability studies to assess the effectiveness of the system in improving user autonomy, reducing errors, and enhancing the overall cooking experience. These studies will serve as a validation of both the physical information space created and the OSCAR system as a practical application of AI-enabled assistive technology.
Proposal Document
https://drive.google.com/