What’s next in the future of user interfaces? Look no further for some possibilities.
Faculty and students from the Human-Computer Interaction Institute applied creativity to the technology of fabricating textiles and interactive things, sensing and pose estimation, managing tasks, privacy, and a variety of other fields at the recent UIST conference.
The Association of Computing Machinery (ACM) Symposium on User Interface Software and Technology (UIST) is known for its interdisciplinary focus and hands-on format. UIST is a gathering place for the dreamers and the makers, uniting the strengths of their diverse fields to push the boundaries of human-computer interaction (HCI).
For many UIST (pronounced “wist”) attendees, one favorite aspect of the conference is the annual Demo Reception. Accepted authors showcase their novel interactive concepts, new interaction techniques, devices and systems, while attendees get the opportunity to engage with researchers and enjoy hands-on experiences with the new technologies.
One HCII demo associated with the paper “Personalized Bistable Orthoses for Rehabilitation of Finger Joints” was awarded the Best Demo Honorable Mention Award (Jury’s Choice).
HCII participants contributed to UIST this year as paper authors, as well as participated in workshops, demos and the doctoral consortium. Jump below for a list of accepted papers with HCII contributing authors. A search for Carnegie Mellon in the UIST 2025 program returns a full list of involvement. For a more visual overview of HCII works at UIST, view this YouTube playlist.
Kinethreads: Soft Full-Body Haptic Exosuit using Low-Cost Motor-Pulley Mechanisms
Vivian Shen, Chris Harrison
Our bodies experience a wide variety of kinesthetic forces as we go about our daily lives, including the weight of held objects, contact with surfaces, gravitational loads, and acceleration and centripetal forces while driving, to name just a few. These forces are crucial to realism, yet are simply not possible to render with today's consumer haptic suits, which primarily rely on arrays of vibration actuators built into vests. In this work, we present Kinethreads: a new full-body haptic exosuit design built around string-based motor-pulley mechanisms, which keeps our suit lightweight (<5kg), soft and flexible, quick-to-wear (<30 seconds), comparatively low-cost (~$400), and yet capable of rendering expressive, distributed, and forceful (up to 120N) effects.
📄 Paper
▶ Video
🔗 Project page
Sculptable Mesh Structures for Large-Scale Form-Finding
Jesse T Gonzalez, Yanzhen Zhang, Dian Zhu, Alice Yu, Sapna Tayal, Nazm Furniturewala, Ziying Qi, Somin Ella Moon, Leyi Han, Alexandra Ion, Scott E Hudson
To better capture the interplay between real-world objects and a designer’s work-in-progress, practitioners will often go through a sequence of low-fidelity prototypes (paper, clay, foam) before arriving at a form that satisfies both functional and aesthetic concerns. While necessary, this model-making process can be quite time-consuming, particularly at larger scales, and the resulting geometry can be difficult to translate into a CAD environment, where it will be further refined. This paper introduces a user-adjustable, room-scale, "shape-aware" mesh structure for low-fidelity prototyping. A user physically manipulates the mesh by lengthening and shortening the edges, altering the overall curvature and sculpting coarse forms.
📄 Paper
▶ Video
🔗 Project page
Transforming Everyday Objects into Dynamic Interfaces using Smart Flat-Foldable Structures
Violet Yinuo Han, Amber Yinglei Chen, Mason Zadan, Jesse T Gonzalez, Dinesh K Patel, Wendy Fangwen Yu, Carmel Majidi, Alexandra Ion
Dynamic physical interfaces are often dedicated devices designed to adapt their physical properties to user needs. In this paper, we present an actuation system that allows users to transform their existing objects into dynamic physical user interfaces. We design our actuation system to integrate as a self-contained locomotion layer into existing objects that are small-scale, i.e., hand-size rather than furniture-size. We envision that such objects can act as collaborators: as a studio assistant in a painter's palette, as tutors in a student's ruler, or as caretakers for plants evading direct sunlight.
📄 Paper
▶ Video
Using an Array of Needles to Create Solid Knitted Shapes
Francois Guimbretiere, Victor F Guimbretiere, Amritansh Kwatra, Scott E Hudson
Textiles offer many advantages as a fabrication material, particularly when viewed from the perspective of human use. We present a different machine design using a 2D bed of knitting needles to fabricate additional solid knitted forms. This approach provides substantially more flexibility on how to structure stitches as yarn paths inside a volume. We describe a small prototype 6x6 needle machine, and demonstrate that it can create traditional knits, horizontal knits (knitted in the plane of the needles), and solid knits, including overhangs, and pyramidal forms.
📄 Paper
▶ Video
Designing Looms as Kits for Collaborative Assembly
Samantha Speer, Nickolina Yankova, Joey Huang, Carolyn Rosé, Kylie A Peppler, James McCann, Melisa Orta Martinez
Both engineering kits and collaboration skills are increasingly prevalent in education and are often used in conjunction, yet little research addresses how kit hardware can be designed to support collaboration. This paper posits that certain design features of kits can promote or hinder collaboration. We present a user study evaluating the collaborative assembly of two loom kits: one for higher education (RoboLoom) and one for individuals (Ashford Loom). We developed a coding scheme from the 3Cs framework (coordination, cooperation, and communication) to analyze which hardware features influenced collaboration. We find five design feature categories that may influence collaboration: repetitiveness, specificity, difficulty, parallelizability, and physicality.
📄 Paper
▶ Video
DropPop: Designing Drop-to-Deploy Mechanisms with Bistable Scissors Structures
Yibo Fu, Emily Guan, Jianzhe Gu, Dinesh K Patel, Justin U Soza Soto, Yichi Luo, Carmel Majidi, Josiah Hester, Lining Yao
Deployable structures often rely on complex deployment mechanisms such as external pneumatic pumps, electric motors, or manual assembly. These conventional methods, which are intended for applications in shape morphing architectures, robotics, and product design, can be bulky and unwieldy for everyday interaction and daily use. We introduce a new class of deployable structures that harness the locomotion of a single bistable cap to drive the expansion of a scissor-like mechanism. We explore various input modalities for deployment such as hand dropping, and drone deployment, and showcase demo applications.
📄 Paper
Managing Tasks
StepWrite: Adaptive Planning for Speech-Driven Text Generation
Hamza El Alaoui, Atieh Taheri, Yi-Hao Peng, Jeffrey P Bigham
People frequently use speech-to-text systems to compose short texts with voice. However, current voice-based interfaces struggle to support composing more detailed, contextually complex texts, especially in scenarios where users cannot visually track progress. We introduce StepWrite, a large language model-driven voice-based interaction system that augments human writing ability by enabling structured, hands-free and eyes-free composition of longer-form texts while on the move. StepWrite decomposes the writing process into manageable subtasks and reduces cognitive load by offloading the context-tracking and adaptive planning tasks to the models. This work highlights the potential of structured, context-aware voice interactions in enhancing hands-free and eye-free communication in everyday multitasking scenarios.
📄 Paper
▶ Video
Scaling Context-Aware Task Assistants that Learn from Demonstration and Adapt through Mixed-Initiative Dialogue
Riku Arakawa, Prasoon Patidar, Will Page, Jill Lehman, Mayank Goel
Daily tasks such as cooking, machine operation, and medical self-care often require context-aware assistance, yet existing systems are hard to scale due to high training costs and unpredictable and imperfect performance. This work introduces the PrISM framework, which streamlines the process of creating an assistant for users' own tasks using demonstration and dialogue. Evaluated through multiple studies involving several examples of daily tasks in a user's life, our approach demonstrates improved step-tracking accuracy, enhanced user interaction, and an improved sense of collaboration.
📄 Paper
▶ Video
AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multimodal Information Between Reality and Videos
Zheng Ning, Leyang Li, Daniel Killough, JooYoung Seo, Patrick Carrington, Yapeng Tian, Yuhang Zhao, Franklin Mingzhe Li, Toby Jia-Jun Li
Videos offer rich audiovisual information that can support people in performing activities of daily living (ADLs), but they remain largely inaccessible to blind or low-vision (BLV) individuals. In cooking, BLV people often rely on non-visual cues--such as touch, taste, and smell--to navigate their environment, making it difficult to follow the predominantly audiovisual instructions found in video recipes. To address this problem, we introduce AROMA, an AI system that provides timely responses to the user based on real-time, context-aware assistance by integrating non-visual cues perceived by the user, a wearable camera feed, and video recipe content.
📄 Paper
▶ Video
Sensing and Pose Estimation
EclipseTouch: Touch Segmentation on Ad Hoc Surfaces using Worn Infrared Shadow Casting
Vimal Mollyn, Nathan DeVrio, Chris Harrison
The ability to detect touch events on uninstrumented, everyday surfaces has been a long-standing goal for mixed reality systems. Prior work has shown that virtual interfaces bound to physical surfaces offer performance and ergonomic benefits over tapping at interfaces floating in the air. A wide variety of approaches have been previously developed, to which we contribute a new headset-integrated technique called EclipseTouch. We use a combination of a computer-triggered camera and one or more infrared emitters to create structured shadows, from which we can accurately estimate hover distance (mean error of 6.9 mm) and touch contact (98.0% accuracy). We discuss how our technique works across a range of conditions, including surface material, interaction orientation, and environmental lighting.
📄 Paper
▶ Video
🔗 Project page
IMUCoCo: Enabling Flexible On-Body IMU Placement for Human Pose Estimation and Activity Recognition
Haozhe Zhou, Riku Arakawa, Yuvraj Agarwal, Mayank Goel
IMUs are regularly used to sense human motion, recognize activities, and estimate full-body pose. In this paper, we rethink IMU-based motion sensing by acknowledging that signals can be captured from any point on the human body. We introduce IMU over Continuous Coordinates (IMUCoCo), a novel framework that maps signals from a variable number of IMUs placed on the body surface into a unified feature space based on their spatial coordinates. Our evaluations demonstrate that IMUCoCo supports accurate pose estimation in a wide range of typical and atypical sensor placements. Overall, IMUCoCo supports significantly more flexible use of IMUs for motion sensing than the state-of-the-art, allowing users to place their sensors-laden devices according to their needs and preferences.
📄 Paper
▶ Video
Privacy
Imago Obscura: An Image Privacy AI Co-pilot to Enable Identification and Mitigation of Risks
Kyzyl Monteiro, Yuchen Wu, Sauvik Das
Users often struggle to navigate the privacy vs. publicity boundary in sharing images online. To address this challenge, we introduce and evaluate Imago Obscura, an intent-aware AI-powered image-editing copilot that enables users to identify and mitigate privacy risks in images they intend to share. Imago Obscura enables users to articulate their image-sharing intent and privacy concerns, and then recommends and facilitates application of a suite of obfuscation techniques found to be effective in prior literature - e.g., inpainting, blurring, and generative content replacement. We evaluated Imago Obscura with 15 end-users in a lab study and found that it improved users' awareness of image privacy risks and their ability to address them, enabling more informed sharing decisions.
📄 Paper
▶ Video
UI Agents, Gen AI Agents, Object Agents
Morae: Proactively Pausing UI Agents for User Choices
Yi-Hao Peng, Ding Li, Jeffrey P Bigham, Amy Pavel
User interface (UI) agents promise to make inaccessible or complex UIs easier to access for blind and low-vision (BLV) users. However, current UI agents typically perform tasks end-to-end without involving users in critical choices or making them aware of important contextual information, thus reducing user agency. To address this problem, we introduce Morae, a UI agent that automatically identifies decision points during task execution and pauses so that users can make choices. Morae uses large multimodal models to interpret user queries alongside UI code and screenshots, and prompts users for clarification when there is a choice to be made. This work exemplifies a mixed-initiative approach in which users benefit from the automation of UI agents while being able to express their preferences.
📄 Paper
▶ Video
POET: Supporting Prompting Creativity and Personalization with Automated Expansion of Text-to-Image Generation
Evans Xu Han, Alice Qian, Haiyi Zhu, Hong Shen, Paul Pu Liang, Jane Hsieh
State-of-the-art visual generative AI tools hold immense potential to assist users in the early ideation stages of creative tasks. We introduce POET, a real-time interactive tool that (1) automatically discovers dimensions of homogeneity in text-to-image generative models, (2) expands these dimensions to diversify the output space of generated images, and (3) learns from user feedback to personalize expansions. Focusing on visual creativity, POET offers a first glimpse of how interaction techniques of future text-to-image generation tools may support and align with more pluralistic values and the needs of end users during the ideation stages of their work.
📄 Paper
▶ Video
Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors
Michelle S. Lam, Fred Hohman, Dominik Moritz, Jeffrey P Bigham, Kenneth Holstein, Mary Beth Kery
AI policy sets boundaries on acceptable behavior for AI models, but this is challenging in the context of large language models (LLMs). We introduce policy maps, an approach to AI policy design inspired by the practice of physical mapmaking. Instead of aiming for full coverage, policy maps aid effective navigation through intentional design choices about which aspects to capture and which to abstract away. With Policy Projector, an interactive tool for designing LLM policy maps, an AI practitioner can survey the landscape of model input-output pairs, define custom regions (e.g., “violence”), and navigate these regions with if-then policy rules that can act on LLM outputs (e.g., if output contains “violence” and “graphic details,” then rewrite without “graphic details”).
📄 Paper
▶ Video
Towards Unobtrusive Physical AI: Augmenting Everyday Objects with Intelligence and Robotic Movement for Proactive Assistance
Violet Yinuo Han, Jesse T Gonzalez, Christina Yang, Zhiruo Wang, Scott E Hudson, Alexandra Ion
What if familiar objects could proactively assist users? For example, a stapler moving across the table to help users organize documents. In this paper, we build on the qualities of tangible interaction and focus on recognizing user needs in everyday tasks to enable ubiquitous yet unobtrusive tangible interaction. To achieve this, we introduce an architecture that leverages Large Language Models (LLMs) to perceive users’ environment and daily activities, perform spatial-temporal reasoning, and generate object actions aligned with inferred user intentions and object properties.
📄 Paper
▶ Video
🔗 Coverage: This Stapler Knows When You Need It
