TapSense Improved Touch Detection on Millions of Smartphones
Impact: We trained smartphone touchscreens to detect more inputs
We discovered that a smartphone’s onboard microphone could be used to reliably detect different touchscreen inputs – for example, a tap by a finger as compared to a knuckle – which created opportunities for new interactions.
Our novel lightweight machine learning (ML) engine could run on smartphones and small embedded devices at a time when industry was mostly focused on heavyweight ML that ran on big and expensive GPUs. The approximately 500M smartphones that installed our lightweight machine learning engine could recognize more inputs and power new user-facing features.
This technical CMU research led to a pioneering company in the fields of TinyML and AutoML.
This work led to...
- A spin-off company, Qeexo. The company grew to 5 worldwide offices with about 60 employees. Qeexo was acquired by TDK in 2023 and now provides edge AI solutions to industry as TDK SensEI.
- Multiple industrial awards at CES, MWC and Sensors Expo.
Supported by: National Science Foundation Graduate Research Fellowship Program (GRFP) award, and later by venture capital.
Timing: This research started around 2011, with the paper TapSense. In order to ship this interaction technique, advances in TinyML engines had to be made, which became the core of the company.
Related work:
- TapSense overview
- TDK to acquire Qeexo to enable complete smart edge platforms press release
- TDK SensEI website
Researchers: Chris Harrison, Scott Hudson, and team
Research Area: Technical Invention
Looking at Additional HCII Impacts...
NoRILLA Interactive Mixed-Reality Science for Kids
Our novel, mixed-reality intelligent science stations bridge the physical and virtual worlds. Millions of children and families across the US are learning more science and improving their critical thinking skills after predicting, observing and explaining experiments with our patented system at their schools and museums.
Protecting Millions of People from Phishing Scams
We studied the social aspects of phishing attacks and protected millions of people through education and training. This work led to new educational methods to raise awareness, effective anti-phishing warnings, and algorithmic detection of phishing attacks.
Grading Your Smartphone Apps on Their Privacy Practices
We created a model that analyzed over 1 million smartphone apps on their privacy and data collection practices, and then assigned them a public-facing privacy grade. This easy-to-follow grading system raised public awareness and led to improved privacy practices from several app developers.
TapSense Improved Touch Detection on Millions of Smartphones
We trained smartphones to reliably detect four different touchscreen inputs – for example, a tap by a finger pad as compared to a knuckle – which created opportunities for new interactions and features.
Millions Learned More Math with Our AI Tutor
Millions of middle school and high school students across the US have learned more math as a result of our decades of work with AI-based cognitive tutoring systems.
Human-Centered Privacy for the Internet of Things
As more gadgets become “smart” things and the Internet of Things (IoT) expands to tens of billions of connected devices, we want consumers to be aware of what their devices are doing with their personal data.
Improved Math Mastery with the Decimal Point Game
We explored the learning analytics behind our digital learning game to see how student learning responds to curriculum changes to the game. Over the past 10 years, over 1,500 students have benefited from the Decimal Point game and curriculum materials.
Zensors Turns Cameras Into Smart Sensors
In one application installed at the Pittsburgh International Airport in 2019, Zensors use existing cameras as powerful general-purpose sensors to provide wait time estimates for the security line, which benefits almost 10M PIT travelers per year.
Recognizing new inputs unlocked opportunities for new touchscreen interactions.