Impact: We trained smartphone touchscreens to detect more inputs

We discovered that a smartphone’s onboard microphone could be used to reliably detect different touchscreen inputs – for example, a tap by a finger as compared to a knuckle – which created opportunities for new interactions. 

Our novel lightweight machine learning (ML) engine could run on smartphones and small embedded devices at a time when industry was mostly focused on heavyweight ML that ran on big and expensive GPUs. The approximately 500M smartphones that installed our lightweight machine learning engine could recognize more inputs and power new user-facing features. 

This technical CMU research led to a pioneering company in the fields of TinyML and AutoML. 

This work led to... 

  • A spin-off company, Qeexo.  The company grew to 5 worldwide offices with about 60 employees. Qeexo was acquired by TDK in 2023 and now provides edge AI solutions to industry as TDK SensEI.
  • Multiple industrial awards at CES, MWC and Sensors Expo.


Supported by:  National Science Foundation Graduate Research Fellowship Program (GRFP) award, and later by venture capital.

Timing:  This research started around 2011, with the paper TapSense. In order to ship this interaction technique, advances in TinyML engines had to be made, which became the core of the company. 

Related work:  

Researchers:  Chris Harrison, Scott Hudson, and team 

Research Area:  Technical Invention
 

 

Looking at Additional HCII Impacts...

Nesra sits at a NoRILLA table and reaches for a tower of blocks

NoRILLA Interactive Mixed-Reality Science for Kids

Our novel, mixed-reality intelligent science stations bridge the physical and virtual worlds. Millions of children and families across the US are learning more science and improving their critical thinking skills after predicting, observing and explaining experiments with our patented system at their schools and museums.

screenshot of the game Anti-Phishing Phil, named after an orange fish in an underwater world that helps players to learn more about phishing

Protecting Millions of People from Phishing Scams

We studied the social aspects of phishing attacks and protected millions of people through education and training. This work led to new educational methods to raise awareness, effective anti-phishing warnings, and algorithmic detection of phishing attacks.

PrivacyGrade

Grading Your Smartphone Apps on Their Privacy Practices

We created a model that analyzed over 1 million smartphone apps on their privacy and data collection practices, and then assigned them a public-facing privacy grade. This easy-to-follow grading system raised public awareness and led to improved privacy practices from several app developers.

hands interacting with a black touch screen with labeled inputs of fingernail, knuckle and pad

TapSense Improved Touch Detection on Millions of Smartphones

We trained smartphones to reliably detect four different touchscreen inputs – for example, a tap by a finger pad as compared to a knuckle – which created opportunities for new interactions and features.

screenshot from PAT, a cognitive tutor for algebra

Millions Learned More Math with Our AI Tutor

Millions of middle school and high school students across the US have learned more math as a result of our decades of work with AI-based cognitive tutoring systems.

proposed data label for IoT devices, modeled after a nutritional label

Human-Centered Privacy for the Internet of Things

As more gadgets become “smart” things and the Internet of Things (IoT) expands to tens of billions of connected devices, we want consumers to be aware of what their devices are doing with their personal data.

decimal point game map

Improved Math Mastery with the Decimal Point Game

We explored the learning analytics behind our digital learning game to see how student learning responds to curriculum changes to the game. Over the past 10 years, over 1,500 students have benefited from the Decimal Point game and curriculum materials.

Zensors airport queue screenshot

Zensors Turns Cameras Into Smart Sensors

In one application installed at the Pittsburgh International Airport in 2019, Zensors use existing cameras as powerful general-purpose sensors to provide wait time estimates for the security line, which benefits almost 10M PIT travelers per year.