PhD and Postdoc Win Gold Awards at Reimagine Education Competition

April 16, 2018
Nesra Yannier and Ken Holstein holding their award checks at the Reimagine Education competition

Using mixed-reality to reimagine the classroom from both sides -- an Intelligent Science Station for students and smart glasses for teachers -- earned Gold Awards for an HCII PhD student and Postdoc in the 2017-18 Reimagine Education competition.

Carnegie Mellon University Human-Computer Interaction Postdoctoral Fellow Nesra Yannier earned gold in the Cultivating Curiosity category with Norilla, a mixed-reality learning system for children. In the Virtual Reality / Augmented Reality (VR/AR) category, third-year HCII PhD student Ken Holstein also earned gold with the Lumilo project for teachers.

"It is extraordinary that not only one, but two of our students have won awards at this prestigious competition,” said Jodi Forlizzi, Geschke Director of the HCI Institute and professor. “This is evidence of our growing expertise in learning science and the benefit of working collaboratively."

According to the Reimagine Education website, a panel of international judges selects the winning projects as achieving outstanding standards for innovation, scalability, and depth.




Photo: Children interacting with NoRILLA at Bright Horizons and Pace School (for children with special needs) where it was recently deployed.


Based on proven learning techniques for students in kindergarten through grade 5, Yannier’s project fosters curiosity by combining the benefits of physical play along with technology.

NoRILLA, started by Yannier and her PhD advisors Ken Koedinger and Scott Hudson, is a patent-pending mixed-reality learning system that uses camera sensing and a specialized computer vision algorithm to track students’ interactions with physical objects and give personalized interactive feedback according to their actions in the real world.

EarthShake, the first educational game for the NoRILLA system, was designed to teach early physics principles through hands-on learning.

Students build towers out of blocks enhanced with Kinect and a computer vision algorithm, and a specialized table provides the earthquakes. The supporting AI-based technology can detect how the towers fared during the quake, and the game’s cartoon gorilla can give the students appropriate interactive feedback to understand the underlying principles. NoRILLA can be extended to many different content areas to teach different STEAM topics.

“Nesra’s research is innovative yet practical,” said Koedinger, professor of HCII and one of Yannier’s advisors. “The NoRILLA system she’s created is a cutting-edge technology that she’s demonstrated not only excites kids about learning science but also helps them do so much more effectively than alternative approaches.”

Children have been enjoying NoRILLA in a variety of places [video▶️] around the city of Pittsburgh, including school districts, museums, indoor play areas, and child-care centers.

Research with hundreds of children has shown that the mixed-reality NoRILLA system increases enjoyment and improves children’s science learning by 5 times as compared to an equivalent tablet screen or computer game. This work has been published in multiple venues including AIEDCHI and ToCHI.

The pedagogy implemented in the inquiry cycle above draws on evidence-based techniques in the learning science literature, including predict-explain-observe-explain, contrasting cases, self-explanation, and real-time interactive feedback.

NoRILLA (an acronym for Novel Research-based Intelligent Lifelong Learning Apparatus) has received multiple awards (including a Mister Rogers Scholarship, NSF grants and Innovation Fellowship to turn the research into a start-up to benefit society.

This year’s Cultivating Curiosity Award Gold Winner won a $10,000 prize and $5,000 in Amazon Web Services.  




Right photo: students at David E. Williams Middle School working with Lynnette, an ITS for algebraic equation solving, while their teacher uses Lumilo;
Left photo: illustration of the teacher’s view of class and student level analytics through Lumilo


Holstein’s project, “The Classroom as a Dashboard: Co-designing Wearable Cognitive Augmentation for K-12 Teachers,” was also selected as a finalist, meaning it ranked in the top 7 of the over 1,000 applications received.

Although AI-based software has a long and successful history in K-12 education (with Carnegie Mellon at the vanguard), recent years have seen a surge in the use of AI in K-12 classrooms. AI educational software benefits students by allowing them to work at their own paces, while also enabling the teacher to spend more time working one-on-one with students who need it most. However, students completing self-paced work with AI software also presents challenges for teachers, who are now tasked with monitoring classrooms of students working on a variety of activities at any one time.

Vincent Aleven, associate professor of human-computer interaction and Holstein’s advisor mentioned that many interesting questions remain as the team explores how AI learning software can best be extended to support teachers.

“What analytics are most helpful to teachers and how can they be presented effectively? How are classroom dynamics affected? Do students learn better when their teacher has a support tool based on real-time analytics? In tackling these questions, Kenneth took a user-centered design approach, combined with an innovative new experience prototyping method he created, and advanced statistical analysis. Within this project, he is really coming into his own, which is truly a joy to watch,” said Aleven.

Building on their research with thirty-two K-12 math teachers, Holstein -- together with student interns Gena Hong and Mera Tegene, and guided by HCII faculty Bruce McLaren, and Vincent Aleven --  created Lumilo: mixed-reality smart glasses to support teachers in personalized classrooms.

Holstein and colleagues collaborated closely throughout the design process with “early adopter” teachers who already use AI-based educational software in their classrooms. For example, during an early card sorting exercise, teachers were asked what “superpowers” they would want to help them do their jobs even better. Several responses “centered on abilities to perceive information about individual students’ learning and behavior in real-time” and to be able to instantly see this information “floating over students’ heads” while looking at the classroom.

By focusing on classrooms using intelligent tutoring systems (ITSs), a form of AI-based educational software, the glasses can provide data that augments teachers’ real-time perception [video▶️] of their students’ learning, metacognition, and behavior.

Teachers preferred wearing smart glasses because they could keep their heads up and continue to focus their attention on the classroom. They noted that although real-time analytics from an ITS are helpful, some of the most useful real-time information in the classroom comes from student body language -- something that might be missed by looking down at a dashboard displayed on a tablet computer. At the same time, the glasses allow teachers to keep sensitive information about students private, unlike ambient or projected displays.

Lumilo has recently been deployed in live middle school classrooms, with over 300 middle school students and 11 teachers, The findings from these classroom studies, to be presented at the ICLS 2018 and AIED 2018 conferences, indicate that teachers’ use of Lumilo had a positive impact on students’ learning in AI-enhanced classrooms. In particular, these findings show that providing teachers with real-time analytics about their students’ learning, metacognition, and behavior can serve to narrow the gap in learning outcomes between students starting out with lower versus higher prior knowledge.

Holstein is already planning future research studies with Lumilo and other “teacher/AI hybrid systems”, to explore how human and AI instruction can be best combined to achieve learning outcomes that are greater than either can achieve alone.

Founded in 2014, Reimagine Education is a self-proclaimed "prestigious international competition rewarding innovative initiatives aimed at enhancing student learning outcomes & employability. It culminates in a global conference for those seeking to shape the future of education."

The 2017-18 award ceremony was held at the Franklin Institute in Philadelphia in December.