HCII Students’ Game Wins "Most Creative" at TreeHacks
News
Lucid Drums earned Most Creative Hack and finished in Top 8
Congratulations to Laura Beth Fulton, MHCI class of 2020, and Jiachen (Lilian) Gong, METALS class of 2020, for their recent success at TreeHacks 2020. Their team’s interactive VR game to improve childhood development and patient coordination through drumming finished in the top 8 and received a Grand Prize for “Most Creative Hack.”
Fulton and Gong attended the weekend hackathon at Stanford University February 14 to 16, 2020.
The pair from the Human-Computer Interaction Institute rounded out their team with two Stanford students they met at the in-person team formation event at the hackathon. The four teammates combined their passion for improving early childhood education with their strengths in user experience (UX), learning sciences, VR game design and machine learning to develop an educational game.
Over the course of the 36-hour event, the team developed the VR game Lucid Drums. This interactive VR game features a tropical environment where virtual purple hands tap virtual drums to the beat of any given song.
Players are judged on their accuracy to tap the drums with the rhythm of the song as indicated by approaching balls representing song beats. As players improve their coordination and skills, the game offers players differing levels of complexity.
Not only is Lucid Drums educational for young musicians but additional research showed that rhythmic drumming has also proved to be beneficial for rehab patients. Identifying rhythms and patterns helps adults recovering from motor impairments and drug addiction.
Another reason Lucid Drums is accommodating to patients in recovery is because the game can be played while sitting or standing. And, although it was an added challenge for the developers, players do not need the grip strength to hold onto a controller to play -- the system can recognize the player’s own hands for a more authentic “hands-on” drumming experience.
“With VR, we leveraged hand tracking which is currently a very new, developer-only ability with the Oculus Quest,” said Fulton. “Our team was proud that we successfully leveraged hand tracking to capture real-time movement and after several fixes, we removed lag and could pin-point accuracy for drum hits versus misses.”
The team built the VR application for the Oculus Quest headset using the game development platform Unity. Gong applied knowledge from the course Introduction to Machine Learning (10-601) last semester to build the auto beat detection and generation algorithm based on the Recurrent Neural Network (RNN). This deep learning algorithm allows the users to play Lucid Drums with any music they choose.
Achieving a place in the top 8 at this event is no easy task; TreeHacks is a competitive event with a selective application process. This year it brought together 1,300 students -- beginners and hackathon veterans alike -- from universities across the globe.
For some students like Gong, it was their first hackathon experience. “I’m a formal social science major student, so I didn’t have the opportunity to attend a hackathon during my undergraduate,” said Gong. “After joining CMU, the idea of attending a hackathon came up and that’s the reason I decided to take part in the TreeHacks.”
Fulton is a hackathon veteran from her undergraduate days, and even founded an all-female hackathon at the University of Pittsburgh. She submitted her application to TreeHacks over winter break with the goal of trying something new this time.
“On previous hackathon projects, I’ve always been on the engineering side,” said Fulton. “Through TreeHacks, I wanted to apply the design (UI/UX) skills I’ve been developing through the Master of Human-Computer Interaction program at CMU. The TreeHack hackathon VR and health domains were especially appealing to me as I am interested in game design and how VR can promote recovery in clinical settings.”
With experience comes the wisdom of making sure to prioritize a balance of wellness during an intense event like this. Fulton said, “I’ve been to hackathons before and left the weekend feeling exhausted and delirious. At this hackathon, our team was intentional to take time to go to sleep. I really believe that getting good sleep helps us be more creative and makes the experience more fun.”
Both of our masters students felt that their time at the HCII prepared them for their success at TreeHacks. For Fulton, it was the interdisciplinary nature of the HCII.
“I have seen that the HCII has a strong focus on cross-discipline collaboration. This has been evident with my research with Dr. Perer (co-advised by Dr. Hammer) where our small research team is comprised of students from departments across the School of Computer Science. Specific to MHCI, my coursework in Interaction Design Studio (05-651) was very applicable; the focus on team collaboration, designing for user needs and rapid prototyping, and pitching has prepared me to be a better UX Designer and was instrumental in my contribution to the hackathon team,” Fulton said.
Gong drew inspiration in designing the educational aspects for the game from the HCII courses E-learning Design Principles (05-823) and Personalized Online Learning (05-432/05-832). “I reflected on a lot of learning science principles and tried my best to implement them in this game to yield the optimal learning experience and outcome for children,” said Gong.
TreeHacks did not have traditional 1st, 2nd or 3rd place winners, but rather recognized six category winners and four overall grand prizes. The Lucid Drums team was awarded one of the grand prizes for “Most Creative Hack.” They also placed in the Top 8 (out of almost 200) teams, which provided the opportunity to present their pitch on stage to everyone at the hackathon as well as to a live online audience via livestream.
Congratulations, students.
Watch a demo of Lucid Drums below.