NEWMT
Overview
Neurodiverse Educational Wearable Music Tech was a National Science Foundation funded project to increase STEAM education access to neurodiverse children and their teachers.
Goal & Team
The goal was to explore new methodologies of instruction that encouraged a body-mind connection. The final product was embodied musical instruments coupled with computation thinking curriculum. I worked with a team of neurodiversity and education experts as well as an audio programmer.
Details
Role: UI/UX Designer and Researcher, Programmer, Web developer, Co-writer
Duration: 1 year on design and implementation; 2 years on team
Tools: HTML, CSS, JavaScript, RNBO (audio framework), Arduino, M5Stick Sensors
Research
First, I conducted Contextual Inquiry to learn what forms of interaction would benefit a neurodiverse classroom. From here developing the initial instrument designs were informed by our teachers' needs.
Throughout the project, I conducted mainly qualitative research (Interviews, Focus Groups, and Usability Testing) since quantitative testing had the tendency to cause anxiety for some neurodiverse students. We had quiet rooms at our in-person events, and they were used a lot during any form of quantitative inquiry.
Findings
I enjoyed watching student engagement rates increase with each new design iteration (300% increase in total). By the final version students would even forgo their free time during sessions to make amazing creations like horror generators, peaceful soundscapes, or full-body air-drums. Students even vocalized their desire for school to be more like our curriculum.
The main findings in my research revolved around student agency lacking in Ed-Tech. This insight was the key to unlocking student retention, and I think it's why many Ed-Tech products become viewed as homework rather than creative exploration. We need students to feel like our tools not only help them learn, but also encourage them to grow and express themselves uniquely.
Design
I start my design process with pen and paper. The speed of prototyping and rapid feedback cycles make it a necessary part of my process. It helped to have my notebook on me during teacher interviews, so I could draft and propose new interface ideas quickly.
Iterations
1) The first version of the application focused only on functionality.
Web-audio instruments using RNBO
Bluetooth sensor connection utilizing BLE MIDI protocol
Separate instrument pages to align with curriculum lesson
Students could move freely and interact with the instruments
2) The next iteration added interactive UI elements.
Easier for students to see what their sensors are doing
Clear UI to guide users to what they want
3) The last iteration added more student agency which was the key to a more engaging curriculum and higher student retention.
Semi-modular design to encourage student to create their own instruments
Added mouse-mode and inversion to increase the number of interactions available
Visualize the incoming sensor data
Finalized six instruments
Styles & Branding
In the styling and branding, I focused on clear colors, strong readable typography, and redundant cues. The main accent colors, orange and purple, elicit creativity and calmness respectively. They also contrast in all types of color blindness.
I chose the typography with dyslexia in mind. My choices are not better than a font like Open Dyslexia; however, I think they find a balance between aesthetic and accessibility. In general, I used strong but playful choice for type, icons, shape design, and color. I wanted the application to feel fun but empowering.
Interactivity
Interactivity and motion were essential to the design to give our users the visual feedback they needed to predictably create embodied sound. Every instrument and parameter have a visual component that reacts in real time to user input or motion data. My favorite is the ear shaped volume slider that grows as the instrument gets louder!
Below are some examples, including our style guide, motion demonstrations, and our logo process (the W and M in NEWMT arranged like the neurodiversity infinity symbol).
Reflections
I am proud that we created a medium for discovery, expression, and learning through embodied exploration. We succeeded in our goals of pushing new methodologies for stem learning and creating an interactive experience to accompany them.
Looking Forward
There are still improvements to be made at NEWMT on the technology side.
Move away from Arduino sensors since there were many connection difficulties (possibly use a phone app with built in sensors).
Move away from lazy loading audio instruments since users preferred a single longer loading screen to hiccups in the experience.
Looking Back
I designed four unique iterations and all assets
I conducted 40+ teacher interviews
I co-facilitated 6 workshops and 3 in-person events
I co-wrote audio code
I designed, researched, and programmed the application in house using a novel audio framework
I created 18 instructional videos for teachers to get started
I assisted with curriculum and paper writing
