Chloe and Shannon are finalists from our first Neosensory developer contest. We are delighted to host the following guest write-up from them about their extraordinary submission!
When I first saw a demo for Neosensory’s vest prototype back in 2018, my mind was mystified about the seemingly endless possibilities for data to feed into the haptic wristband. As our experience of reality evolves with new technology, the line between human and computer has become increasingly blurred, creating incredible opportunities to augment and improve our consciousness in previously inconceivable ways.
After hearing about the innovation challenge to build new sensory substitution applications with Buzz, I reached out to my tech-wiz of a friend, Shannon, to partner with. While brainstorming, we were intrigued by Buzz’s ability to seamlessly translate sound into haptic feedback. We wanted to create a product that fit with Neosensory’s initial premise: reimagining the way sensation is delivered to accommodate people that experience it differently. After pondering the myriad possibilities and applications of the Buzz, we landed on autism spectrum disorder (ASD) as a viable option because individuals with ASD struggle with emotional perception due to a difficulty in interpreting the modality it is expressed in. By adjusting the type of emotional input people take in to a more objective signal, haptic vibration, we hope to help people with ASD learn to better discern the emotions of people they talk to.
We created Valence, an app that pairs with a person’s Buzz to help them interpret the emotional valence of speech through vibrations on their wrist. Using the same functionality as the classic Buzz technology, our app senses emotions from vocal pitch data picked up by your smart phone’s microphone. By receiving audio input and quickly converting audio signals into frequency diagrams, our data pipeline and machine learning model are able to determine the emotional valence of a statement, the current possibilities of which are neutral, surprise, disgust, happy, sad, fear, and angry. Valence offers a non-invasive tool to assist individuals with ASD in learning to interpret emotions of the people they talk to, and an opportunity for any person to augment their perception of emotion and strengthen their conversations and relationships.
As full-time undergraduate students at the University of Southern California, Shannon and I worked tirelessly over many late nights after class to get a viable product and demo up and running. I pulled information and research from my neuroscience background to help understand underlying causes and issues resulting from the inability to distinguish emotion, while Shannon pulled from her computer science and machine learning background to find and process the data and to subsequently build our main model to predict the sentiment of a statement based on vocal patterns.
We plan to continue changing the model and adding training data to increase its accuracy. We hope to also add a natural language processing pipeline to run concurrently which would allow for quick transcription straight to the device to go along with the audio-based emotional classification. This transcription can be run through a sentiment classifier to add an extra level of accuracy to the final output. By focusing on the speech patterns rather than solely on the words of the spoken statement, we’re aiming to build a more accurate and useful model to help those with ASD. We are also working on building an app GUI for both Android and iOS platforms to allow for accessibility and ease of use.
We’d love to hear any questions or comments from those who may be interested in our application, or those who may benefit from an app such as this!
Interested in building your own amazing sensory expanding applications? Dive in by checking out our developer site and developer posts on our blog. If you have any questions regarding this project or a project you’re working on, visit the Neosensory developer Slack. If you create your own project and would like to share it, email us at firstname.lastname@example.org – we’ll feature select projects on our site in the coming months!