Interaction design & machine learning
Beyond the mask
ADI INDEX - SELECTION TARGA GIOVANI
objective
This year, as a result of a health emergency situation, our daily habits have been disrupted. Anti-covid masks have been a huge problem for deaf people, because they greatly reduce the ability to communicate, filtering and hiding our expressive and communicative nature. The aim of this project is to break down this communicative wall and provide an expressive and constantly-available medium, that reveals words and emotions, through the power of Machine Learning.
WHAT IS
The project concept is an app that makes use of the camera and the microphone of a smartphone device. The system, while framing the interlocutor, decodes the verbal language and non-verbal communication that is expressed through subtitles and colored shapes popping out on the face of the interlocutor.
Words are not enough to globally convey what you want to communicate to others. FaceApi’s algorithm allows the face to be framed and recognized, P5 speech library allows the text to be recorded and transcribed. As the interlocutor speaks, Sentiment Analysis is activated, which changes the shape to the frame on the face, based on the semantics of the text, to the positive (an oval) and negative (a rectangle) range, respectively. Determining the color change is the facial expression; the emotion will be detected by the full version of FaceApi, trained on the face covered by the mask. Finally, the tone and intensity of the voice will be able to change the font size.
Prototype
The prototype is composed of three interconnected autonomous modules. The p5 speech library listens to and transcribes the conversation, the Sentiment Analysis algorithm analyzes the semantics of the text and derives a prediction that is expressed on a scale ranging from 0 to 100, identifying three parameters (positive, neutral, and negative), and the FaceApi system, on the other hand, recognizes facial coordinates and facial expression on a model trained to recognize the face covered by the mask.
The color red was associated with the negative sentiment detected by Sentiment Analysis, yellow for positive sentiment, and blue for neutral. In addition, frame movement on the face was added to the prototype. It changes according to the vowels in the words. For the time being, the movement is not smooth; the goal is to mimic the movement of the lips while speaking.
