Audiovisual Analysis of the Verbal and Nonverbal Communication Channels

This project will delve into the research and development of techniques that would allow paving the path to a full comprehension and exploitation of the verbal and non-verbal communication channels that use facial expression, body language and hands expressivity as their signal sources.

Nonverbal communication is an interdisciplinary area where linguists, psychologists, anthropologists, sociologists and neuroscientists developed a plethora of theories. The study of non-verbal communication finds application fields in security, law enforcement, recruitment, negotiation, consumer research, medical practice or psychiatry. Non verbal communication can be defined as  the transfer and exchange of messages in any and all modalities that do not involve words. The modalities are as diverse as facial expressions, gestures and body movements, nonverbal vocals, behavior in interpersonal space, and even physiognomy (face, body, clothes). Most of the research advances done by engineers around these modalities were intended to improve Human-Computer Interaction and Affective Computing, so there are large bodies of research on gesture recognition and facial expressions of emotion.  Sign Languages are primarily based on hand gestures and a language grammar, thus, many of the techniques developed for gesture recognition applications and speech recognition have been leveraged for Sign Language Recognition (SLR). However, facial expressions and body movements, being crucial for SL, haven’t received the deserved attention in this scenario yet. 

Therefore, the project global objective is developing new algorithms, systems and datasets, based on speech and video processing, and machine learning techniques, to extract multimodal  information to allow decoding verbal and nonverbal communication channels from Spoken and Sign Languages. 

Funded by: 
Ministerio de Ciencia, Innovación y Universidades
Start date: 
End date: 
Number of investigators: