Multimedia and Multilingual Human-Centered Content Discovery

In TraceThem, we will carry out research in algorithmic techniques of multimedia and multilingual search, working in real environments where current techniques fail in performance, generalisation and scalability. Apart from the traditional audiovisual contents (TV shows, news, movies, series...) new scenarios and types of contents have emerged in recent years (MOOCs, video blogs, tutorials...) where the automation of the search process for accessing the contents is a key aspect; processing these multimedia documents involves the added difficulty that, often, contents appear in different languages, representing a higher technological challenge, as tools adapted to different languages are needed, which is not always possible due to the lack of resources or tools to enable completely language independent content indexing. The information that we intend to extract is always within a communicative context (“from” someone and “for” someone), so the characterisation of the people involved in this context will play a central role. We will focus on finding information about people and their way of interacting (“who they are”, “what they say”, “how they communicate”, “how they are doing”), with a special interest in discovering people and content. The extraction of information related to people will be performed through audio processing, video processing and combined audio and video processing. To do this, we will focus on searching for technologies and new solutions for: multimedia content analysis, voice and face biometrics, audio segmentation and speaker diarization, detection of the emotional state and detection of people interacting. Content extraction will be primarily performed by processing audio using both language-dependent and language-independent search on speech. The scientific-technical impact and dissemination of project results will be favored by participation in international competitive evaluations related to the described issue, as these are important in the development of this project activity because they allow to use data sets related with tasks that constitute the current technological challenges. Moreover, in these competitions, common experimental frameworks are set up to enhance collaboration with other research groups and to allow comparisons of different algorithms, helping to discover the strengths and weaknesses of algorithms and developed systems.
Funded by: 
Ministerio de Economía y Competitividad
University of Vigo
Start date: 
End date: 
Number of investigators: