Using Active Speaker Faces for Diarization in TV shows

Understand

Speaker diarization is one of the critical components of computational media intelligence as it enables a character-level analysis of story portrayals and media content understanding.

• Automated audio-based speaker diarization of entertainment media poses challenges due to the diverse acoustic conditions present in media content, be it background music, overlapping speakers, or sound effects.
• At the same time, speaking faces in the visual modality provide complementary information and not prone to the errors seen in the audio modality.
• In this paper, we address the problem of speaker diarization in TV shows using the active speaker faces.