Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors

Understand

Humans convey their intentions through the usage of both verbal and nonverbal behaviors during face-to-face communication.

• Speaker intentions often vary dynamically depending on different nonverbal contexts, such as vocal patterns and facial expressions.
• As a result, when modeling human language, it is essential to not only consider the literal meaning of the words but also the nonverbal contexts in which these words appear.
• To better model human language, we first model expressive nonverbal representations by analyzing the fine-grained visual and acoustic patterns that occur during word segments.