Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks

Understand

Linking human whole-body motion and natural language is of great interest for the generation of semantic representations of observed human behaviors as well as for the generation of robot behaviors based on natural language input.

• While there has been a large body of research in this area, most approaches that exist today require a symbolic representation of motions (e.g.
• in the form of motion primitives), which have to be defined a-priori or require complex segmentation algorithms.
• In contrast, recent advances in the field of neural networks and especially deep learning have demonstrated that sub-symbolic representations that can be learned end-to-end usually outperform more traditional approaches, for applications such as machine translation.