Training Recurrent Answering Units with Joint Loss Minimization for VQA

Understand

We propose a novel algorithm for visual question answering based on a recurrent deep neural network, where every module in the network corresponds to a complete answering unit with attention mechanism by itself.

• The network is optimized by minimizing loss aggregated from all the units, which share model parameters while receiving different information to compute attention probability.
• For training, our model attends to a region within image feature map, updates its memory based on the question and attended image feature, and answers the question based on its memory state.
• This procedure is performed to compute loss in each step.