Translating Math Formula Images to LaTeX Sequences Using Deep Neural Networks with Sequence-level Training

Understand

In this paper we propose a deep neural network model with an encoder-decoder architecture that translates images of math formulas into their LaTeX markup sequences.

• The encoder is a convolutional neural network (CNN) that transforms images into a group of feature maps.
• To better capture the spatial relationships of math symbols, the feature maps are augmented with 2D positional encoding before being unfolded into a vector.
• The decoder is a stacked bidirectional long short-term memory (LSTM) model integrated with the soft attention mechanism, which works as a language model to translate the encoder output into a sequence of LaTeX tokens.