Teaching Machines to Code: Neural Markup Generation with Visual Attention

by   Sumeet S. Singh, et al.

We present a deep recurrent neural network model with soft visual attention that learns to generate LaTeX markup of real-world math formulas given their images. Applying neural sequence generation techniques that have been very successful in the fields of machine translation and image/handwriting/speech captioning, recognition, transcription and synthesis, we construct an image-to-markup model that learns to produce syntactically and semantically correct LaTeX markup code of over 150 words long and achieves a BLEU score of 89 demonstrate that the model learns to scan the image left-right / up-down much as a human would read it.


page 1

page 2

page 3

page 4


Sequence Modeling with Unconstrained Generation Order

The dominant approach to sequence generation is to produce a sequence in...

Multiple Object Recognition with Visual Attention

We present an attention-based model for recognizing multiple objects in ...

Summarize and Generate to Back-translate: Unsupervised Translation of Programming Languages

Back-translation is widely known for its effectiveness for neural machin...

Does Multimodality Help Human and Machine for Translation and Image Captioning?

This paper presents the systems developed by LIUM and CVC for the WMT16 ...

Towards Neural Machine Translation with Latent Tree Attention

Building models that take advantage of the hierarchical structure of lan...

Recurrent Soft Attention Model for Common Object Recognition

We propose the Recurrent Soft Attention Model, which integrates the visu...

Approximating How Single Head Attention Learns

Why do models often attend to salient words, and how does this evolve th...

Code Repositories


Solution to im2latex request for research of openai

view repo

Please sign up or login with your details

Forgot password? Click here to reset