Teaching Machines to Code: Neural Markup Generation with Visual Attention

02/15/2018

∙

by   Sumeet S. Singh, et al.

∙

0

∙

share

We present a deep recurrent neural network model with soft visual attention that learns to generate LaTeX markup of real-world math formulas given their images. Applying neural sequence generation techniques that have been very successful in the fields of machine translation and image/handwriting/speech captioning, recognition, transcription and synthesis, we construct an image-to-markup model that learns to produce syntactically and semantically correct LaTeX markup code of over 150 words long and achieves a BLEU score of 89 demonstrate that the model learns to scan the image left-right / up-down much as a human would read it.