Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, Alexander M. Rush
2016 · arXiv
We present a neural encoder-decoder model to convert images into presentational markup based on a scalable coarse-to-fine attention mechanism. Our method is evaluated in the context of image-to-LaTeX generation, and we introduce a new dataset of real-world rendered mathematical expressions paired with LaTeX markup. We show that unlike neural OCR techniques using CTC-based models, attention-based approaches can tackle this non-standard OCR task. Our approach outperforms classical mathematical OCR systems by a large margin on in-domain rendered data, and, with pretraining, also performs well on out-of-domain handwritten data. To reduce the inference complexity associated with the attention-based approaches, we introduce a new coarse-to-fine attention layer that selects a support region before applying attention.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
3 selected excerpts · 3 total mentions
Want to give proper credit to my former student for starting this: Yuntian Deng et al., 2016 (https://arxiv.org/abs/1609.04938). I believe this repo uses the dataset from that paper. Some recent cool work he's been doing: https://www.youtube.com/watch?v=lx1XcTdhalU.
Are you sure that extracting document structure from a screenshot is a solved task? I'm aware of "What You Get Is What You See" [1], but they are nowhere close to 100% accuracy even for reconstructing the original markup, which does not even require making inferences about the meaning of the content. You seem to be overestimating how close to practicality neural-network approaches are for most problems. Very much of the potential application space is still firmly within the domain of research. That doesn't mean what you described is completely impossible, just that you could get a PhD for pulling it off. [1] https://arxiv.org/abs/1609.04938v1 (incidentally, I noticed that the version of the paper that was accepted for ICML 2017 no longer has the HTML part)
If you somewhat familiar with DL literature, you will see this paper, while having a very interesting angle, the underlying architecture is a standard, enc-dec network, with encoder being CNN and decoding being LSTM. Such application, has been studied before: https://arxiv.org/pdf/1609.04938.pdf The above paper shows nice result that turns image to latex expression, and image to html.
More from the same topics.
William P. Thurston
In response to Jaffe and Quinn [math.HO/9307227], the author discusses forms of progress in mathematics that are not captured by formal proofs of theorems, especially in his own work in the theory of foliations and geometrization of 3-manifolds and dynamical systems.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Martin Kleppmann
The CAP Theorem is a frequently cited impossibility result in distributed systems, especially among NoSQL distributed databases.
Eric Chisolm
This is an introduction to geometric algebra, an alternative to traditional vector algebra that expands on it in two ways: 1.