2020

ScanSSD: Scanning Single Shot Detector for Mathematical Formulas in PDF Document Images

Mali, Parag, Kukkadapu, Puneeth, Mahdavi, Mahshad et al.

Understand

We introduce the Scanning Single Shot Detector (ScanSSD) for locating math formulas offset from text and embedded in textlines.

ScanSSD uses only visual features for detection: no formatting or typesetting information such as layout, font, or character labels are employed.
Given a 600 dpi document page image, a Single Shot Detector (SSD) locates formulas at multiple scales using sliding windows, after which candidate detections are pooled to obtain page-level results.
For our experiments we use the TFD-ICDAR2019v2 dataset, a modification of the GTDB scanned math article collection.

Built on

G. Nagy and S. Seth, “Hierarchical representation of optically scanned documents,” in Proc. Seventh Int’l Conf. Pattern Recognition , Montreal, Canada, 1984, pp. 347–349
1984
Earlier work this paper cites.
B. Chaudhuri and U. Garain, “An approach for processing mathematical expressions in printed document,” in International Workshop on Document Analysis Systems . Springer, 1998, pp. 310–321
1998
Earlier work this paper cites.
U. Garain and B. Chaudhuri, “A syntactic approach for processing mathematical expressions in printed documents,” in Proceedings 15th International Conference on Pattern Recognition. ICPR-2000 , vol. 4. IEEE, 2000, pp. 523–526
2000
Earlier work this paper cites.
A. Kacem, A. Belaïd, and M. B. Ahmed, “Automatic extraction of printed mathematical formulas using fuzzy logic and propagation of context,” International Journal on Document Analysis and Recognition , vol. 4, no. 2, pp. 97–108, 2001
2001
Earlier work this paper cites.
M. Suzuki, F. Tamari, R. Fukuda, S. Uchida, and T. Kanahori, “Infty: an integrated ocr system for mathematical documents,” in Proceedings of the 2003 ACM symposium on Document engineering . ACM, 2003, pp. 95–104
2003
Earlier work this paper cites.
D. M. Drake and H. S. Baird, “Distinguishing mathematics notation from english text using computational geometry,” in Eighth International Conference on Document Analysis and Recognition (ICDAR’05) . IEEE, 2005, pp. 1270–1274
2005
Earlier work this paper cites.
U. Garain and B. B. Chaudhuri, “Ocr of printed mathematical expressions,” in Digital Document Processing . Springer, 2007, pp. 235–259
2007
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . IEEE, 2009, pp. 248–255
2009
Earlier work this paper cites.
X. Lin, L. Gao, Z. Tang, X. Lin, and X. Hu, “Mathematical formula identification in pdf documents,” in 2011 International Conference on Document Analysis and Recognition . IEEE, 2011, pp. 1419–1423
2011
Earlier work this paper cites.
X. Lin, L. Gao, Z. Tang, X. Hu, and X. Lin, “Identification of embedded mathematical formulas in pdf documents using svm,” in Document Recognition and Retrieval XIX , vol. 8297. International Society for Optics and Photonics, 2012, p. 82970D
2012
Earlier work this paper cites.
F. Alvaro and R. Zanibbi, “A shape-based layout descriptor for classifying spatial relationships in handwritten math,” in Proceedings of the 2013 ACM symposium on Document engineering . ACM, 2013, pp. 123–126
2013
Earlier work this paper cites.

Similar

K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014
Original
2014
Cited alongside, same era.
R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2014, pp. 580–587
2014
Cited alongside, same era.
R. Girshick, “Fast r-cnn,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1440–1448
2015
Cited alongside, same era.
S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Advances in neural information processing systems , 2015, pp. 91–99
2015
Cited alongside, same era.
C. Clark and S. Divvala, “Pdffigures 2.0: Mining figures from research papers,” in 2016 IEEE/ACM Joint Conference on Digital Libraries (JCDL) . IEEE, 2016, pp. 143–152
2016
Cited alongside, same era.
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “SSD: Single shot multibox detector,” in European conference on computer vision . Springer, 2016, pp. 21–37
2016
Cited alongside, same era.
Y. Deng, A. Kanervisto, J. Ling, and A. M. Rush, “Image-to-markup generation with coarse-to-fine attention,” arXiv preprint arXiv:1609.04938 , 2016
Original
2016
Cited alongside, same era.
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 779–788
2016
Cited alongside, same era.
M. Condon, “Applying hierarchical contextual parsing with visual density and geometric features to typeset formula recognition,” Master’s thesis, Rochester Institute of Technology, Rochester, NY, USA, 2017
2017
Cited alongside, same era.
J. Zhang, J. Du, S. Zhang, D. Liu, Y. Hu, J. Hu, S. Wei, and L. Dai, “Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition,” Pattern Recognition , vol. 71, pp. 196–206, 2017
2017
Cited alongside, same era.
K. Iwatsuki, T. Sagara, T. Hara, and A. Aizawa, “Detecting in-line mathematical expressions in scientific documents,” in Proceedings of the 2017 ACM Symposium on Document Engineering . ACM, 2017, pp. 141–144
2017
Cited alongside, same era.
K. Davila, R. Joshi, S. Setlur, V. Govindaraju, and R. Zanibbi, “Tangent-v: Math formula image search using line-of-sight graphs,” in ECIR , ser. LNCS, vol. 11437, pp. 681–695
Cited in the paper.

Then

L. Gao, X. Yi, Y. Liao, Z. Jiang, Z. Yan, and Z. Tang, “A deep learning-based formula detection method for pdf documents,” in 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) , vol. 1. IEEE, 2017, pp. 553–558
2017
Later among the works it cites.
J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama et al. , “Speed/accuracy trade-offs for modern convolutional object detectors,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 7310–7311
2017
Later among the works it cites.
M. Liao, B. Shi, X. Bai, X. Wang, and W. Liu, “Textboxes: A fast text detector with a single deep neural network,” in Thirty-First AAAI Conference on Artificial Intelligence , 2017
2017
Later among the works it cites.
T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 2980–2988
2017
Later among the works it cites.
J. Zhang, J. Du, and L. Dai, “Track, attend, and parse (tap): An end-to-end framework for online handwritten mathematical expression recognition,” IEEE Transactions on Multimedia , vol. 21, no. 1, pp. 221–233, 2018
2018
Later among the works it cites.
Z. Yu, S. Lyu, Y. Lu, and P. S. Wang, “A fusion strategy for the single shot text detector,” in 2018 24th International Conference on Pattern Recognition (ICPR) . IEEE, 2018, pp. 3687–3691
2018
Later among the works it cites.
W. Ohyama, M. Suzuki, and S. Uchida, “Detecting mathematical expressions in scientific document images using a u-net trained on a diverse dataset,” IEEE Access , vol. 7, pp. 144 030–144 042, 2019
2019
Later among the works it cites.
M. Mahdavi, R. Zanibbi, H. Mouchere, and U. Garain, “ICDAR 2019 CROHME + TFD: Competition on recognition of handwritten mathematical expressions and typeset formula detection,” in ICDAR 2019) . IEEE
2019
Later among the works it cites.
M. Mahdavi, M. Condon, K. Davila, and R. Zanibbi, “LPGA: Line-of-sight parsing with graph-based attention for math formula recognition,” in Proc. International Conference on Document Analysis and Recognition . Sydney, Australia: IAPR, September 2019, pp. 647–654
2019
Later among the works it cites.
L. Pfahler, J. Schill, and K. Morik, “The search for equations - learning to identify similarities between mathematical expressions,” in Proc. ECML-PKDD , 2019
2019
Later among the works it cites.
P. Mali, “Scanning single shot detector for math in document images,” Master’s thesis, Rochester Institute of Technology, Rochester, NY, USA, August 2019
2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…