2020

Differentiable Weighted Finite-State Transducers

Hannun, Awni, Pratap, Vineel, Kahn, Jacob et al.

Understand

We introduce a framework for automatic differentiation with weighted finite-state transducers (WFSTs) allowing them to be used dynamically at training time.

Through the separation of graphs from operations on graphs, this framework enables the exploration of new structured loss functions which in turn eases the encoding of prior knowledge into learning algorithms.
We show how the framework can combine pruning and back-off in transition models with various sequence-level loss functions.
We also show how to learn over the latent decomposition of phrases into word pieces.

Built on

Estimation of probabilities from sparse data for the language model component of a speech recognizer
Slava Katz · 1987
Earlier work this paper cites.
A tutorial on hidden markov models and selected applications in speech recognition
Lawrence R Rabiner · 1989
Earlier work this paper cites.
Une Approche théorique de l’Apprentissage Connexionniste: Applications à la Reconnaissance de la Parole
Léon Bottou · 1991
Earlier work this paper cites.
The design for the wall street journal-based csr corpus
Douglas B Paul and Janet Baker · 1992
Earlier work this paper cites.
Weighted rational transductions and their application to human language processing
Fernando Pereira, Michael Riley, and Richard Sproat · 1994
Earlier work this paper cites.
Document analysis with transducers, July 1996
Léon Bottou, Yoshua Bengio, and Yann LeCun · 1996
Earlier work this paper cites.
Global training of document processing systems using graph transformer networks
Léon Bottou, Yann Le Cun, and Yoshua Bengio · 1997
Earlier work this paper cites.
Finite-state transducers in language and speech processing
Mehryar Mohri · 1997
Earlier work this paper cites.
Speech & language processing
Dan Jurafsky · 2000
Earlier work this paper cites.
The design principles of a weighted finite-state transducer library
Mehryar Mohri, Fernando Pereira, and Michael Riley · 2000
Earlier work this paper cites.
Conditional random fields: Probabilistic models for segmenting and labeling sequence data
John Lafferty, Andrew McCallum, and Fernando CN Pereira · 2001
Earlier work this paper cites.
The iam-database: an english sentence database for offline handwriting recognition
U-V Marti and Horst Bunke · 2002
Earlier work this paper cites.
Weighted finite-state transducers in speech recognition
Mehryar Mohri, Fernando Pereira, and Michael Riley · 2002
Earlier work this paper cites.
Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber · 2006
Earlier work this paper cites.
Openfst: A general and efficient weighted finite-state transducer library
Cyril Allauzen, Michael Riley, Johan Schalkwyk, Wojciech Skut, and Mehryar Mohri · 2007
Earlier work this paper cites.

Similar

The OCRopus open source OCR system
Thomas M Breuel · 2008
Cited alongside, same era.
A novel connectionist system for unconstrained handwriting recognition
Alex Graves, Marcus Liwicki, Santiago Fernández, Roman Bertolami, Horst Bunke, and Jürgen Schmidhuber · 2008
Cited alongside, same era.
Speech recognition with weighted finite-state transducers
Mehryar Mohri, Fernando Pereira, and Michael Riley · 2008
Cited alongside, same era.
Applications of weighted automata in natural language processing
Kevin Knight and Jonathan May · 2009
Cited alongside, same era.
Weighted automata algorithms
Mehryar Mohri · 2009
Cited alongside, same era.
Torch7: A matlab-like environment for machine learning
Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet · 2011
Cited alongside, same era.
The kaldi speech recognition toolkit
Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al · 2011
Cited alongside, same era.
Sequence-discriminative training of deep neural networks
Karel Veselỳ, Arnab Ghoshal, Lukás Burget, and Daniel Povey · 2013
Cited alongside, same era.
Dropout improves recurrent neural networks for handwriting recognition
Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour · 2014
Cited alongside, same era.
LibriSpeech: an ASR corpus based on public domain audio books
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur · 2015
Cited alongside, same era.
Sequence level training with recurrent neural networks
Original
Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba · 2015
Cited alongside, same era.
Chainer: a next-generation open source framework for deep learning
Seiya Tokui, Kenta Oono, and Shohei Hido · 2015
Cited alongside, same era.

Then

Neural machine translation of rare words with subword units
Rico Sennrich, Barry Haddow, and Alexandra Birch · 2016
Later among the works it cites.
Handwriting recognition with large multidimensional long short-term memory recurrent neural networks
Paul Voigtlaender, Patrick Doetsch, and Hermann Ney · 2016
Later among the works it cites.
Latent sequence decompositions
William Chan, Yu Zhang, Quoc Le, and Navdeep Jaitly · 2017
Later among the works it cites.
Image-to-image translation with conditional adversarial networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros · 2017
Later among the works it cites.
Gram-ctc: automatic unit selection and target decomposition for sequence labelling
Hairong Liu, Zhenyao Zhu, Xiangang Li, and Sanjeev Satheesh · 2017
Later among the works it cites.
State-of-the-art speech recognition with sequence-to-sequence models
Chung-Cheng Chiu, Tara N Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron J Weiss, Kanishka Rao, Ekaterina Gonina, et al · 2018
Later among the works it cites.
Subword regularization: Improving neural network translation models with multiple subword candidates
Taku Kudo · 2018
Later among the works it cites.
Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Taku Kudo and John Richardson · 2018
Later among the works it cites.
Connectionist temporal classification with maximum entropy regularization
Hu Liu, Sheng Jin, and Changshui Zhang · 2018
Later among the works it cites.
A fully differentiable beam search decoder
Ronan Collobert, Awni Hannun, and Gabriel Synnaeve · 2019
Later among the works it cites.
Sequence-to-sequence speech recognition with time-depth separable convolutions
Awni Hannun, Ann Lee, Qiantong Xu, and Ronan Collobert · 2019
Later among the works it cites.
Specaugment: A simple data augmentation method for automatic speech recognition
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le · 2019
Later among the works it cites.
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al · 2019
Later among the works it cites.
Wav2letter++: A fast open-source speech recognition system
Vineel Pratap, Awni Hannun, Qiantong Xu, Jeff Cai, Jacob Kahn, Gabriel Synnaeve, Vitaliy Liptchinsky, and Ronan Collobert · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…