2016

Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation

Serban, Iulian Vlad, Klinger, Tim, Tesauro, Gerald et al.

Understand

We introduce the multiresolution recurrent neural network, which extends the sequence-to-sequence framework to model natural language generation as two parallel discrete stochastic processes: a sequence of high-level coarse tokens, and a sequence of natural language tokens.

There are many ways to estimate or learn the high-level coarse tokens, but we argue that a simple extraction procedure is sufficient to capture a wealth of high-level discourse semantics.
Such procedure allows training the multiresolution recurrent neural network by maximizing the exact joint log-likelihood over both sequences.
In contrast to the standard log- likelihood objective w.r.t.

Built on

Long short-term memory
Hochreiter, S. and Schmidhuber, J. (1997) · 1997
Earlier work this paper cites.
A neural probabilistic language model
Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. (2003) · 2003
Earlier work this paper cites.
Quantitative evaluation of user simulation techniques for spoken dialogue systems
Schatzmann, J., Georgila, K., and Young, S. (2005) · 2005
Earlier work this paper cites.
Manual and automatic evaluation of machine translation between european languages
Koehn, P. and Monz, C. (2006) · 2006
Earlier work this paper cites.
Chatbots: are they really useful?
Shawar, B. A. and Atwell, E. (2007) · 2007
Earlier work this paper cites.
Recurrent neural network based language model
Mikolov, T. et al · 2010
Earlier work this paper cites.
Practical recommendations for gradient-based training of deep architectures
Bengio, Y. (2012) · 2012
Earlier work this paper cites.
Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription
Boulanger-Lewandowski, N., Bengio, Y., and Vincent, P. (2012) · 2012
Earlier work this paper cites.
Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups
Hinton, G. et al · 2012
Earlier work this paper cites.
On the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., and Bengio, Y. (2012) · 2012
Earlier work this paper cites.
Generating sequences with recurrent neural networks
Original
Graves, A. (2013) · 2013
Earlier work this paper cites.

Similar

Distributed representations of words and phrases and their compositionality
Mikolov, T. et al · 2013
Cited alongside, same era.
Improved part-of-speech tagging for online conversational text with word clusters
Owoputi, O. et al · 2013
Cited alongside, same era.
Learning phrase representations using rnn encoder–decoder for statistical machine translation
Cho, K. et al · 2014
Cited alongside, same era.
Neural turing machines
Original
Graves, A., Wayne, G., and Danihelka, I. (2014) · 2014
Cited alongside, same era.
Sequence to sequence learning with neural networks
Sutskever, I., Vinyals, O., and Le, Q. V. (2014) · 2014
Cited alongside, same era.
Neural machine translation by jointly learning to align and translate
Bahdanau, D., Cho, K., and Bengio, Y. (2015) · 2015
Cited alongside, same era.
Attention-based models for speech recognition
Chorowski, J. K., Bahdanau, D., Serdyuk, D., Cho, K., and Bengio, Y. (2015) · 2015
Cited alongside, same era.
A recurrent latent variable model for sequential data
Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A., and Bengio, Y. (2015) · 2015
Cited alongside, same era.
Adam: A method for stochastic optimization
Kingma, D. and Ba, J. (2015) · 2015
Cited alongside, same era.
The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems
Lowe, R., Pow, N., Serban, I., and Pineau, J. (2015) · 2015
Cited alongside, same era.
Addressing the rare word problem in neural machine translation
Luong, M. T., Sutskever, I., Le, Q. V., Vinyals, O., and Zaremba, W. (2015) · 2015
Cited alongside, same era.
Data-driven response generation in social media
Ritter, A., Cherry, C., and Dolan, W. B. (2011a)
Cited in the paper.

Then

A survey of available corpora for building data-driven dialogue systems
Original
Serban, I. V., Lowe, R., Charlin, L., and Pineau, J. (2015) · 2015
Later among the works it cites.
A hierarchical recurrent encoder-decoder for generative context-aware query suggestion
Sordoni, A., Bengio, Y., Vahabi, H., Lioma, C., Simonsen, J. G., and Nie, J.-Y. (2015) · 2015
Later among the works it cites.
Memory networks
Weston, J., Chopra, S., and Bordes, A. (2015) · 2015
Later among the works it cites.
A latent variable recurrent neural network for discourse relation language models
Ji, Y., Haffari, G., and Eisenstein, J. (2016) · 2016
Closest in time.
Ask me anything: Dynamic memory networks for natural language processing
Kumar, A., Irsoy, O., Su, J., Bradbury, J., English, R., Pierce, B., Ondruska, P., Gulrajani, I., and Socher, R. (2016) · 2016
Closest in time.
A diversity-promoting objective function for neural conversation models
Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B. (2016) · 2016
Closest in time.
How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation
Original
Liu, C.-W., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., and Pineau, J. (2016) · 2016
Closest in time.
Theano: A Python framework for fast computation of mathematical expressions
Original
Theano Development Team (2016) · 2016
Closest in time.
A network-based end-to-end trainable task-oriented dialogue system
Original
Wen, T.-H., Gasic, M., Mrksic, N., Rojas-Barahona, L. M., Su, P.-H., Ultes, S., Vandyke, D., and Young, S. (2016) · 2016
Closest in time.
POMDP-based statistical spoken dialog systems: A review
Young, S., Gasic, M., Thomson, B., and Williams, J. D. (2013) · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…