Understand
Representing a dialog policy as a recurrent neural network (RNN) is attractive because it handles partial observability, infers a latent representation of state, and can be optimized with supervised learning (SL) or reinforcement learning (RL).
- For RL, a policy gradient approach is natural, but is sample inefficient.
- In this paper, we present 3 methods for reducing the number of dialogs required to optimize an RNN-based dialog policy with RL.
- The key idea is to maintain a second RNN which predicts the value of the current policy, and to apply experience replay to both networks.
Built on
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J Williams · 1992
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jurgen Schmidhuber · 1997
Earlier work this paper cites.
Reinforcement Learning: an Introduction
R Sutton and A Barto · 1998
Earlier work this paper cites.
Policy gradient methods for reinforcement learning with function approximation
Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour · 2000
Earlier work this paper cites.
Off-policy temporal-difference learning with function approximation
Doina Precup, Richard S Sutton, and Sanjoy Dasgupta · 2001
Earlier work this paper cites.
Similar
Policy gradient methods for robotics
Jan Peters and Stefan Schaal · 2006
Cited alongside, same era.
Thomas Degris, Martha White, and Richard S Sutton · 2012
Cited alongside, same era.
Learning phrase representations using RNN encoder-decoder for statistical machine translation
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Diederik Kingma and Jimmy Ba · 2014
Cited alongside, same era.
Deep recurrent q-learning for partially observable mdps
Matthew J. Hausknecht and Peter Stone · 2015
Cited alongside, same era.
Then
An empirical exploration of recurrent network architectures
Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever · 2015
Later among the works it cites.
Policy networks with two-stage training for dialogue systems
Mehdi Fatemi, Layla El Asri, Hannes Schulz, Jing He, and Kaheer Suleman · 2016
Closest in time.
Mastering the game of Go with deep neural networks and tree search
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Sander Dieleman Marc Lanctot, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis · 2016
Closest in time.
Theano: A Python framework for fast computation of mathematical expressions
Theano Development Team · 2016
Closest in time.
End-to-end lstm-based dialog control optimized with supervised and reinforcement learning
Jason D. Williams and Geoffrey Zweig · 2016
Closest in time.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…