2021

Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification

Eysenbach, Benjamin, Levine, Sergey, Salakhutdinov, Ruslan

Understand

Reinforcement learning (RL) algorithms assume that users specify tasks by manually writing down a reward function.

However, this process can be laborious and demands considerable technical expertise.
Can we devise RL algorithms that instead enable users to specify tasks simply by providing examples of successful outcomes? In this paper, we derive a control algorithm that maximizes the future probability of these successful outcome examples.
Prior work has approached similar problems with a two-stage process, first learning a reward function and then optimizing this reward function using another RL algorithm.

Built on

Reinforcement learning without ground-truth state
Original
Lin, X., Baweja, H. S., and Held, D. (2019) · 1905
Earlier work this paper cites.
Task-relevant adversarial imitation learning
Original
Zolna, K., Reed, S., Novikov, A., Colmenarej, S. G., Budden, D., Cabi, S., Denil, M., de Freitas, N., and Wang, Z. (2019) · 1910
Earlier work this paper cites.
Positive-unlabeled reward learning
Original
Xu, D. and Denil, M. (2019) · 1911
Earlier work this paper cites.
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · 1988
Earlier work this paper cites.
Alvinn: An autonomous land vehicle in a neural network
Pomerleau, D. A. (1989) · 1989
Earlier work this paper cites.
Function optimization using connectionist reinforcement learning algorithms
Williams, R. J. and Peng, J. (1991) · 1991
Earlier work this paper cites.
Learning to achieve goals
Kaelbling, L. P. (1993) · 1993
Earlier work this paper cites.
On the convergence of stochastic iterative dynamic programming algorithms
Jaakkola, T., Jordan, M. I., and Singh, S. P. (1994) · 1994
Earlier work this paper cites.
Td models: Modeling the world at a mixture of time scales
Sutton, R. S. (1995) · 1995
Earlier work this paper cites.
Apprenticeship learning via inverse reinforcement learning
Abbeel, P. and Ng, A. Y. (2004) · 2004
Earlier work this paper cites.
D4rl: Datasets for deep data-driven reinforcement learning
Original
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. (2020) · 2004
Earlier work this paper cites.
Maximum margin planning
Ratliff, N. D., Bagnell, J. A., and Zinkevich, M. A. (2006) · 2006
Earlier work this paper cites.
Learning classifiers from only positive and unlabeled data
Elkan, C. and Noto, K. (2008) · 2008
Earlier work this paper cites.
Maximum entropy inverse reinforcement learning
Ziebart, B. D., Maas, A. L., Bagnell, J. A., and Dey, A. K. (2008) · 2008
Earlier work this paper cites.

Similar

A reduction of imitation learning and structured prediction to no-regret online learning
Ross, S., Gordon, G., and Bagnell, D. (2011) · 2011
Cited alongside, same era.
Offline learning from demonstrations and unlabeled experience
Original
Zolna, K., Novikov, A., Konyushkova, K., Gulcehre, C., Wang, Z., Aytar, Y., Denil, M., de Freitas, N., and Reed, S. (2020) · 2011
Cited alongside, same era.
Semi-supervised reward learning for offline reinforcement learning
Original
Konyushkova, K., Zolna, K., Aytar, Y., Novikov, A., Reed, S., Cabi, S., and de Freitas, N. (2020) · 2012
Cited alongside, same era.
Universal value function approximators
Schaul, T., Horgan, D., Gregor, K., and Silver, D. (2015) · 2015
Cited alongside, same era.
Maximum entropy deep inverse reinforcement learning
Original
Wulfmeier, M., Ondruska, P., and Posner, I. (2015) · 2015
Cited alongside, same era.
Taming the noise in reinforcement learning via soft updates
Fox, R., Pakman, A., and Tishby, N. (2016) · 2016
Cited alongside, same era.
Generative adversarial imitation learning
Ho, J. and Ermon, S. (2016) · 2016
Cited alongside, same era.
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016) · 2016
Cited alongside, same era.
The feeling of success: Does touch sensing help predict grasp outcomes?
Calandra, R., Owens, A., Upadhyaya, M., Yuan, W., Lin, J., Adelson, E. H., and Levine, S. (2017) · 2017
Cited alongside, same era.
Addressing function approximation error in actor-critic methods
Fujimoto, S., Hoof, H., and Meger, D. (2018) · 2018
Cited alongside, same era.
Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018) · 2018
Cited alongside, same era.
Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning
Kostrikov, I., Agrawal, K. K., Dwibedi, D., Levine, S., and Tompson, J. (2018) · 2018
Cited alongside, same era.

Then

Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S. (2018) · 2018
Later among the works it cites.
Few-shot goal inference for visuomotor learning and planning
Xie, A., Singh, A., Levine, S., and Finn, C. (2018) · 2018
Later among the works it cites.
TF-Agents: A library for reinforcement learning in tensorflow
Guadarrama, S., Korattikara, A., Ramirez, O., Castro, P., Holly, E., Fishman, S., Wang, K., Gonina, E., Wu, N., Kokiopoulou, E., Sbaiz, L., Smith, J., Bartók, G., Berent, J., Harris, C., Vanhoucke, V., and Brevdo, E. (2018) · 2019
Later among the works it cites.
Off-policy evaluation via off-policy classification
Irpan, A., Rao, K., Bousmalis, K., Harris, C., Ibarz, J., and Levine, S. (2019) · 2019
Later among the works it cites.
Imitation learning via off-policy distribution matching
Kostrikov, I., Nachum, O., and Tompson, J. (2019) · 2019
Later among the works it cites.
End-to-end robotic reinforcement learning without reward engineering
Singh, A., Yang, L., Finn, C., and Levine, S. (2019) · 2019
Later among the works it cites.
A practical approach to insertion with variable socket position using deep reinforcement learning
Vecerik, M., Sushkov, O., Barker, D., Rothörl, T., Hester, T., and Scholz, J. (2019) · 2019
Later among the works it cites.
Curl: Contrastive unsupervised representations for reinforcement learning
Laskin, M., Srinivas, A., and Abbeel, P. (2020) · 2020
Later among the works it cites.
Multifingered grasp planning via inference in deep neural networks: Outperforming sampling by learning differentiable models
Lu, Q., Van der Merwe, M., Sundaralingam, B., and Hermans, T. (2020) · 2020
Later among the works it cites.
DisCo RL: Distribution-Conditioned Reinforcement Learning for General-Purpose Policies
Nasiriany, S. (2020) · 2020
Later among the works it cites.
{SQIL}: Imitation learning via reinforcement learning with sparse rewards
Reddy, S., Dragan, A. D., and Levine, S. (2020) · 2020
Later among the works it cites.
Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. (2020) · 2020
Later among the works it cites.
C-learning: Learning to achieve goals via recursive classification
Eysenbach, B., Salakhutdinov, R., and Levine, S. (2021) · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…