2022

Selective Credit Assignment

Chelu, Veronica, Borsa, Diana, Precup, Doina et al.

Understand

Efficient credit assignment is essential for reinforcement learning algorithms in both prediction and control settings.

We describe a unified view on temporal-difference algorithms for selective credit assignment.
These selective algorithms apply weightings to quantify the contribution of learning updates.
We present insights into applying weightings to value-based learning and planning algorithms, and describe their role in mediating the backward credit distribution in prediction and control.

Built on

On inductive biases in deep reinforcement learning
Original
M. Hessel, H. van Hasselt, J. Modayil, and D. Silver · 1907
Earlier work this paper cites.
Hindsight credit assignment
Original
A. Harutyunyan, W. Dabney, T. Mesnard, M. G. Azar, B. Piot, N. M. O. Heess, H. van Hasselt, G. Wayne, S. Singh, D. Precup, and R. Munos · 1912
Earlier work this paper cites.
A stochastic approximation method
H. Robbins and S. Monro · 1951
Earlier work this paper cites.
Temporal credit assignment in reinforcement learning
R. Sutton · 1984
Earlier work this paper cites.
Q-learning
C. J. Watkins and P. Dayan · 1992
Earlier work this paper cites.
Efficient learning and planning within the dyna framework
J. Peng and R. J. Williams · 1993
Earlier work this paper cites.
On step-size and bias in temporal-difference learning
R. Sutton and S. Singh · 1994
Earlier work this paper cites.
Incremental multi-step q-learning
J. Peng and R. J. Williams · 1996
Earlier work this paper cites.
An analysis of temporal-difference learning with function approximation
J. N. Tsitsiklis and B. Van Roy · 1997
Earlier work this paper cites.
Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning
R. Sutton, D. Precup, and S. Singh · 1999
Earlier work this paper cites.
Off-policy temporal-difference learning with function approximation
D. Precup, R. S. Sutton, and S. Dasgupta · 2001
Earlier work this paper cites.
Prioritized sweeping: Reinforcement learning with less data and less time
A. Moore and C. Atkeson · 2004
Earlier work this paper cites.
Fast exact planning in markov decision processes
H. B. McMahan and G. J. Gordon · 2005
Earlier work this paper cites.
Dyna-style planning with linear function approximation and prioritized sweeping
R. Sutton, C. Szepesvari, A. Geramifard, and M. Bowling · 2008
Earlier work this paper cites.
The fixed points of off-policy td
J. Z. Kolter · 2011
Earlier work this paper cites.
Gradient temporal-difference learning algorithms
H. R. Maei · 2011
Earlier work this paper cites.

Similar

The arcade learning environment: An evaluation platform for general agents
Original
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling · 2012
Cited alongside, same era.
A new Q( λ \lambda ) with interim forward view and Monte Carlo equivalence
R. S. Sutton, A. R. Mahmood, D. Precup, and H. van Hasselt · 2014
Cited alongside, same era.
True online TD( λ \lambda )
H. van Seijen and R. S. Sutton · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
D. P. Kingma and J. Ba · 2015
Cited alongside, same era.
Emphatic temporal-difference learning
Original
A. R. Mahmood, H. Yu, M. White, and R. S. Sutton · 2015
Cited alongside, same era.
Learning to predict independent of span
Original
H. van Hasselt and R. Sutton · 2015
Cited alongside, same era.
Generalized emphatic temporal difference learning: Bias-variance analysis
A. Hallak, A. Tamar, R. Munos, and S. Mannor · 2016
Cited alongside, same era.
An emphatic approach to the problem of off-policy temporal-difference learning
R. S. Sutton, A. R. Mahmood, and M. White · 2016
Cited alongside, same era.
A greedy approach to adapting the trace parameter for temporal difference learning
Original
M. White and A. White · 2016
Cited alongside, same era.
The option-critic architecture
Original
P. Bacon, J. Harb, and D. Precup · 2017
Cited alongside, same era.
Unifying task specification in reinforcement learning
M. White · 2017
Cited alongside, same era.
JAX: composable transformations of Python+NumPy programs, 2018
J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang · 2018
Cited alongside, same era.

Then

Meta-gradient reinforcement learning
Z. Xu, H. van Hasselt, and D. Silver · 2018
Later among the works it cites.
Rudder: Return decomposition for delayed rewards
Original
J. A. Arjona-Medina, M. Gillhofer, M. Widrich, T. Unterthiner, and S. Hochreiter · 2019
Later among the works it cites.
Optimizing agent behavior over long time scales by transporting value
C.-C. Hung, T. P. Lillicrap, J. Abramson, Y. Wu, M. Mirza, F. Carnevale, A. Ahuja, and G. Wayne · 2019
Later among the works it cites.
When to use parametric models in reinforcement learning?
H. van Hasselt, M. Hessel, and J. Aslanides · 2019
Later among the works it cites.
Forethought and hindsight in credit assignment
V. Chelu, D. Precup, and H. P. van Hasselt · 2020
Later among the works it cites.
Haiku: Sonnet for JAX, 2020
T. Hennigan, T. Cai, T. Norman, and I. Babuschkin · 2020
Later among the works it cites.
Optax: composable gradient transformation and optimisation, in jax!, 2020
M. Hessel, D. Budden, F. Viola, M. Rosca, E. Sezener, and T. Hennigan · 2020
Later among the works it cites.
Discor: Corrective feedback in reinforcement learning via distribution correction
Original
A. Kumar, A. Gupta, and S. Levine · 2020
Later among the works it cites.
Counterfactual credit assignment in model-free reinforcement learning
Original
T. Mesnard, T. Weber, F. Viola, S. Thakoor, A. Saade, A. Harutyunyan, W. Dabney, T. Stepleton, N. Heess, A. Guez, M. Hutter, L. Buesing, and R. Munos · 2020
Later among the works it cites.
Expected eligibility traces
Original
H. van Hasselt, S. Madjiheurem, M. Hessel, D. Silver, A. Barreto, and D. Borsa · 2020
Later among the works it cites.
A self-tuning actor-critic algorithm
T. Zahavy, Z. Xu, V. Veeriah, M. Hessel, J. Oh, H. P. van Hasselt, D. Silver, and S. Singh · 2020
Later among the works it cites.
Learning retrospective knowledge with reverse reinforcement learning
S. Zhang, V. Veeriah, and S. Whiteson · 2020
Later among the works it cites.
Preferential temporal difference learning, 2021
N. Anand and D. Precup · 2021
Later among the works it cites.
Learning expected emphatic traces for deep RL
Original
R. Jiang, S. Zhang, V. Chelu, A. White, and H. van Hasselt · 2021
Later among the works it cites.
Expected eligibility traces
H. van Hasselt, S. Madjiheurem, M. Hessel, D. Silver, A. Barreto, and D. Borsa · 2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…