2023

A Definition of Open-Ended Learning Problems for Goal-Conditioned Agents

Sigaud, Olivier, Baldassarre, Gianluca, Colas, Cedric et al.

Understand

A lot of recent machine learning research papers have ``open-ended learning'' in their title.

But very few of them attempt to define what they mean when using the term.
Even worse, when looking more closely there seems to be no consensus on what distinguishes open-ended learning from related concepts such as continual learning, lifelong learning or autotelic learning.
In this paper, we contribute to fixing this situation.

Built on

Skew-fit: State-covering self-supervised reinforcement learning
Original
Pong, V. H., Dalal, M., Lin, S., Nair, A., Bahl, S., and Levine, S. (2019) · 1903
Earlier work this paper cites.
Computing machinery and intelligence
Turing, A. M. (1950) · 1950
Earlier work this paper cites.
Steps towards genuine artificial intelligence
Reader, A. V. (1969) · 1969
Earlier work this paper cites.
Making the world differentiable: on using fully recurrent self-supervised neural networks for dynamic reinforcement learning and planning in non-stationary environments
Schmidhuber, J. (1990) · 1990
Earlier work this paper cites.
The artificial evolution of behaviour
Meyer, J.-A. and Wilson, S. W. (1991) · 1991
Earlier work this paper cites.
Curious model-building control systems
Schmidhuber, J. (1991) · 1991
Earlier work this paper cites.
Beyond modularity: A developmental perspective on cognitive science
Karmiloff-Smith, A. (1994) · 1994
Earlier work this paper cites.
Creativity and artificial intelligence
Boden, M. A. (1998) · 1998
Earlier work this paper cites.
Computational autonomous mental development: A white paper for suggesting a new initiative
Weng, J., McClelland, J., Pentland, A., Sporns, O., Stockman, I., Sur, M., and Thelen, E. (2000) · 2000
Earlier work this paper cites.
A planning modular neural-network robot for asynchronous multi-goal navigation tasks
Baldassarre, G. (2001) · 2001
Earlier work this paper cites.
Autonomous mental development by robots and animals
Weng, J., McClelland, J., Pentland, A., Sporns, O., Stockman, I., Sur, M., and Thelen, E. (2001) · 2001
Earlier work this paper cites.
Developmental robotics: a survey
Lungarella, M., Metta, G., Pfeifer, R., and Sandini, G. (2003) · 2003
Earlier work this paper cites.
Open-ended artificial evolution
Standish, R. K. (2003) · 2003
Earlier work this paper cites.
Intrinsically motivated learning of hierarchical collections of skills
Barto, A. G., Singh, S., and Chentanez, N. (2004) · 2004
Earlier work this paper cites.
Intrinsically motivated reinforcement learning
Singh, S. P., Barto, A. G., and Chentanez, N. (2004) · 2004
Earlier work this paper cites.
Ongoing emergence: A core concept in epigenetic robotics
Prince, C., Helder, N., and Hollich, G. (2005) · 2005
Earlier work this paper cites.
In search of the neural circuits of intrinsic motivation
Kaplan, F. and Oudeyer, P.-Y. (2007) · 2007
Earlier work this paper cites.
What is intrinsic motivation? a typology of computational approaches
Oudeyer, P.-Y. and Kaplan, F. (2007) · 2007
Earlier work this paper cites.
Evolution and learning in an intrinsically motivated reinforcement learning robot
Schembri, M., Mirolli, M., and Baldassarre, G. (2007) · 2007
Earlier work this paper cites.
Intrinsically motivated reinforcement learning: An evolutionary perspective
Singh, S., Lewis, R. L., Barto, A. G., and Sorg, J. (2010) · 2010
Earlier work this paper cites.
What are intrinsic motivations? a biological perspective
Baldassarre, G. (2011) · 2011
Earlier work this paper cites.

Similar

Intrinsic motivation mechanisms for competence acquisition
Santucci, V.-G., Baldassarre, G., and Mirolli, M. (2012) · 2012
Cited alongside, same era.
Continually adding self-invented problems to the repertoire: first experiments with powerplay
Srivastava, R. K., Steunebrink, B. R., Stollenga, M., and Schmidhuber, J. (2012) · 2012
Cited alongside, same era.
Intrinsically motivated learning in natural and artificial systems
Baldassarre, G. and Mirolli, M., editors (2013) · 2013
Cited alongside, same era.
Novelty or surprise?
Barto, A., Mirolli, M., and Baldassarre, G. (2013) · 2013
Cited alongside, same era.
Curious replay for model-based adaptation
Original
Kauvar, I., Doyle, C., Zhou, L., and Haber, N. (2013) · 2013
Cited alongside, same era.
Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem
Schmidhuber, J. (2013) · 2013
Cited alongside, same era.
Identifying necessary conditions for open-ended evolution through the artificial life world of chromaria
Soros, L. B. and Stanley, K. O. (2014) · 2014
Cited alongside, same era.
Artificial Intelligence: A Modern Approach
Russell, S. J. and Norvig, P. (2016) · 2016
Cited alongside, same era.
Grail: A goal-discovering robotic architecture for intrinsically-motivated learning
Santucci, V. G., Baldassarre, G., and Mirolli, M. (2016) · 2016
Cited alongside, same era.
Open-ended evolution: Perspectives from the oee workshop in york
Taylor, T., Bedau, M., Channon, A., Ackley, D., Banzhaf, W., Beslon, G., Dolson, E., Froese, T., Hickinbotham, S., Ikegami, T., et al. (2016) · 2016
Cited alongside, same era.
Intrinsically motivated discovered outcomes boost user’s goals achievement in a humanoid robot
Seepanomwan, K., Santucci, V. G., and Baldassarre, G. (2017) · 2017
Cited alongside, same era.
Autonomous discovery of the goal space to learn a parameterized skill
Original
Cartoni, E. and Baldassarre, G. (2018) · 2018
Cited alongside, same era.

Then

Co-generation of game levels and game-playing agents
Dharna, A., Togelius, J., and Soros, L. B. (2020) · 2020
Later among the works it cites.
Why general artificial intelligence will not be realized
Fjelland, R. (2020) · 2020
Later among the works it cites.
Maximum entropy gain exploration for long horizon multi-goal reinforcement learning
Pitis, S., Chan, H., Zhao, S., Stadie, B., and Ba, J. (2020) · 2020
Later among the works it cites.
Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions
Wang, R., Lehman, J., Rawal, A., Zhi, J., Li, Y., Clune, J., and Stanley, K. (2020) · 2020
Later among the works it cites.
Learning to generate levels from nothing
Bontrager, P. and Togelius, J. (2021) · 2021
Later among the works it cites.
Meta-diversity search in complex systems, a recipe for artificial open-endedness?
Etcheverry, M., Chan, B. W.-C., Moulin-Frier, C., and Oudeyer, P.-Y. (2021) · 2021
Later among the works it cites.
The role of executive function in shaping reinforcement learning
Rmus, M., McDougle, S. D., and Collins, A. G. (2021) · 2021
Later among the works it cites.
Open-ended learning leads to generally capable agents
Original
Stooke, A., Mahajan, A., Barros, C., Deck, C., Bauer, J., Sygnowski, J., Trebacz, M., Jaderberg, M., Mathieu, M., et al. (2021) · 2021
Later among the works it cites.
Stein variational goal generation for reinforcement learning in hard exploration problems
Original
Castanet, N., Lamprier, S., and Sigaud, O. (2022) · 2022
Later among the works it cites.
Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: a short survey
Colas, C., Karch, T., Sigaud, O., and Oudeyer, P.-Y. (2022) · 2022
Later among the works it cites.
Watts: Infrastructure for open-ended learning
Original
Dharna, A., Summers, C., Dasari, R., Togelius, J., and Hoover, A. K. (2022) · 2022
Later among the works it cites.
Minedojo: Building open-ended embodied agents with internet-scale knowledge
Original
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. (2022) · 2022
Later among the works it cites.
Chemgrid: An open-ended benchmark domain for an open-ended learner
Kepes, M., Guttenberg, N., and Soros, L. (2022) · 2022
Later among the works it cites.
Goal-conditioned reinforcement learning: Problems and solutions
Original
Liu, M., Zhu, M., and Zhang, W. (2022) · 2022
Later among the works it cites.
Abstraction for deep reinforcement learning
Original
Shanahan, M. and Mitchell, M. (2022) · 2022
Later among the works it cites.
A definition of continual reinforcement learning
Original
Abel, D., Barreto, A., Van Roy, B., Precup, D., van Hasselt, H., and Singh, S. (2023) · 2023
Closest in time.
Human-timescale adaptation in an open-ended task space
Original
Adaptive Agent Team, D. (2023) · 2023
Closest in time.
General intelligence requires rethinking exploration
Jiang, M., Rocktäschel, T., and Grefenstette, E. (2023) · 2023
Closest in time.
Learning multiple tasks with non-stationary interdependencies in autonomous robots
Romero, A., Baldassarre, G., Duro, R. J., and Santucci, V.-G. (2023a) · 2023
Closest in time.
Towards teachable autonomous agents
Sigaud, O., Caselles-Dupré, H., Colas, C., Akakzia, A., Oudeyer, P.-Y., and Chetouani, M. (2023) · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…