2023

Vision-Language Models as Success Detectors

Du, Yuqing, Konyushkova, Ksenia, Denil, Misha et al.

Understand

Detecting successful behaviour is crucial for training intelligent agents.

As such, generalisable reward models are a prerequisite for agents that can learn to generalise their behaviour.
In this work we focus on developing robust success detectors that leverage large, pretrained vision-language models (Flamingo, Alayrac et al.
(2022)) and human reward annotations.

Built on

Rank analysis of incomplete block designs: I. the method of paired comparisons
R. A. Bradley and M. E. Terry · 1952
Earlier work this paper cites.
Algorithms for inverse reinforcement learning
A. Y. Ng, S. Russell, et al · 2000
Earlier work this paper cites.
Apprenticeship learning via inverse reinforcement learning
P. Abbeel and A. Y. Ng · 2004
Earlier work this paper cites.
Tamer: Training an agent manually via evaluative reinforcement
W. B. Knox and P. Stone · 2008
Earlier work this paper cites.
APRIL: Active preference learning-based reinforcement learning
R. Akrour, M. Schoenauer, and M. Sebag · 2012
Earlier work this paper cites.
Playing atari with deep reinforcement learning
Original
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller · 2013
Earlier work this paper cites.
Programming by feedback
M. Schoenauer, R. Akrour, M. Sebag, and J.-C. Souplet · 2014
Earlier work this paper cites.
Guided cost learning: Deep inverse optimal control via policy optimization
C. Finn, S. Levine, and P. Abbeel · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Earlier work this paper cites.
Generative adversarial imitation learning
J. Ho and S. Ermon · 2016
Earlier work this paper cites.
Msr-vtt: A large video description dataset for bridging video and language
J. Xu, T. Mei, T. Yao, and Y. Rui · 2016
Earlier work this paper cites.
End-to-end differentiable adversarial imitation learning
N. Baram, O. Anschel, I. Caspi, and S. Mannor · 2017
Earlier work this paper cites.
Deep reinforcement learning from human preferences
P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei · 2017
Earlier work this paper cites.
Making the v in vqa matter: Elevating the role of image understanding in visual question answering
Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh · 2017
Earlier work this paper cites.
InfoGAIL: Interpretable imitation learning from visual demonstrations
Y. Li, J. Song, and S. Ermon · 2017
Earlier work this paper cites.
Interactive learning from policy-dependent human feedback
J. MacGlashan, M. K. Ho, R. Loftin, B. Peng, G. Wang, D. L. Roberts, M. E. Taylor, and M. L. Littman · 2017
Earlier work this paper cites.
Learning human behaviors from motion capture by adversarial imitation
Original
J. Merel, Y. Tassa, S. Srinivasan, J. Lemmon, Z. Wang, G. Wayne, and N. Heess · 2017
Earlier work this paper cites.
Active preference-based learning of reward functions
D. Sadigh, A. D. Dragan, S. Sastry, and S. A. Seshia · 2017
Earlier work this paper cites.

Similar

Learning robust rewards with adversarial inverse reinforcement learning
J. Fu, K. Luo, and S. Levine · 2018
Cited alongside, same era.
Reward learning from human preferences and demonstrations in atari
B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei · 2018
Cited alongside, same era.
Reinforcement and imitation learning for diverse visuomotor skills
Y. Zhu, Z. Wang, J. Merel, A. A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Hadsell, N. de Freitas, and N. Heess · 2018
Cited alongside, same era.
Deep reinforcement learning from policy-dependent human feedback
Original
D. Arumugam, J. K. Lee, S. Saskin, and M. L. Littman · 2019
Cited alongside, same era.
Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations
D. Brown, W. Goo, P. Nagarajan, and S. Niekum · 2019
Cited alongside, same era.
End-to-end robotic reinforcement learning without reward engineering
A. Singh, L. Yang, K. Hartikainen, C. Finn, and S. Levine · 2019
Cited alongside, same era.
Imitating interactive intelligence
Original
J. Abramson, A. Ahuja, I. Barr, A. Brussee, F. Carnevale, M. Cassin, R. Chhaparia, S. Clark, B. Damoc, A. Dudzik, et al · 2020
Cited alongside, same era.
Scaling data-driven robotics with reward sketching and batch reinforcement learning
S. Cabi, S. Gómez Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zolna, Y. Aytar, D. Budden, M. Vecerik, O. Sushkov, D. Barker, J. Scholz, M. Denil, N. de Freitas, and Z. Wang · 2020
Cited alongside, same era.
Univl: A unified video and language pre-training model for multimodal understanding and generation
Original
H. Luo, L. Ji, B. Shi, H. Huang, N. Duan, T. Li, J. Li, T. Bharti, and M. Zhou · 2020
Cited alongside, same era.
Learning human objectives by evaluating hypothetical behavior
S. Reddy, A. Dragan, S. Levine, S. Legg, and J. Leike · 2020
Cited alongside, same era.
Learning to summarize from human feedback
N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. Christiano · 2020
Cited alongside, same era.
dm_control: Software and tasks for continuous control
S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, N. Heess, and Y. Tassa · 2020
Cited alongside, same era.

Then

Learning transferable visual models from natural language supervision
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al · 2021
Later among the works it cites.
Flamingo: a visual language model for few-shot learning
Original
J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al · 2022
Later among the works it cites.
Training a helpful and harmless assistant with reinforcement learning from human feedback
Original
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al · 2022
Later among the works it cites.
Can foundation models perform zero-shot task specification for robot manipulation?
Y. Cui, S. Niekum, A. Gupta, V. Kumar, and A. Rajeswaran · 2022
Later among the works it cites.
Enabling multimodal generation on clip via vision-language knowledge distillation
Original
W. Dai, L. Hou, L. Shang, X. Jiang, Q. Liu, and P. Fung · 2022
Later among the works it cites.
Minedojo: Building open-ended embodied agents with internet-scale knowledge
Original
L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anandkumar · 2022
Later among the works it cites.
Improving alignment of dialogue agents via targeted human judgements
Original
A. Glaese, N. McAleese, M. Trebacz, J. Aslanides, V. Firoiu, T. Ewalds, M. Rauh, L. Weidinger, M. Chadwick, P. Thacker, L. Campbell-Gillingham, J. Uesato, P.-S. Huang, R. Comanescu, F. Yang, A. See, S. Dathathri, R. Greig, C. Chen, D. Fritz, J. S. Elias, R. Green, S. Mokrá, N. Fernando, B. Wu, R. Foley, S. Young, I. Gabriel, W. Isaac, J. Mellor, D. Hassabis, K. Kavukcuoglu, L. A. Hendricks, and G. Irving · 2022
Later among the works it cites.
Ego4d: Around the world in 3,000 hours of egocentric video
K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al · 2022
Later among the works it cites.
Vip: Towards universal visual reward and representation via value-implicit pre-training
Original
Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang · 2022
Later among the works it cites.
Zero-shot reward specification via grounded natural language
P. Mahmoudieh, D. Pathak, and T. Darrell · 2022
Later among the works it cites.
Teaching language models to support answers with verified quotes
Original
J. Menick, M. Trebacz, V. Mikulik, J. Aslanides, F. Song, M. Chadwick, M. Glaese, S. Young, L. Campbell-Gillingam, G. Irving, and N. McAleese · 2022
Later among the works it cites.
Webgpt: Browser-assisted question-answering with human feedback
Original
R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman · 2022
Later among the works it cites.
Training language models to follow instructions with human feedback
Original
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al · 2022
Later among the works it cites.
A generalist agent
Original
S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas · 2022
Later among the works it cites.
Plug-and-play vqa: Zero-shot vqa by conjoining large pretrained models with zero training
Original
A. M. H. Tiong, J. Li, B. Li, S. Savarese, and S. C. Hoi · 2022
Later among the works it cites.
Mastering diverse domains through world models
Original
D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap · 2023
Closest in time.
Grounding language models to images for multimodal generation
Original
J. Y. Koh, R. Salakhutdinov, and D. Fried · 2023
Closest in time.
Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models
Original
J. Li, D. Li, S. Savarese, and S. Hoi · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…