2020

QuerYD: A video dataset with high-quality text and audio narrations

Oncescu, Andreea-Maria, Henriques, João F., Liu, Yang et al.

Understand

We introduce QuerYD, a new large-scale dataset for retrieval and event localisation in video.

A unique feature of our dataset is the availability of two audio tracks for each video: the original audio, and a high-quality spoken description of the visual content.
The dataset is based on YouDescribe, a volunteer project that assists visually-impaired people by attaching voiced narrations to existing YouTube videos.
This ever-growing collection of videos contains highly detailed, temporally aligned audio and text annotations.

Built on

“Hierarchical mixtures of experts and the em algorithm,”
Michael I Jordan and Robert A Jacobs, · 1994
Earlier work this paper cites.
“Natural language description of human activities from video images based on concept hierarchy of actions,”
Atsuhiro Kojima, Takeshi Tamura, and Kunio Fukunaga, · 2002
Earlier work this paper cites.
“Collecting highly parallel data for paraphrase evaluation,”
David L Chen and William B Dolan, · 2011
Earlier work this paper cites.
“Video in sentences out,”
Original
Andrei Barbu, Alexander Bridge, Zachary Burchill, Dan Coroian, Sven Dickinson, Sanja Fidler, Aaron Michaux, Sam Mussman, Siddharth Narayanaswamy, Dhaval Salvi, et al., · 2012
Earlier work this paper cites.
“Youdescribe,” 2013
Video Description Research and Development Center, · 2013
Earlier work this paper cites.
“A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching,”
Pradipto Das, Chenliang Xu, Richard F Doell, and Jason J Corso, · 2013
Earlier work this paper cites.
“Coherent multi-sentence video description with variable level of detail,”
Anna Rohrbach, Marcus Rohrbach, Wei Qiu, Annemarie Friedrich, Manfred Pinkal, and Bernt Schiele, · 2014
Earlier work this paper cites.
“A dataset for movie description,”
Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele, · 2015
Earlier work this paper cites.
“Describing videos by exploiting temporal structure,”
Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville, · 2015
Earlier work this paper cites.
“Msr-vtt: A large video description dataset for bridging video and language,”
Jun Xu, Tao Mei, Ting Yao, and Yong Rui, · 2016
Earlier work this paper cites.

Similar

“Spacy: natural language processing library,” https://spacy.io , 2016
Explosion AI, · 2016
Cited alongside, same era.
“Localizing moments in video with natural language,”
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell, · 2017
Cited alongside, same era.
“Dense-captioning events in videos,”
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles, · 2017
Cited alongside, same era.
“Tall: Temporal activity localization via language query,”
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia, · 2017
Cited alongside, same era.
“A simple neural network module for relational reasoning,”
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap, · 2017
Cited alongside, same era.
“Deep text mining of instagram data without strong supervision,”
Kim Hammar, Shatha Jaradat, Nima Dokoohaki, and Mihhail Matskin, · 2018
Cited alongside, same era.
“Do CIFAR-10 classifiers generalize to cifar-10?,”
Original
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar, · 2018
Cited alongside, same era.
“Towards automatic learning of procedures from web instructional videos,”
Luowei Zhou, Chenliang Xu, and Jason J Corso, · 2018
Cited alongside, same era.
“Scaling egocentric vision: The EPIC-KITCHENS dataset,”
Original
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray, · 2018
Cited alongside, same era.
“Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification,”
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, · 2018
Cited alongside, same era.
“Speech-to-Text,” https://cloud.google.com/speech-to-text
Google,
Cited in the paper.

Then

“Learning a text-video embedding from incomplete and heterogeneous data,”
Original
Antoine Miech, Ivan Laptev, and Josef Sivic, · 2018
Later among the works it cites.
“Cross-modal and hierarchical modeling of video and text,”
Bowen Zhang, Hexiang Hu, and Fei Sha, · 2018
Later among the works it cites.
“VATEX: A large-scale, high-quality multilingual dataset for video-and-language research,”
Original
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang, · 2019
Later among the works it cites.
“Vatex: A large-scale, high-quality multilingual dataset for video-and-language research,”
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang, · 2019
Later among the works it cites.
“Cross-task weakly supervised learning from instructional videos,”
Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic, · 2019
Later among the works it cites.
“Howto100m: Learning a text-video embedding by watching hundred million narrated video clips,”
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic, · 2019
Later among the works it cites.
“End-to-end learning of visual representations from uncurated instructional videos,”
Original
Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman, · 2019
Later among the works it cites.
“Use what you have: Video retrieval using representations from collaborative experts,”
Original
Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman, · 2019
Later among the works it cites.
“Temporal localization of moments in video collections with natural language,”
Original
Victor Escorcia, Mattia Soldan, Josef Sivic, Bernard Ghanem, and Bryan Russell, · 2019
Later among the works it cites.
“Condensed movies: Story based retrieval with contextual embeddings,”
Original
Max Bain, Arsha Nagrani, Andrew Brown, and Andrew Zisserman, · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…