2017

Dense-Captioning Events in Videos

Krishna, Ranjay, Hata, Kenji, Ren, Frederic et al.

Understand

Most natural videos contain numerous events.

For example, in a video of a "man playing a piano", the video might also contain "another man dancing" or "a crowd clapping".
We introduce the task of dense-captioning events, which involves both detecting and describing events in a video.
We propose a new model that is able to identify all events in a single pass of the video while simultaneously describing the detected events with natural language.

Built on

Recognizing human action in time-sequential images using hidden markov model
J. Yamato, J. Ohya, and K. Ishii · 1992
Earlier work this paper cites.
Key frame selection by motion analysis
W. Wolf · 1996
Earlier work this paper cites.
An integrated system for content-based video retrieval and browsing
H. J. Zhang, J. Wu, D. Zhong, and S. W. Smoliar · 1997
Earlier work this paper cites.
Recognizing human actions: A local svm approach
C. Schuldt, I. Laptev, and B. Caputo · 2004
Earlier work this paper cites.
Schematic storyboarding for video visualization and editing
D. B. Goldman, B. Curless, D. Salesin, and S. M. Seitz · 2006
Earlier work this paper cites.
Detecting irregularities in images and in video
O. Boiman and M. Irani · 2007
Earlier work this paper cites.
Learning realistic human actions from movies
I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld · 2008
Earlier work this paper cites.
Automatic annotation of human actions in video
O. Duchenne, I. Laptev, J. Sivic, F. Bach, and J. Ponce · 2009
Earlier work this paper cites.
Actions in context
M. Marszałek, I. Laptev, and C. Schmid · 2009
Earlier work this paper cites.
Recurrent neural network based language model
T. Mikolov, M. Karafiát, L. Burget, J. Cernockỳ, and S. Khudanpur · 2010
Earlier work this paper cites.
Modeling temporal structure of decomposable motion segments for activity classification
J. C. Niebles, C.-W. Chen, and L. Fei-Fei · 2010
Earlier work this paper cites.
Collecting highly parallel data for paraphrase evaluation
D. L. Chen and W. B. Dolan · 2011
Earlier work this paper cites.
Hmdb: a large video database for human motion recognition
H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre · 2011
Earlier work this paper cites.
A discriminative key pose sequence model for recognizing human interactions
A. Vahdat, B. Gao, M. Ranjbar, and G. Mori · 2011
Earlier work this paper cites.
A database for fine grained activity detection of cooking activities
M. Rohrbach, S. Amin, M. Andriluka, and B. Schiele · 2012
Earlier work this paper cites.
Ucf101: A dataset of 101 human actions classes from videos in the wild
Original
K. Soomro, A. R. Zamir, and M. Shah · 2012
Earlier work this paper cites.
A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching
P. Das, C. Xu, R. F. Doell, and · 2013
Earlier work this paper cites.
Temporal localization of actions with actoms
A. Gaidon, Z. Harchaoui, and C. Schmid · 2013
Earlier work this paper cites.
Vision meets robotics: The kitti dataset
A. Geiger, P. Lenz, C. Stiller, and R. Urtasun · 2013
Earlier work this paper cites.
3d convolutional neural networks for human action recognition
S. Ji, W. Xu, M. Yang, and K. Yu · 2013
Earlier work this paper cites.
Grounding action descriptions in videos
M. Regneri, M. Rohrbach, D. Wetzel, S. Thater, B. Schiele, and M. Pinkal · 2013
Earlier work this paper cites.
Spatiotemporal deformable part models for action detection
Y. Tian, R. Sukthankar, and M. Shah · 2013
Earlier work this paper cites.

Similar

Fast saliency based pooling of fisher encoded dense trajectories
S. Karaman, L. Seidenari, and A. Del Bimbo · 2014
Cited alongside, same era.
Large-scale video classification with convolutional neural networks
A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei · 2014
Cited alongside, same era.
Multiple granularity analysis for fine-grained action detection
B. Ni, V. R. Paramathayalan, and P. Moulin · 2014
Cited alongside, same era.
Efficient action localization with approximately normalized fisher vectors
D. Oneata, J. Verbeek, and C. Schmid · 2014
Cited alongside, same era.
Parsing videos of actions with segmental grammars
H. Pirsiavash and D. Ramanan · 2014
Cited alongside, same era.
Coherent multi-sentence video description with variable level of detail
A. Rohrbach, M. Rohrbach, W. Qiu, A. Friedrich, M. Pinkal, and B. Schiele · 2014
Cited alongside, same era.
Translating videos to natural language using deep recurrent neural networks
Original
S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. Mooney, and K. Saenko · 2014
Cited alongside, same era.
Action recognition and detection by combining motion and appearance features
L. Wang, Y. Qiao, and X. Tang · 2014
Cited alongside, same era.
Video action detection with relational dynamic-poselets
L. Wang, Y. Qiao, and X. Tang · 2014
Cited alongside, same era.
Videoset: Video summary evaluation through text
Original
S. Yeung, A. Fathi, and L. Fei-Fei · 2014
Cited alongside, same era.
Activitynet: A large-scale video benchmark for human activity understanding
F. Caba Heilbron, V. Escorcia, B. Ghanem, and J. C. Niebles · 2015
Cited alongside, same era.
Long-term recurrent convolutional networks for visual recognition and description
J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell · 2015
Cited alongside, same era.

Then

Sequence to sequence-video to text
S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko · 2015
Later among the works it cites.
A multi-scale multiple instance video description network
Original
H. Xu, S. Venugopalan, V. Ramanishka, M. Rohrbach, and K. Saenko · 2015
Later among the works it cites.
Show, attend and tell: Neural image caption generation with visual attention
K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio · 2015
Later among the works it cites.
Jointly modeling deep video and compositional text to bridge vision and language in a unified framework
R. Xu, C. Xiong, W. Chen, and J. J. Corso · 2015
Later among the works it cites.
Unsupervised extraction of video highlights via robust recurrent auto-encoders
H. Yang, B. Wang, S. Lin, D. Wipf, M. Guo, and B. Guo · 2015
Later among the works it cites.
Describing videos by exploiting temporal structure
L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and A. Courville · 2015
Later among the works it cites.
Social lstm: Human trajectory prediction in crowded spaces
A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese · 2016
Later among the works it cites.
Discovering event structure in continuous narrative perception and memory
C. Baldassano, J. Chen, A. Zadbood, J. W. Pillow, U. Hasson, and K. A. Norman · 2016
Later among the works it cites.
Fast temporal activity proposals for efficient detection of human actions in untrimmed videos
F. Caba Heilbron, J. C. Niebles, and B. Ghanem · 2016
Later among the works it cites.
Daps: Deep action proposals for action understanding
V. Escorcia, F. C. Heilbron, J. C. Niebles, and B. Ghanem · 2016
Later among the works it cites.
Densecap: Fully convolutional localization networks for dense captioning
J. Johnson, A. Karpathy, and L. Fei-Fei · 2016
Later among the works it cites.
Embracing error to enable rapid crowdsourcing
R. A. Krishna, K. Hata, S. Chen, J. Kravitz, D. A. Shamma, L. Fei-Fei, and M. S. Bernstein · 2016
Later among the works it cites.
Learning joint representations of videos and sentences with web image search
M. Otani, Y. Nakashima, E. Rahtu, J. Heikkilä, and N. Yokoya · 2016
Later among the works it cites.
Jointly modeling embedding and translation to bridge video and language
Y. Pan, T. Mei, T. Yao, H. Li, and Y. Rui · 2016
Later among the works it cites.
Hollywood in homes: Crowdsourcing data collection for activity understanding
G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta · 2016
Later among the works it cites.
Msr-vtt: A large video description dataset for bridging video and language
J. Xu, T. Mei, T. Yao, and Y. Rui · 2016
Later among the works it cites.
Dense captioning with joint inference and visual context
Original
L. Yang, K. Tang, J. Yang, and L.-J. Li · 2016
Later among the works it cites.
Highlight detection with pairwise deep ranking for first-person video summarization
T. Yao, T. Mei, and Y. Rui · 2016
Later among the works it cites.
Video paragraph captioning using hierarchical recurrent neural networks
H. Yu, J. Wang, Z. Huang, Y. Yang, and W. Xu · 2016
Later among the works it cites.
A hierarchical approach for generating descriptive image paragraphs
J. Krause, J. Johnson, R. Krishna, and L. Fei-Fei · 2017
Closest in time.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, M. Bernstein, and L. Fei-Fei · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…