2018

MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment

Zhang, Da, Dai, Xiyang, Wang, Xin et al.

Understand

This research strives for natural language moment retrieval in long, untrimmed video streams.

The problem is not trivial especially when a video contains multiple moments of interests and the language describes complex temporal dependencies, which often happens in real scenarios.
We identify two crucial challenges: semantic misalignment and structural misalignment.
However, existing approaches treat different moments separately and do not explicitly model complex moment-wise temporal relations.

Built on

Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Observing human-object interactions: Using spatial and functional compatibility for recognition
Abhinav Gupta, Aniruddha Kembhavi, and Larry S Davis · 2009
Earlier work this paper cites.
Modeling mutual context of object and human pose in human-object interaction activities
Bangpeng Yao and Li Fei-Fei · 2010
Earlier work this paper cites.
Semantic segmentation with second-order pooling
Joao Carreira, Rui Caseiro, Jorge Batista, and Cristian Sminchisescu · 2012
Earlier work this paper cites.
Describing the scene as a whole: Joint object detection, scene classification and semantic segmentation
Jian Yao, Sanja Fidler, and Raquel Urtasun · 2012
Earlier work this paper cites.
Temporal localization of actions with actoms
Adrien Gaidon, Zaid Harchaoui, and Cordelia Schmid · 2013
Earlier work this paper cites.
Action and event recognition with fisher vectors on a compact feature set
Dan Oneata, Jakob Verbeek, and Cordelia Schmid · 2013
Earlier work this paper cites.
Combining the right features for complex event recognition
Kevin Tang, Bangpeng Yao, Li Fei-Fei, and Daphne Koller · 2013
Earlier work this paper cites.
Action localization with tubelets from motion
Mihir Jain, Jan Van Gemert, Hervé Jégou, Patrick Bouthemy, and Cees GM Snoek · 2014
Earlier work this paper cites.
Glove: Global vectors for word representation
Jeffrey Pennington, Richard Socher, and Christopher Manning · 2014
Earlier work this paper cites.
Two-stream convolutional networks for action recognition in videos
Karen Simonyan and Andrew Zisserman · 2014
Earlier work this paper cites.
Image retrieval using scene graphs
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba · 2015
Earlier work this paper cites.
Bag-of-fragments: Selecting and encoding video fragments for event detection and recounting
Pascal Mettes, Jan C van Gemert, Spencer Cappallo, Thomas Mensink, and Cees GM Snoek · 2015
Earlier work this paper cites.
Faster r-cnn: Towards real-time object detection with region proposal networks
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun · 2015
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman · 2015
Earlier work this paper cites.
Learning spatiotemporal features with 3d convolutional networks
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri · 2015
Earlier work this paper cites.
Towards good practices for very deep two-stream convnets
Original
Limin Wang, Yuanjun Xiong, Zhe Wang, and Yu Qiao · 2015
Earlier work this paper cites.
Tensorflow: a system for large-scale machine learning
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al · 2016
Earlier work this paper cites.
Convolutional two-stream network fusion for video action recognition
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.

Similar

Ssd: Single shot multibox detector
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg · 2016
Cited alongside, same era.
Progressively parsing interactional objects for fine grained action detection
Bingbing Ni, Xiaokang Yang, and Shenghua Gao · 2016
Cited alongside, same era.
Temporal action localization in untrimmed videos via multi-stage cnns
Zheng Shou, Dongang Wang, and Shih-Fu Chang · 2016
Cited alongside, same era.
Hollywood in homes: Crowdsourcing data collection for activity understanding
Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta · 2016
Cited alongside, same era.
End-to-end, single-stream temporal action detection in untrimmed videos
S Buch, V Escorcia, B Ghanem, L Fei-Fei, and JC Niebles · 2017
Cited alongside, same era.
Quo vadis, action recognition? a new model and the kinetics dataset
Joao Carreira and Andrew Zisserman · 2017
Cited alongside, same era.
Detecting visual relationships with deep relational networks
Bo Dai, Yuqi Zhang, and Dahua Lin · 2017
Cited alongside, same era.
Fason: First and second order information fusion network for texture recognition
Xiyang Dai, Joe Yue-Hei Ng, and Larry S Davis · 2017
Cited alongside, same era.
Temporal context network for activity localization in videos
Xiyang Dai, Bharat Singh, Guyue Zhang, Larry S Davis, and Yan Qiu Chen · 2017
Cited alongside, same era.
Tall: Temporal activity localization via language query
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia · 2017
Cited alongside, same era.
Localizing moments in video with natural language
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell · 2017
Cited alongside, same era.
Modeling relationships in referential expressions with compositional modular networks
Ronghang Hu, Marcus Rohrbach, Jacob Andreas, Trevor Darrell, and Kate Saenko · 2017
Cited alongside, same era.

Then

R-c3d: Region convolutional 3d network for temporal activity detection
Huijuan Xu, Abir Das, and Kate Saenko · 2017
Later among the works it cites.
Temporal action detection with structured segment networks
Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin · 2017
Later among the works it cites.
Rethinking the faster r-cnn architecture for temporal action localization
Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A Ross, Jia Deng, and Rahul Sukthankar · 2018
Closest in time.
Temporally grounding natural sentence in video
Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua · 2018
Closest in time.
Tan: Temporal aggregation network for dense multi-label action recognition
Xiyang Dai, Bharat Signh, Joe Yue-Hei Ng, and Larry S. Davis · 2018
Closest in time.
Detecting and recognizing human-object interactions
Georgia Gkioxari Ross Girshick Piotr Dollár and Kaiming He · 2018
Closest in time.
Actor and action video segmentation from a sentence
Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, and Cees GM Snoek · 2018
Closest in time.
Localizing moments in video with temporal language
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell · 2018
Closest in time.
Relation networks for object detection
Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei · 2018
Closest in time.
Temporal modular networks for retrieving complex compositional activities in videos
Bingbin Liu, Serena Yeung, Edward Chou, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles · 2018
Closest in time.
Attend and interact: Higher-order object interactions for video understanding
Chih-Yao Ma, Asim Kadav, Iain Melvin, Zsolt Kira, Ghassan AlRegib, and Hans Peter Graf · 2018
Closest in time.
A closer look at spatiotemporal convolutions for action recognition
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri · 2018
Closest in time.
Video captioning via hierarchical reinforcement learning
Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, and William Yang Wang · 2018
Closest in time.
Non-local neural networks
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He · 2018
Closest in time.
Learning to compose topic-aware mixture of experts for zero-shot video captioning
Original
Xin Wang, Jiawei Wu, Da Zhang, Yu Su, and William Yang Wang · 2018
Closest in time.
Spatial temporal graph convolutional networks for skeleton-based action recognition
Sijie Yan, Yuanjun Xiong, and Dahua Lin · 2018
Closest in time.
Graph r-cnn for scene graph generation
Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh · 2018
Closest in time.
S3d: Single shot multi-span detector via fully 3d convolutional network
Da Zhang, Xiyang Dai, Xin Wang, and Yuan-Fang Wang · 2018
Closest in time.
Dynamic temporal pyramid network: A closer look at multi-scale modeling for activity detection
Da Zhang, Xiyang Dai, and Yuan-Fang Wang · 2018
Closest in time.
Multilevel language and vision integration for text-to-clip retrieval
Huijuan Xu, Kun He, Leonid Sigal, Stan Sclaroff, and Kate Saenko · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…