2023

VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimation

Shi, Xiaoyu, Huang, Zhaoyang, Bian, Weikang et al.

Understand

We introduce VideoFlow, a novel optical flow estimation framework for videos.

In contrast to previous methods that learn to estimate optical flow from two frames, VideoFlow concurrently estimates bi-directional optical flows for multiple frames that are available in videos by sufficiently exploiting temporal cues.
We first propose a TRi-frame Optical Flow (TROF) module that estimates bi-directional optical flows for the center frame in a three-frame manner.
The information of the frame triplet is iteratively fused onto the center frame.

Built on

Determining optical flow
Berthold KP Horn and Brian G Schunck · 1981
Earlier work this paper cites.
A framework for the robust estimation of optical flow
Michael J Black and Padmanabhan Anandan · 1993
Earlier work this paper cites.
Probabilistic and sequential computation of optical flow using temporal coherence
Toshio M Chin, William Clement Karl, and Alan S Willsky · 1994
Earlier work this paper cites.
Recursive optical flow estimation—adaptive filtering approach
Michael Elad and Arie Feuer · 1998
Earlier work this paper cites.
Lucas/kanade meets horn/schunck: Combining local and global optic flow methods
Andrés Bruhn, Joachim Weickert, and Christoph Schnörr · 2005
Earlier work this paper cites.
A naturalistic open source movie for optical flow evaluation
Daniel J Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black · 2012
Earlier work this paper cites.
Vision meets robotics: The kitti dataset
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun · 2013
Earlier work this paper cites.
A quantitative analysis of current practices in optical flow estimation and the principles behind them
Deqing Sun, Stefan Roth, and Michael J Black · 2014
Earlier work this paper cites.
Flownet: Learning optical flow with convolutional networks
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox · 2015
Earlier work this paper cites.
The hci benchmark suite: Stereo and flow ground truth with uncertainties for urban autonomous driving
Daniel Kondermann, Rahul Nair, Katrin Honauer, Karsten Krispin, Jonas Andrulis, Alexander Brock, Burkhard Gussefeld, Mohsen Rahimimoghaddam, Sabine Hofmann, Claus Brenner, et al · 2016
Earlier work this paper cites.
A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation
Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox · 2016
Earlier work this paper cites.
Flownet 2.0: Evolution of optical flow estimation with deep networks
Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox · 2017
Earlier work this paper cites.
Deep laplacian pyramid networks for fast and accurate super-resolution
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang · 2017
Earlier work this paper cites.
Optical flow estimation using a spatial pyramid network
Anurag Ranjan and Michael J Black · 2017
Earlier work this paper cites.
Flow-guided feature aggregation for video object detection
Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen Wei · 2017
Earlier work this paper cites.
Liteflownet: A lightweight convolutional neural network for optical flow estimation
Tak-Wai Hui, Xiaoou Tang, and Chen Change Loy · 2018
Earlier work this paper cites.
Unsupervised learning of multi-frame optical flow with occlusions
Joel Janai, Fatma G”uney, Anurag Ranjan, Michael J. Black, and Andreas Geiger · 2018
Earlier work this paper cites.
Proflow: Learning to predict optical flow
Original
Daniel Maurer and Andrés Bruhn · 2018
Earlier work this paper cites.
Continual occlusion and optical flow estimation
Michal Neoral, Jan Šochman, and Jiří Matas · 2018
Earlier work this paper cites.
Frame-recurrent video super-resolution
Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown · 2018
Earlier work this paper cites.
Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume
Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz · 2018
Earlier work this paper cites.
Optical flow guided feature: A fast and robust motion representation for video action recognition
Shuyang Sun, Zhanghui Kuang, Lu Sheng, Wanli Ouyang, and Wei Zhang · 2018
Earlier work this paper cites.
Fully motion-aware network for video object detection
Shiyao Wang, Yucong Zhou, Junjie Yan, and Zhidong Deng · 2018
Earlier work this paper cites.
Towards high performance video object detection
Xizhou Zhu, Jifeng Dai, Lu Yuan, and Yichen Wei · 2018
Earlier work this paper cites.
Deep video inpainting
Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon · 2019
Earlier work this paper cites.
Selflow: Self-supervised learning of optical flow
Pengpeng Liu, Michael Lyu, Irwin King, and Jia Xu · 2019
Earlier work this paper cites.
Representation flow for action recognition
AJ Piergiovanni and Michael S Ryoo · 2019
Earlier work this paper cites.

Similar

A fusion approach for multi-frame optical flow estimation
Zhile Ren, Orazio Gallo, Deqing Sun, Ming-Hsuan Yang, Erik B Sudderth, and Jan Kautz · 2019
Cited alongside, same era.
Models matter, so does training: An empirical study of cnns for optical flow estimation
Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz · 2019
Cited alongside, same era.
Deep flow-guided video inpainting
Rui Xu, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy · 2019
Cited alongside, same era.
Volumetric correspondence networks for optical flow
Gengshan Yang and Deva Ramanan · 2019
Cited alongside, same era.
Hierarchical discrete distribution decomposition for match density estimation
Zhichao Yin, Trevor Darrell, and Fisher Yu · 2019
Cited alongside, same era.
Flow-edge guided video completion
Chen Gao, Ayush Saraf, Jia-Bin Huang, and Johannes Kopf · 2020
Cited alongside, same era.
Improving optical flow on a pyramid level
Markus Hofinger, Samuel Rota Bulò, Lorenzo Porzi, Arno Knapitsch, Thomas Pock, and Peter Kontschieder · 2020
Cited alongside, same era.
Rife: Real-time intermediate flow estimation for video frame interpolation
Original
Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou · 2020
Cited alongside, same era.
A lightweight optical flow cnn—revisiting data fidelity and regularization
Tak-Wai Hui, Xiaoou Tang, and Chen Change Loy · 2020
Cited alongside, same era.
Softmax splatting for video frame interpolation
Simon Niklaus and Feng Liu · 2020
Cited alongside, same era.
Raft: Recurrent all-pairs field transforms for optical flow
Zachary Teed and Jia Deng · 2020
Cited alongside, same era.
Optical flow in dense foggy scenes using semi-supervised learning
Wending Yan, Aashish Sharma, and Robby T Tan · 2020
Cited alongside, same era.

Then

High-resolution optical flow from 1d attention and correlation
Haofei Xu, Jiaolong Yang, Jianfei Cai, Juyong Zhang, and Xin Tong · 2021
Later among the works it cites.
Separable flow: Learning motion cost volumes for optical flow estimation
Feihu Zhang, Oliver J Woodford, Victor Adrian Prisacariu, and Philip HS Torr · 2021
Later among the works it cites.
Basicvsr++: Improving video super-resolution with enhanced propagation and alignment
Kelvin CK Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy · 2022
Later among the works it cites.
Tap-vid: A benchmark for tracking any point in a video
Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens Continente, Kucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang · 2022
Later among the works it cites.
Particle video revisited: Tracking through occlusions using point trajectories
Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki · 2022
Later among the works it cites.
Neuralmarker: A framework for learning general marker correspondence
Zhaoyang Huang, Xiaokun Pan, Weihong Pan, Weikang Bian, Yan Xu, Ka Chun Cheung, Guofeng Zhang, and Hongsheng Li · 2022
Later among the works it cites.
Flowformer: A transformer architecture for optical flow
Original
Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li · 2022
Later among the works it cites.
Learning degradation representations for image deblurring
Dasong Li, Yi Zhang, Ka Chun Cheung, Xiaogang Wang, Hongwei Qin, and Hongsheng Li · 2022
Later among the works it cites.
Efficient burst raw denoising with variance stabilization and multi-frequency denoising network, 2022
Dasong Li, Yi Zhang, Ka Lung Law, Xiaogang Wang, Hongwei Qin, and Hongsheng Li · 2022
Later among the works it cites.
Craft: Cross-attentional flow transformer for robust optical flow
Xiuchao Sui, Shaohua Li, Xue Geng, Yan Wu, Xinxing Xu, Yong Liu, Rick Goh, and Hongyuan Zhu · 2022
Later among the works it cites.
Skflow: Learning optical flow with super kernels
Original
Shangkun Sun, Yuanqi Chen, Yu Zhu, Guodong Guo, and Ge Li · 2022
Later among the works it cites.
Gmflow: Learning optical flow via global matching
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao · 2022
Later among the works it cites.
Global matching with overlapping attention for optical flow estimation
Shiyu Zhao, Long Zhao, Zhixing Zhang, Enyu Zhou, and Dimitris Metaxas · 2022
Later among the works it cites.
Context-tap: Tracking any point demands spatial context features
Original
Weikang Bian, Zhaoyang Huang, Xiaoyu Shi, Yitong Dong, Yijin Li, and Hongsheng Li · 2023
Closest in time.
A dynamic multi-scale voxel flow network for video prediction
Xiaotao Hu, Zhewei Huang, Ailin Huang, Jun Xu, and Shuchang Zhou · 2023
Closest in time.
Videocontrolnet: A motion-guided video-to-video translation framework by using diffusion model with controlnet
Original
Zhihao Hu and Dong Xu · 2023
Closest in time.
Flowformer: A transformer architecture and its masked cost volume autoencoding for optical flow
Original
Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Yijin Li, Hongwei Qin, Jifeng Dai, Xiaogang Wang, and Hongsheng Li · 2023
Closest in time.
A simple baseline for video restoration with grouped spatial-temporal shift
Dasong Li, Xiaoyu Shi, Yi Zhang, Ka Chun Cheung, Simon See, Xiaogang Wang, Hongwei Qin, and Hongsheng Li · 2023
Closest in time.
Blinkflow: A dataset to push the limits of event-based optical flow estimation
Original
Yijin Li, Zhaoyang Huang, Shuo Chen, Xiaoyu Shi, Hongsheng Li, Hujun Bao, Zhaopeng Cui, and Guofeng Zhang · 2023
Closest in time.
Flowformer++: Masked cost volume autoencoding for pretraining optical flow estimation
Original
Xiaoyu Shi, Zhaoyang Huang, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li · 2023
Closest in time.
Tracking everything everywhere all at once
Original
Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely · 2023
Closest in time.
Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis
Original
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li · 2023
Closest in time.
Better aligning text-to-image models with human preference
Original
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li · 2023
Closest in time.
Rerender a video: Zero-shot text-guided video-to-video translation
Original
Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy · 2023
Closest in time.
Kbnet: Kernel basis network for image restoration, 2023
Yi Zhang, Dasong Li, Xiaoyu Shi, Dailan He, Kangning Song, Xiaogang Wang, Hongwei Qin, and Hongsheng Li · 2023
Closest in time.
A unified conditional framework for diffusion-based image restoration
Original
Yi Zhang, Xiaoyu Shi, Dasong Li, Xiaogang Wang, Jian Wang, and Hongsheng Li · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…