Built on

N. Das, S. Bechtle, T. Davchev, D. Jayaraman, A. Rai, and F. Meier, “Model-based inverse reinforcement learning from visual demonstrations,” in
1942
Earlier work this paper cites.
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in
2015
Earlier work this paper cites.
L. Sun, K. Jia, D.-Y. Yeung, and B. E. Shi, “Human action recognition using factorized spatio-temporal convolutional networks,” in
2015
Earlier work this paper cites.
S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,”
2016
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in
2016
Earlier work this paper cites.
C. Finn and S. Levine, “Deep visual foresight for planning robot motion,” in
2017
Earlier work this paper cites.
A. Ten Pas, M. Gualtieri, K. Saenko, and R. Platt, “Grasp pose detection in point clouds,”
2017
Earlier work this paper cites.
J. Zhou, S. Chen, and Z. Wang, “A soft-robotic gripper with enhanced object adaptation and grasping reliability,”
2017
Earlier work this paper cites.
S.-H. Sun, H. Noh, S. Somasundaram, and J. Lim, “Neural program synthesis from diverse demonstration videos,” in
2018
Earlier work this paper cites.
Y. Liu, A. Gupta, P. Abbeel, and S. Levine, “Imitation from observation: Learning to imitate behaviors from raw video via context translation,” in
2018
Earlier work this paper cites.
P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain, “Time-contrastive networks: Self-supervised learning from video,” in
2018
Earlier work this paper cites.
P. Sharma, D. Pathak, and A. Gupta, “Third-person visual imitation learning via decoupled hierarchical controller,”
2019
Earlier work this paper cites.
L. Smith, N. Dhawan, M. Zhang, P. Abbeel, and S. Levine, “Avid: Learning multi-stage tasks via pixel-level translation of human videos,”
Original
2019
Earlier work this paper cites.
A. D. Edwards and C. L. Isbell, “Perceptual values from observation,”
Original
2019
Earlier work this paper cites.
R. Scalise, J. Thomason, Y. Bisk, and S. Srinivasa, “Improving robot success detection using static object data,” in
2019
Earlier work this paper cites.
S. Pirk, M. Khansari, Y. Bai, C. Lynch, and P. Sermanet, “Online object representations with contrastive learning,”
Original
2019
Earlier work this paper cites.
L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, “Visualbert: A simple and performant baseline for vision and language,”
Original
2019
Earlier work this paper cites.
A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman, “Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning,”
Original
2019
Earlier work this paper cites.
G. Garcia-Hernando, E. Johns, and T.-K. Kim, “Physics-based dexterous manipulations with estimated hand poses and residual reinforcement learning,” in
2020
Earlier work this paper cites.
K. Schmeckpeper, O. Rybkin, K. Daniilidis, S. Levine, and C. Finn, “Reinforcement learning with videos: Combining offline observations with interaction,”
Original
2020
Earlier work this paper cites.

Similar

J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,”
2020
Cited alongside, same era.
T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,” in
2020
Cited alongside, same era.
J. Yamada, Y. Lee, G. Salhotra, K. Pertsch, M. Pflueger, G. Sukhatme, J. Lim, and P. Englert, “Motion planner augmented reinforcement learning for robot manipulation in obstructed environments,” in
2021
Cited alongside, same era.
X. Yang, Z. Ji, J. Wu, Y.-K. Lai, C. Wei, G. Liu, and R. Setchi, “Hierarchical reinforcement learning with universal policies for multistep robotic manipulation,”
2021
Cited alongside, same era.
H. Xiong, Q. Li, Y.-C. Chen, H. Bharadhwaj, S. Sinha, and A. Garg, “Learning by watching: Physical imitation of manipulation skills from human videos,” in

Then

R. Mendonca, S. Bahl, and D. Pathak, “Structured world models from human videos,”
Original
2023
Later among the works it cites.
S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak, “Affordances from human videos as a versatile representation for robotics,” in
2023
Later among the works it cites.
P.-C. Ko, J. Mao, Y. Du, S.-H. Sun, and J. B. Tenenbaum, “Learning to act from actionless videos through dense correspondences,”
Original
2023
Later among the works it cites.
N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rupprecht, “Cotracker: It is better to track together,”
Original
2023
Later among the works it cites.
Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar, “Eureka: Human-level reward design via coding large language models,”

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

Embodiment-Agnostic Action Planning via Object-Part Scene Flow

Tang, Weiliang, Pan, Jia-Hui, Zhan, Wei et al.

Understand

Observing that the key for robotic action planning is to understand the target-object motion when its associated part is manipulated by the end effector, we propose to generate the 3D object-part scene flow and extract its transformations to solve the action trajectories for diverse embodiments.

The advantage of our approach is that it derives the robot action explicitly from object motion prediction, yielding a more robust policy by understanding the object motions.
Also, beyond policies trained on embodiment-centric data, our method is embodiment-agnostic, generalizable across diverse embodiments, and being able to learn from human demonstrations.
Our method comprises three components: an object-part predictor to locate the part for the end effector to manipulate, an RGBD video generator to predict future RGBD videos, and a trajectory planner to extract embodiment-agnostic transformation sequences and solve the trajectory for diverse embodiments.

Built on

N. Das, S. Bechtle, T. Davchev, D. Jayaraman, A. Rai, and F. Meier, “Model-based inverse reinforcement learning from visual demonstrations,” in
1942
Earlier work this paper cites.
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in
2015
Earlier work this paper cites.
L. Sun, K. Jia, D.-Y. Yeung, and B. E. Shi, “Human action recognition using factorized spatio-temporal convolutional networks,” in
2015
Earlier work this paper cites.
S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,”
2016
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in
2016
Earlier work this paper cites.
C. Finn and S. Levine, “Deep visual foresight for planning robot motion,” in
2017
Earlier work this paper cites.
A. Ten Pas, M. Gualtieri, K. Saenko, and R. Platt, “Grasp pose detection in point clouds,”
2017
Earlier work this paper cites.
J. Zhou, S. Chen, and Z. Wang, “A soft-robotic gripper with enhanced object adaptation and grasping reliability,”
2017
Earlier work this paper cites.
S.-H. Sun, H. Noh, S. Somasundaram, and J. Lim, “Neural program synthesis from diverse demonstration videos,” in
2018
Earlier work this paper cites.
Y. Liu, A. Gupta, P. Abbeel, and S. Levine, “Imitation from observation: Learning to imitate behaviors from raw video via context translation,” in
2018
Earlier work this paper cites.
P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain, “Time-contrastive networks: Self-supervised learning from video,” in
2018
Earlier work this paper cites.
P. Sharma, D. Pathak, and A. Gupta, “Third-person visual imitation learning via decoupled hierarchical controller,”
2019
Earlier work this paper cites.
L. Smith, N. Dhawan, M. Zhang, P. Abbeel, and S. Levine, “Avid: Learning multi-stage tasks via pixel-level translation of human videos,”
Original
2019
Earlier work this paper cites.
A. D. Edwards and C. L. Isbell, “Perceptual values from observation,”
Original
2019
Earlier work this paper cites.
R. Scalise, J. Thomason, Y. Bisk, and S. Srinivasa, “Improving robot success detection using static object data,” in
2019
Earlier work this paper cites.
S. Pirk, M. Khansari, Y. Bai, C. Lynch, and P. Sermanet, “Online object representations with contrastive learning,”
Original
2019
Earlier work this paper cites.
L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, “Visualbert: A simple and performant baseline for vision and language,”
Original
2019
Earlier work this paper cites.
A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman, “Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning,”
Original
2019
Earlier work this paper cites.
G. Garcia-Hernando, E. Johns, and T.-K. Kim, “Physics-based dexterous manipulations with estimated hand poses and residual reinforcement learning,” in
2020
Earlier work this paper cites.
K. Schmeckpeper, O. Rybkin, K. Daniilidis, S. Levine, and C. Finn, “Reinforcement learning with videos: Combining offline observations with interaction,”
Original
2020
Earlier work this paper cites.

Similar

J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,”
2020
Cited alongside, same era.
T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,” in
2020
Cited alongside, same era.
J. Yamada, Y. Lee, G. Salhotra, K. Pertsch, M. Pflueger, G. Sukhatme, J. Lim, and P. Englert, “Motion planner augmented reinforcement learning for robot manipulation in obstructed environments,” in
2021
Cited alongside, same era.
X. Yang, Z. Ji, J. Wu, Y.-K. Lai, C. Wei, G. Liu, and R. Setchi, “Hierarchical reinforcement learning with universal policies for multistep robotic manipulation,”
2021
Cited alongside, same era.
H. Xiong, Q. Li, Y.-C. Chen, H. Bharadhwaj, S. Sinha, and A. Garg, “Learning by watching: Physical imitation of manipulation skills from human videos,” in

Then

R. Mendonca, S. Bahl, and D. Pathak, “Structured world models from human videos,”
Original
2023
Later among the works it cites.
S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak, “Affordances from human videos as a versatile representation for robotics,” in
2023
Later among the works it cites.
P.-C. Ko, J. Mao, Y. Du, S.-H. Sun, and J. B. Tenenbaum, “Learning to act from actionless videos through dense correspondences,”
Original
2023
Later among the works it cites.
N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rupprecht, “Cotracker: It is better to track together,”
Original
2023
Later among the works it cites.
Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar, “Eureka: Human-level reward design via coding large language models,”

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2021

Cited alongside, same era.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark,

2021

Cited alongside, same era.

W. Kim, B. Son, and I. Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” in

2021

Cited alongside, same era.

S. Nasiriany, H. Liu, and Y. Zhu, “Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks,” in

2022

Cited alongside, same era.

P. Mandikal and K. Grauman, “Dexvip: Learning dexterous grasping with human hand pose priors from video,” in

2022

Cited alongside, same era.

M. Goyal, S. Modi, R. Goyal, and S. Gupta, “Human hands as probes for interactive object understanding,” in

2022

Cited alongside, same era.

S. Liu, S. Tripathi, S. Majumdar, and X. Wang, “Joint hand motion and interaction hotspots prediction from egocentric videos,” in

2022

Cited alongside, same era.

S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta, “R3m: A universal visual representation for robot manipulation,”

Original

2022

Cited alongside, same era.

Original

2023

Later among the works it cites.

K. Shaw, A. Agarwal, and D. Pathak, “Leap hand: Low-cost, efficient, and anthropomorphic hand for robot learning,”

Original

2023

Later among the works it cites.

H. Zhen, X. Qiu, P. Chen, J. Yang, X. Yan, Y. Du, Y. Hong, and C. Gan, “3d-vla: A 3d vision-language-action generative world model,”

Original

2024

Closest in time.

C. Wang, H. Shi, W. Wang, R. Zhang, L. Fei-Fei, and C. K. Liu, “Dexcap: Scalable and portable mocap data collection system for dexterous manipulation,”

Original

2024

Closest in time.

H. Bharadhwaj, R. Mottaghi, A. Gupta, and S. Tulsiani, “Track2act: Predicting point tracks from internet videos enables diverse zero-shot robot manipulation,”

Original

2024

Closest in time.

C. Yuan, C. Wen, T. Zhang, and Y. Gao, “General flow as foundation affordance for scalable robot learning,”

Original

2024

Closest in time.

H. He, C. Bai, L. Pan, W. Zhang, B. Zhao, and X. Li, “Large-scale actionless video pre-training via discrete diffusion for efficient policy learning,”

Original

2024

Closest in time.

M. Xu, Z. Xu, Y. Xu, C. Chi, G. Wetzstein, M. Veloso, and S. Song, “Flow as the cross-domain manipulation interface,”

Original

2024

Closest in time.

P. Li, T. Liu, Y. Li, M. Han, H. Geng, S. Wang, Y. Zhu, S.-C. Zhu, and S. Huang, “Ag2manip: Learning novel manipulation skills with agent-agnostic visual and action representations,”

Original

2024

Closest in time.

Y. Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel, “Learning universal policies via text-guided video generation,”

2024

Closest in time.

J. Wu, S. Yin, N. Feng, X. He, D. Li, J. Hao, and M. Long, “ivideogpt: Interactive videogpts are scalable world models,”

Original

2024

Closest in time.

M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi,

Original

2024

Closest in time.

X. Lai, Z. Tian, Y. Chen, Y. Li, Y. Yuan, S. Liu, and J. Jia, “Lisa: Reasoning segmentation via large language model,” in

2024

Closest in time.

T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y. Chen, F. Yan,

Original

2024

Closest in time.

Z. Zheng, X. Peng, T. Yang, C. Shen, S. Li, H. Liu, Y. Zhou, T. Li, and Y. You, “Open-sora: Democratizing efficient video production for all,” March 2024. [Online]. Available:

2024

Closest in time.