Fetching the paper…

Reading the bibliography…

2024

NeRF-Aug: Data Augmentation for Robotics with Neural Radiance Fields

Zhu, Eric, Levy, Mara, Gwilliam, Matthew et al.

Original
PDF
Post on X

Understand

Training a policy that can generalize to unknown objects is a long standing challenge within the field of robotics.

The performance of a policy often drops significantly in situations where an object in the scene was not seen during training.
To solve this problem, we present NeRF-Aug, a novel method that is capable of teaching a policy to interact with objects that are not present in the dataset.
This approach differs from existing approaches by leveraging the speed, photorealism, and 3D consistency of a neural radiance field for augmentation.

NeRF-Aug: Data Augmentation for Robotics with Neural Radiance Fields · Around

Built on

Sqil: Imitation learning via reinforcement learning with sparse rewards, 2019
Original
S. Reddy, A. D. Dragan, and S. Levine · 1905
Earlier work this paper cites.
Nerf: Representing scenes as neural radiance fields for view synthesis, 2020
Original
B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng · 2003
Earlier work this paper cites.
Generative adversarial imitation learning, 2016
Original
J. Ho and S. Ermon · 2016
Earlier work this paper cites.
Behavioral cloning from observation, 2018
Original
F. Torabi, G. Warnell, and P. Stone · 2018
Earlier work this paper cites.
Learning robust rewards with adversarial inverse reinforcement learning, 2018
Original
J. Fu, K. Luo, and S. Levine · 2018
Earlier work this paper cites.
A supervised approach to predicting noise in depth images
C. Sweeney, G. Izatt, and R. Tedrake · 2019
Earlier work this paper cites.
Hg-dagger: Interactive imitation learning with human experts, 2019
Original
M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer · 2019
Earlier work this paper cites.
6-dof contrastive grasp proposal network
X. Zhu, L. Sun, Y. Fan, and M. Tomizuka · 2021
Earlier work this paper cites.
Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps
C. Reiser, S. Peng, Y. Liao, and A. Geiger · 2021
Earlier work this paper cites.
Baking neural radiance fields for real-time view synthesis
P. Hedman, P. P. Srinivasan, B. Mildenhall, J. T. Barron, and P. E. Debevec · 2021
Earlier work this paper cites.
Efficient neural radiance fields for interactive free-viewpoint video
H. Lin, S. Peng, Z. Xu, Y. Yan, Q. Shuai, H. Bao, and X. Zhou · 2021
Earlier work this paper cites.
Plenoctrees for real-time rendering of neural radiance fields
A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa · 2021
Earlier work this paper cites.
Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction
C. Sun, M. Sun, and H.-T. Chen · 2021
Earlier work this paper cites.
Dex-nerf: Using a neural radiance field to grasp transparent objects, 2021
Original
J. Ichnowski, Y. Avigal, J. Kerr, and K. Goldberg · 2021
Earlier work this paper cites.
Fleet-dagger: Interactive robot fleet learning with scalable human supervision, 2022
Original
R. Hoque, L. Y. Chen, S. Sharma, K. Dharmarajan, B. Thananjeyan, P. Abbeel, and K. Goldberg · 2022
Earlier work this paper cites.
Nan: Noise-aware nerfs for burst-denoising, 2022
Original
N. Pearl, T. Treibitz, and S. Korman · 2022
Earlier work this paper cites.
Nerfren: Neural radiance fields with reflections, 2022
Original
Y.-C. Guo, D. Kang, L. Bao, Y. He, and S.-H. Zhang · 2022
Earlier work this paper cites.

Similar

Plenoxels: Radiance fields without neural networks
A. Yu, S. Fridovich-Keil, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa · 2022
Cited alongside, same era.
Nerf2real: Sim2real transfer of vision-guided bipedal motion skills using neural radiance fields, 2022
Original
A. Byravan, J. Humplik, L. Hasenclever, A. Brussee, F. Nori, T. Haarnoja, B. Moran, S. Bohez, F. Sadeghi, B. Vujatovic, and N. Heess · 2022
Cited alongside, same era.
Reinforcement learning with neural radiance fields, 2022
Original
D. Driess, I. Schubert, P. Florence, Y. Li, and M. Toussaint · 2022
Cited alongside, same era.
Uncertainty guided policy for active robotic 3d reconstruction using neural radiance fields, 2022
Original
S. Lee, L. Chen, J. Wang, A. Liniger, S. Kumar, and F. Yu · 2022
Cited alongside, same era.
Evo-neRF: Evolving neRF for sequential robot grasping of transparent objects

Then

Perceiving unseen 3d objects by poking the objects, 2023
Original
L. Chen, Y. Song, H. Bao, and X. Zhou · 2023
Later among the works it cites.
Nerf in the palm of your hand: Corrective augmentation for robotics via novel-view synthesis, 2023
Original
A. Zhou, M. J. Kim, L. Wang, P. Florence, and C. Finn · 2023
Later among the works it cites.
Ditto in the house: Building articulation models of indoor scenes through interactive perception, 2023
Original
C.-C. Hsu, Z. Jiang, and Y. Zhu · 2023
Later among the works it cites.
Learning universal policies via text-guided video generation, 2023
Original
Y. Du, M. Yang, B. Dai, H. Dai, O. Nachum, J. B. Tenenbaum, D. Schuurmans, and P. Abbeel · 2023
Later among the works it cites.
Zero-shot robotic manipulation with pretrained image-editing diffusion models, 2023

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

J. Kerr, L. Fu, H. Huang, Y. Avigal, M. Tancik, J. Ichnowski, A. Kanazawa, and K. Goldberg · 2022

Cited alongside, same era.

Ditto: Building digital twins of articulated objects from interaction, 2022

Z. Jiang, C.-C. Hsu, and Y. Zhu · 2022

Cited alongside, same era.

Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman · 2022

Cited alongside, same era.

Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking, 2023

H. Bharadhwaj, J. Vakil, M. Sharma, A. Gupta, S. Tulsiani, and V. Kumar · 2023

Cited alongside, same era.

Genaug: Retargeting behaviors to unseen situations via generative augmentation, 2023

Z. Chen, S. Kiami, A. Gupta, and V. Kumar · 2023

Cited alongside, same era.

Cacti: A framework for scalable multi-task multi-scene visual imitation learning, 2023

Z. Mandi, H. Bharadhwaj, V. Moens, S. Song, A. Rajeswaran, and V. Kumar · 2023

Cited alongside, same era.

Scaling robot learning with semantically imagined experience

T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, M. Dee, J. Peralta, B. Ichter, K. Hausman, and F. Xia · 2023

Cited alongside, same era.

Dall-e-bot: Introducing web-scale diffusion models to robotics

I. Kapelyukh, V. Vosylius, and E. Johns · 2023

Cited alongside, same era.

K. Black, M. Nakamoto, P. Atreya, H. Walke, C. Finn, A. Kumar, and S. Levine · 2023

Later among the works it cites.

Nerfstudio: A modular framework for neural radiance field development

M. Tancik, E. Weber, E. Ng, R. Li, B. Yi, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja, D. Mcallister, J. Kerr, and A. Kanazawa · 2023

Later among the works it cites.

Segment anything, 2023

A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick · 2023

Later among the works it cites.

Instructpix2pix: Learning to follow image editing instructions, 2023

T. Brooks, A. Holynski, and A. A. Efros · 2023

Later among the works it cites.

Baku: An efficient transformer for multi-task policy learning, 2024

S. Haldar, Z. Peng, and L. Pinto · 2024

Closest in time.

Diffusion meets dagger: Supercharging eye-in-hand imitation learning, 2024

X. Zhang, M. Chang, P. Kumar, and S. Gupta · 2024

Closest in time.

Mega-dagger: Imitation learning with multiple imperfect experts, 2024

X. Sun, S. Yang, M. Zhou, K. Liu, and R. Mangharam · 2024

Closest in time.

Inverse reinforcement learning without reinforcement learning, 2024

G. Swamy, S. Choudhury, J. A. Bagnell, and Z. S. Wu · 2024

Closest in time.

Dream2real: Zero-shot 3d object rearrangement with vision-language models, 2024

I. Kapelyukh, Y. Ren, I. Alzugaray, and E. Johns · 2024

Closest in time.

Reconciling reality through simulation: A real-to-sim-to-real approach for robust manipulation, 2024

M. Torne, A. Simeonov, Z. Li, A. Chan, T. Chen, A. Gupta, and P. Agrawal · 2024

Closest in time.

Ardup: Active region video diffusion for universal policies, 2024

S. Huang, M. Levy, Z. Jiang, A. Anandkumar, Y. Zhu, L. Fan, D.-A. Huang, and A. Shrivastava · 2024

Closest in time.

Rovi-aug: Robot and viewpoint augmentation for cross-embodiment robot learning, 2024

L. Y. Chen, C. Xu, K. Dharmarajan, M. Z. Irshad, R. Cheng, K. Keutzer, M. Tomizuka, Q. Vuong, and K. Goldberg · 2024

Closest in time.

Putting the object back into video object segmentation, 2024

H. K. Cheng, S. W. Oh, B. Price, J.-Y. Lee, and A. Schwing · 2024

Closest in time.