Reconstructing Hand-Held Objects in 3D from Images and Videos

Understand

Objects manipulated by the hand (i.e., manipulanda) are particularly challenging to reconstruct from Internet videos.

• Not only does the hand occlude much of the object, but also the object is often only visible in a small number of image pixels.
• At the same time, two strong anchors emerge in this setting: (1) estimated 3D hands help disambiguate the location and scale of the object, and (2) the set of manipulanda is small relative to all possible objects.
• With these insights in mind, we present a scalable paradigm for hand-held object reconstruction that builds on recent breakthroughs in large language/vision models and 3D object datasets.