Track, Articulate, Act Generating Articulation from Casual Human Videos
Brains, Bots, and Behavior LabWe explore how far pretrained vision models, combined with optimization, can take us in recovering an object’s articulation and re-targeting the observed human interaction in physics simulation, using only a single casually captured monocular video.
Overview
Human videos contain rich causal evidence for robot manipulation: they reveal how hand motion induces object motion and produces task-relevant changes in object state. In this work, we study articulated objects such as doors, drawers, cabinets, laptops, ovens, and hinged containers that are ubiquitous in daily life and present unique challenges for embodied interaction. These objects cannot be represented by a single pose; their motion depends on the underlying parts and joints. We introduce a real-to-sim framework that reconstructs a simulation-ready articulated object and hand–object interaction from a casual monocular RGB video, without RGB-D or multi-view input, prior scans, manually specified joints, or robot demonstrations. Our key insight is that dense 3D point tracks provide an embodiment-agnostic articulation cue: points on the fixed link remain approximately stationary, while points on the moving link follow coherent revolute or prismatic motion. Our method segments the links, estimates the joint and its state trajectory, reconstructs an articulated asset, and aligns the recovered 3D hand motion with the object. Central to our approach is a modular recipe that repurposes powerful pretrained models for single-image 3D reconstruction, mesh segmentation, and 3D scene flow, connecting their predictions through explicit geometric reasoning to infer articulation. We use the reconstructed articulated object and the human hand trajectory to replay interactions through contact in MuJoCo. The framework shows how pretrained vision models and explicit motion reasoning can turn casual human videos into articulated object models suitable for downstream embodied interactions.
Method
We start with a few frames that show different articulation states and where the hand obscures as little of the object as possible. SAM 3 tracks the object and hand masks, SAM 3D reconstructs an object mesh from each selected frame, and Depth Anything 3 estimates depth and camera geometry. We use part masks to guide SegviGen in separating each mesh into static and moving links, then align the reconstructions through the static link.
Track
TrackCraft3R follows points across the video. We lift these tracks into 3D, separate static and moving trajectories, and use them to guide the part masks. Grounding the tracks on the registered meshes brings the motion and geometry into a common object frame.
Articulate
We fit revolute and prismatic joint hypotheses to the moving points, then refine the joint and its motion against the tracks and object silhouettes. We fuse the aligned link meshes and add joint limits and collision geometry to form a simulation-ready asset.
Act
HaWoR recovers the 3D hand trajectory. We align its scale and pose with the object, then re-target the motion to a simulated hand in MuJoCo. The object joint stays passive, so its movement comes from hand contact rather than a commanded joint trajectory.
Human videos
The RGB recordings used to reconstruct these objects and their motion.
Recovered articulation
The reconstructed objects and their joints, in the same order as the input videos above.
Actionable simulation results
We replay the recovered interactions in MuJoCo with a human hand and an Allegro robot hand, demonstrating how the reconstructed articulated object can be reliably manipulated.
Retargeting to a simulated human hand
Retargeting to a simulated robot hand (Allegro)
Comparison with RGB-D
When depth from an iPhone camera is available, our framework can use it directly to improve the result. Below, we compare the same suitcase interaction using RGB alone and RGB-D with iPhone depth.