Small-object handling
Close-up demonstration of precise grasping.
Hardware components
A ROBOT-FREE DATA COLLECTION SYSTEM
Handheld Universal Manipulation Interface (UMI) devices let human experts demonstrate tasks directly. Preparing these demonstrations for learning still requires consistent hardware, calibration, and data processing. We present UMI-Live, a bimanual hardware and software system for in-the-wild collection and robot learning. Each gripper integrates camera and LiDAR–inertial sensing, onboard compute, and power, without external tracking equipment or calibration of the collection environment.
A single button controls recording while an asynchronous pipeline performs parallel and batched reconstruction and publishes validated LeRobot datasets. We evaluate π0.5 and MolmoAct2 on placing six oranges in a bowl, with up to 10/10 and 8/10 successful trials, respectively. We release approximately 40 hours across three tasks and plan to release hardware designs, software, and reproduction guides at the project repository.
Loading 3D view…
Drag to rotate; scroll to zoom. A 3D Gaussian splat of one UMI-Live unit, reconstructed from handheld phone video.
Bimanual tasks directly collected with the handheld UMI devices.
Close-up demonstration of precise grasping.
Bimanual match-lighting demonstration with the handheld UMI devices.
Long-horizon bimanual collection, shown as a time-lapse.
Each clip pairs a human task recording with synchronized wrist-camera views and the gripper motion reconstructed from it.
A fruit-task recording, shown alongside its wrist-camera views and reconstructed gripper motion.
A second task example with the recorded camera views and reconstructed gripper motion shown together.
π0.5 + RTC uses local inference; all other policy rollouts shown here use remote inference.
Local inference. Autonomous rollout; all six oranges finish in the bowl.
Wrist-camera-only rollout; one orange remains on the table.
Rollout without the top camera; an operator adjustment is visible.
Operator interaction is visible; the run ends before all oranges are placed.
For teleoperation, VIVE trackers replace the LiDAR on each UMI gripper, and Lighthouse base stations track their pose. After tracker-to-robot alignment and a captured reference pose, UMI motion sets Cartesian targets for each YAM arm; inverse kinematics converts them to joint commands. Wrist-camera ArUco tags provide gripper opening.
The corresponding dual-arm YAM task, controlled from the handheld setup.
The dual YAM arms approach, grasp, and move the matchbox using the teleoperation setup above.
@article{velamoor2026umilive,
author = {Velamoor, Govind and Lin, Adrian and Jung, Sam and
Mateega, Spencer and Yang, Brandon Y. and Sethi, Amish},
title = {UMI-Live for In-The-Wild Bimanual Robotics Data Collection},
year = {2026}
}