Dexterous Teleoperation of 20-DoF ByteDexter Hand <br> via Human Motion Retargeting

Abstract

Replicating human-level dexterity remains a fundamental robotics challenge, requiring integrated solutions from mechatronic design to the control of high degree-of-freedom (DoF) robotic hands. While imitation learning shows promise in transferring human dexterity to robots, the efficacy of trained policies relies on the quality of human demonstration data. We bridge this gap with a hand-arm teleoperation system featuring: (1) a 20-DoF linkage-driven anthropomorphic robotic hand for biomimetic dexterity, and (2) an optimization-based motion retargeting for real-time, high-fidelity reproduction of intricate human hand motions and seamless hand–arm coordination. We validate the system via extensive empirical evaluations, including dexterous in-hand manipulation tasks and a long-horizon task requiring the organization of a cluttered makeup table randomly populated with nine objects. Experimental results demonstrate its intuitive teleoperation interface with real-time control and the ability to generate high-quality demonstration data.

Method

ByteDexter Hand Design

ByteDexter is an anthropomorphic dexterous hand developed by ByteDance Seed. It features 20 degrees of freedom (DoF), with 15 fully actuated DoFs. The seamless integration of 15 motors and embedded boards within the palm enables ByteDexter to function as a standalone hand module. Both the fingertips and palm feature dedicated designs to facilitate the integration of tactile sensors.

Finger Dexterity Demonstration

Teleoperation via Data Glove

Grasping of Daily Objects

In-Hand Manipulation

ByteDexter’s finger design adopts a parallel-serial kinematic structure: a 2-DoF MCP joint, 1-DoF PIP joint, and passively driven 1-DoF DIP joint replicate the kinematics of human index, middle, ring, and pinky fingers. Its novel thumb design decouples the MCP joint’s two DoFs, expanding the workspace and enhancing precision grasping performance.

Hand-Arm Teleoperation System

The teleoperation interface comprises a Meta Quest 3 headset to track wrist poses and a Manus Quantum Metaglove for hand motions. The Quest controller is mounted to Manus glove’s back via a custom holder to ensure synchronous tracking of wrist and finger movements. On the robotic side, the ByteDexter hand is mounted to a Franka Research 3 (FR3) arm, with its wrist–fingertip axis aligned to the arm’s seventh joint. The operators’ wrist poses captured by Quest headset are mapped directly to the FR3 end-effector, while hand motions from the Manus Glove are retargeted into joint position commands of the ByteDexter hand.

Hand Motion Retargeting

We retarget human hand pose data obtained from the Manus Glove into joint position references of the ByteDexter hand by solving an optimization problem that minimizes the difference between corresponding keyvectors in the robotic and human hand.

The right figure shows only the index-finger keyvectors for illustration. Iterating this process from the thumb to pinky and eliminating duplicates results in 15 unique keyvectors. Incorporating these keyvectors into optimization captures the majority of the grasping types: for pinch grasping, minimizing inter-finger distances aligns robotic and human hand poses to reduce operator strain; for power grasping, preserving fingertip-to-palm distances are prioritized.

Results

In-hand Manipulation Results

We evaluate three canonical in-hand manipulation primitives—(i) regrasping to translate objects between fingers, (ii) sliding objects relative to the palm, and (iii) rotating objects or their components, and instantiate them in four benchmark tasks: (1) regrasping a bottle from a precision to a power grasp; (2) multi-object grasping; (3) lid unscrewing; and (4) push-to-open lids.

In-hand Manipulation Tasks

Long-Horizon Teleoperation Results

To assess long-horizon teleoperation capabilities, we evaluated the system using a table organization task. The scenario featured a cluttered workspace filled with randomly arranged cosmetic and skincare items alongside a multi-drawer organizer. This required the robotic hand-arm system to: (1) retrieve objects from the table; (2) place items into designated organizer compartments; and (3) successfully open the drawer and insert objects into it.

Long-Horizon Table Organization Task

Citation

@misc{bytedance2025bytedexter,
      title={Dexterous Teleoperation of 20-DoF ByteDexter Hand via Human Motion Retargeting}, 
      author={Ruoshi Wen and Jiajun Zhang and Guangzeng Chen and Zhongren Cui and Min Du and Yang Gou and Zhigang Han and Junkai Hu and Liqun Huang and Hao Niu and Wei Xu and Haoxiang Zhang and Zhengming Zhu and Hang Li and Zeyu Ren},
      year={2025},
      eprint={2507.03227},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2507.03227}, 
}