Fine Virtual Manipulation with Hands of Different Sizes

Suzanne Sorli, Dan Casas, Mickeal Verschoor, Ana Tajadura Jiménez and Miguel A. Otaduy
Proc. of the International Symposium on Mixed and Augmented Reality (ISMAR), 2021


Natural interaction with virtual objects relies on two major technology components: hand tracking and hand-object physics simulation. There are functional solutions for these two components, but their hand representations may differ in size and skeletal morphology, hence making the connection non-trivial. In this paper, we introduce a pose retargeting strategy to connect the tracked and simulated hand representations, and we have formulated and solved this hand retargeting as an optimization problem. We have also carried out a user study that demonstrates the effectiveness of our approach to enable fine manipulations that are slow and awkward with naïve approaches.

Keywords: Hand simulation, hand tracking, pose retargeting



author       = "Sorli, Suzanne and Casas, Dan and Verschoor, Mickeal and Tajadura-Jiménez, Ana and Otaduy, Miguel A.",
title        = "Fine Virtual Manipulation with Hands of Different Sizes",
booktitle    = "Proc. of the International Symposium on Mixed and Augmented Reality (ISMAR)",
year         = "2021",
url          = ""

Description and Results

We introduced a pose retargeting strategy to connect the tracked hand and the simulated hand. Our approach works with any type of tracking or simulation method, as it stands at the interface between both tasks. We use an intermediate hand representation that shares the size and morphology of the simulated hand, but which tries to match the configuration of the tracked hand. The retargeting strategy formulates a constrained optimization of the pose of this intermediate hand, based on features that represent the pose of the tracked hand. We have used finger tip positions as features, as they represent key information for fine manipulation. Finally, we solved the optimization iteratively using the Gauss Newton method.

We wanted to evaluate the practical impact and see if our retargeting strategy had any effect on the dexterity of interaction. To do so, we designed a user experiment in which the users had to execute a manipulation task that consisted of picking a cube with their thumb and index fingers from position A, move it to position B, and drop it there. Two different strategies and manipulations were studied in this experiment.

First, we compared our pose retargeting strategy, referred to as Ours, with a naïve retargeting strategy, which has been carried out by aligning the palms of the tracked and intermediate hands and then directly copying the joint angles of the tracked hand to the intermediate hand, which is referred to as Naïve.

Secondly, to study fine and gross manipulations, we respectively used a small 1cm wide cube and another one, bigger, of 6 cm wide. In each experimental trial, we asked to the users to execute the cube manipulation task, testing both types of manipulation and under both strategy conditions which amounts to 4 trials per block, whose order was randomized. Then, each block has been repeated 5 times in order to evaluate the learning effect. For each experimental condition, we measured the time needed to complete the manipulation task. Moreover, for the last experimental block, participants were asked to complete a questionnaire to assess participant’s subjective feelings of dexterity, in terms of precision, ease and naturalness. Finally, a total of 20 right-handed participants took part in the user study and we asked them their age, hand size and prior VR experience.

left: Our strategy (Ours), right: Naïve strategy (Naïve).

The analysis of task performance and self-reporting of the user experiment suggests benefits of the proposed pose retargeting strategy. Moreover, these benefits appear independent of hand size, VR experience or age. First and foremost, Our Strategy exhibits significantly better performance than the Naïve Strategy for Fine Manipulation. In addition, the Naïve Strategy performs significantly worse on Fine Manipulation vs. Gross Manipulation, while Our Strategy does not exhibit a significant performance difference on these two Manipulation conditions. Hand size has an effect on performance for Fine Manipulation regardless of the retargeting Strategy, but Our Strategy performs better than the Naïve Strategy consistently across hand sizes.

The questionnaires suggest that Our pose retargeting Strategy feels significantly more precise, easier, and more natural for Fine Manipulation. For Gross Manipulation, Our Strategy scores slightly lower than the Naïve Strategy, but the difference is not significant. This is likely due to the inherent easiness of the Gross Manipulation scenario, which is confirmed when analyzing task performance across repetitions: Fine Manipulation benefits from learning more significantly than Gross Manipulation. Similarly, the analysis of task performance across repetitions indicates that the performance gain of Our Strategy is even larger initially, which again suggests that it is more natural, i.e., it requires less training.


We would like to thank the reviewers for their feedback, Jiayi Wang for the hand texture, Jessica Illera for the help with the study, and other members of the MSLab at URJC for their support.


This work was funded in part by the European Research Council (ERC Consolidator Grant no.772738 TouchDesign) and the Spanish Ministry of Science (RTI2018-098694-B-I00 VizLearning; PID2019-105579RB-I00/AEI/10.13039/501100011033).


Suzanne Sorli –
Dan Casas –