PERGAMO: Personalized 3D Garments from Monocular Video

Andrés Casado-Elvira, Marc Comino Trinidad and Dan Casas
Computer Graphics Forum
Proc. of the ACM SIGGRAPH / Eurographics Symposium on Computer Animation
, 2022


Clothing plays a fundamental role in digital humans. Current approaches to animate 3D garments are mostly based on realistic physics simulation, however, they typically suffer from two main issues: high computational run-time cost, which hinders their development; and simulation-to-real gap, which impedes the synthesis of specific real-world cloth samples. To circumvent both issues we propose PERGAMO, a data-driven approach to learn a deformable model for 3D garments from monocular images. To this end, we first introduce a novel method to reconstruct the 3D geometry of garments from a single image, and use it to build a dataset of clothing from monocular videos. We use these 3D reconstructions to train a regression model that accurately predicts how the garment deforms as a function of the underlying body pose. We show that our method is capable of producing garment animations that match the real-world behaviour, and generalizes to unseen body motions extracted from motion capture dataset.


@article {casado2022pergamo,
    journal = {Computer Graphics Forum (Proc. of SCA), 2022},
    title = {{PERGAMO}: Personalized 3D Garments from Monocular video},
    author = {Casado-Elvira, Andrés and Comino Trinidad, Marc and Casas, Dan},
    year = {2022}

Description and Results

We introduce an approach to learn a deformation model for 3D garments from a single monocular video. PERGAMO is based on two key features: it is learned from casual real-world images, hence there is no simulation-to-real gap or need for multi-camera setups; and it is highly-efficient to evaluate, since at inference time it uses a shallow neural network that directly outputs garment deformations. All in all, our main contribution is a 3D clothing reconstruction pipeline that is able to recover the explicit layer of a garment from just monocular RGB input. These reconstructions enable us to train a data-driven model to infer how a specific garment deforms.

To formulate PERGAMO, we use a novel two-stage approach where we first build a dataset by reconstructing the 3D geometry of deformed garments, and then learn a nonlinear regressor from the reconstructed meshes. More specifically, we initially extract human-related features such as body segmentation, body pose, and body normals, from the input images, which we leverage to deform a mesh template to reconstruct fine-scale detailed clothing using a differentiable rendering optimization. Then, we use the reconstructed garments as ground truth data to train a 3D garment deformation regressor. We show that the learned model outputs pose-dependent garment surface details, such as fold and wrinkles, that closely match the real-world behaviour of the garment.

Below we depict an overview of our 3D garment reconstructing approach. Starting from an RGB frame, we first extract human semantic and 3D information. We then fit a coarse garment template to the estimated body normals, and finally we add fine-scale wrinkles by optimizing per-vertex displacement using differentiable rendering.

Below we show a visualization of the optimization process used to reconstruct garments.

The following animations show some video clips taken with a mobile phone camera side-by-syde with their respective reconstruction made by our method.

Below are some comparisons against MonoClothCap, another reconstruction method.

After creating a dataset of reconstructions, a regressor can be trained to predict a garment for a given pose. Here we show some examples for test motion sequences from the AMASS dataset.


This work has been partially funded by:
  • Comunidad de Madrid in the framework of the Multiannual Agreement with the Universidad Rey Juan Carlos in line of Action 1, "Encouragement of research for young PhD", project CaptHuRe (M2736)
  • Universidad Rey Juan Carlos through the Distinguished Research position INVESDIST-04 under the call from 17/12/2020
  • Leonardo Fellowship from the Fundación BBVA


Dan Casas –