M3D-VTON: A Monocular-to-3D Virtual Try-On Network
Official code for ICCV2021 paper "M3D-VTON: A Monocular-to-3D Virtual Try-on Network"
Paper | Supplementary | MPV3D Dataset | Pretrained Models
Requirements
python >= 3.8.0, pytorch == 1.6.0, torchvision == 0.7.0
Data Processing
After downloading the MPV3D Dataset, please run the following script to preprocess the data:
python util/data_preprocessing.py --MPV3D_root path/to/MPV3D/dataset
Running Inference
We provide demo inputs under the mpv3d_example folder, where the target clothing and the reference person are like:
with inputs from the mpv3d_example folder, the easiest way to get start is to use the pretrained models and sequentially run the four steps below:
1. Testing MTM Module
python test.py --model MTM --name MTM --dataroot mpv3d_example --datalist test_pairs --results_dir results
2. Testing DRM Module
python test.py --model DRM --name DRM --dataroot mpv3d_example --datalist test_pairs --results_dir results
3. Testing TFM Module
python test.py --model TFM --name TFM --dataroot mpv3d_example --datalist test_pairs --results_dir results
4. Getting colored point cloud and Remeshing
(Note: since the back-side person images are unavailable, in rgbd2pcd.py we provide a fast face inpainting function that produces the mirrored back-side image after a fashion. One may need manually inpaint other back-side texture areas to achieve better visual quality.)
python rgbd2pcd.py
Now you should get the point cloud file prepared for remeshing under results/aligned/pcd/test_pairs/*.ply. MeshLab can be used to remesh the predicted point cloud, with two simple steps below:
-
Normal Estimation: Open MeshLab and load the point cloud file, and then go to Filters --> Normals, Curvatures and Orientation --> Compute normals for point sets
-
Possion Remeshing: Go to Filters --> Remeshing, Simplification and Reconstruction --> Surface Reconstruction: Screen Possion (set reconstruction depth = 9)
Now the final 3D try-on result should be obtained:
Training on MPV3D Dataset
With the pre-processed MPV3D dataset, you can train the model from scratch by folllowing the three steps below:
1. Train MTM module
python train.py --model MTM --name MTM --dataroot path/to/MPV3D/data --datalist train_pairs --checkpoints_dir path/for/saving/model
then run the command below to obtain the --warproot (here refers to the --results_dir) which is necessary for the other two modules:
python test.py --model MTM --name MTM --dataroot path/to/MPV3D/data --datalist train_pairs --checkpoints_dir path/to/saved/MTMmodel --results_dir path/for/saving/MTM/results
2. Train DRM module
python train.py --model DRM --name DRM --dataroot path/to/MPV3D/data --warproot path/to/MTM/warp/cloth --datalist train_pairs --checkpoints_dir path/for/saving/model
3. Train TFM module
python train.py --model TFM --name TFM --dataroot path/to/MPV3D/data --warproot path/to/MTM/warp/cloth --datalist train_pairs --checkpoints_dir path/for/saving/model
(See options/base_options.py and options/train_options.py for more training options.)
License
The use of this code and the MPV3D dataset is RESTRICTED to non-commercial research and educational purposes.
Citation
If our code is helpful to your research, please cite:
@article{Zhao2021M3DVTONAM,
title={M3D-VTON: A Monocular-to-3D Virtual Try-On Network},
author={Fuwei Zhao and Zhenyu Xie and Michael C. Kampffmeyer and Haoye Dong and Songfang Han and Tianxiang Zheng and Tao Zhang and Xiaodan Liang},
journal={ArXiv},
year={2021},
volume={abs/2108.05126}
}
36 Nov 4, 2022
83 Nov 27, 2022
103 Dec 14, 2022
159 Dec 30, 2022
148 Dec 16, 2022
69 Oct 13, 2022
221 Jan 7, 2023
261 Jan 9, 2023
187 Dec 26, 2022
229 Dec 20, 2022
118 Dec 26, 2022
82 Dec 27, 2022
202 Dec 30, 2022
18 Sep 2, 2022
112 Nov 28, 2022
107 Dec 20, 2022