Git Product home page Git Product logo

threestudio-dreamwaltz's Introduction

threestudio-dreamwaltz

DreamWaltz extension of threestudio. To use it, please install threestudio first and then install this extension in threestudio custom directory.

Installation

cd custom
git clone https://github.com/huanngzh/threestudio-dreamwaltz.git
cd threestudio-dreamwaltz

pip install -r requirements.txt

If installing the pytorch3d package fails, please see the detailed instructions at pytorch3d/INSTALL.md.

Prepare SMPL Weights

We use smpl and vposer models for avatar creation and animation learning, please follow the instructions in smplx and human_body_prior to download the model weights, and build a directory with the following structure:

smpl_models
├── smpl
│   ├── SMPL_FEMALE.pkl
│   └── SMPL_MALE.pkl
│   └── SMPL_NEUTRAL.pkl
└── vposer
    └── v2.0
        ├── snapshots
        ├── V02_05.yaml
        └── V02_05.log

Then, update the model paths SMPL_ROOT and VPOSER_ROOT in utils/smpl/smpl_prompt.py.

Quick Start

Static Avatar Creation

All in one (SMPL Initializaion + Canonical Avatar Creation):

python launch.py --config custom/threestudio-dreamwaltz/configs/dreamwaltz-static.yaml --train --gpu 0 system.prompt_processor.prompt="Naruto"

Divided into multiple stages:

# SMPL Initializaion
python launch.py --config custom/threestudio-dreamwaltz/configs/experimental/dreamwaltz-1-warmup.yaml --train --gpu 0 system.prompt_processor.prompt="Naruto"
# Canonical Avatar Creation
python launch.py --config custom/threestudio-dreamwaltz/configs/experimental/dreamwaltz-2-nerf.yaml --train --gpu 0 system.prompt_processor.prompt="Naruto" resume=path/to/trial/dir/ckpts/last.ckpt

Animatable Avatar Learning

Not yet implemented!

Citing

If you find DreamWaltz helpful, please consider citing:

@article{huang2023dreamwaltz,
    title={DreamWaltz: Make a Scene with Complex 3D Animatable Avatars},
    author={Yukun Huang and Jianan Wang and Ailing Zeng and He Cao and Xianbiao Qi and Yukai Shi and Zheng-Jun Zha and Lei Zhang},
    journal = {arXiv:2305.12529},
    year={2023},
}

threestudio-dreamwaltz's People

Contributors

huanngzh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

threestudio-dreamwaltz's Issues

error running launch.py - ImportError: cannot import name 'bool' from 'numpy'

(pytorch3d) andy@andys-pc:~/threestudio$ python launch.py --config custom/threestudio-dreamwaltz/configs/dreamwaltz-static.yaml --train --gpu 0 system.prompt_processor.prompt="Naruto"
/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: '/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/torchvision/image.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKSs'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
  warn(
/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/controlnet_aux/mediapipe_face/mediapipe_face_common.py:7: UserWarning: The module 'mediapipe' is not installed. The package will have limited functionality. Please install it using the command: pip install 'mediapipe'
  warnings.warn(
/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_5m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_5m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_11m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_11m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_384 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_384. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_512 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_512. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
    PyTorch 2.2.0+cu121 with CUDA 1201 (you have 2.1.2.post301)
    Python  3.9.18 (you have 3.9.18)
  Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
  Memory-efficient attention, SwiGLU, sparse and more won't be available.
  Set XFORMERS_MORE_DETAILS=1 for more details

Import times for custom modules:
   0.0 seconds: custom/threestudio-animate124
   0.9 seconds: custom/threestudio-dreamwaltz

Seed set to 0
[INFO] Using prompt [Naruto] and negative prompt []
[INFO] Using view-dependent prompts [side]:[Naruto, side view] [front]:[Naruto, front view] [back]:[Naruto, back view] [overhead]:[Naruto, overhead view]
[INFO] Loading Multi-ControlNet...
[INFO] Loaded Multi-ControlNet!
[INFO] Using 16bit Automatic Mixed Precision (AMP)
[INFO] GPU available: True (cuda), used: True
[INFO] TPU available: False, using: 0 TPU cores
[INFO] IPU available: False, using: 0 IPUs
[INFO] HPU available: False, using: 0 HPUs
[INFO] You are using a CUDA device ('NVIDIA GeForce RTX 3060') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/chumpy/__init__.py:11: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.
  from numpy import bool, int, float, complex, object, unicode, str, nan, inf
/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/chumpy/__init__.py:11: FutureWarning: In the future `np.object` will be defined as the corresponding NumPy scalar.
  from numpy import bool, int, float, complex, object, unicode, str, nan, inf
/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/chumpy/__init__.py:11: FutureWarning: In the future `np.str` will be defined as the corresponding NumPy scalar.
  from numpy import bool, int, float, complex, object, unicode, str, nan, inf
Traceback (most recent call last):
  File "/home/andy/threestudio/launch.py", line 301, in <module>
    main(args, extras)
  File "/home/andy/threestudio/launch.py", line 244, in main
    trainer.fit(system, datamodule=dm, ckpt_path=cfg.resume)
  File "/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 543, in fit
    call._call_and_handle_interrupt(
  File "/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 579, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 948, in _run
    call._call_setup_hook(self)  # allow user to set up LightningModule in accelerator environment
  File "/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py", line 92, in _call_setup_hook
    _call_lightning_datamodule_hook(trainer, "setup", stage=fn)
  File "/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py", line 179, in _call_lightning_datamodule_hook
    return fn(*args, **kwargs)
  File "/home/andy/threestudio/custom/threestudio-dreamwaltz/data/smpl_prior.py", line 300, in setup
    self.train_dataset = RandomSMPLPriorCameraIterableDataset(self.cfg)
  File "/home/andy/threestudio/custom/threestudio-dreamwaltz/data/smpl_prior.py", line 50, in __init__
    self.smpl_prompt = SMPLPrompt(
  File "/home/andy/threestudio/custom/threestudio-dreamwaltz/utils/smpl/smpl_prompt.py", line 1288, in __init__
    self.hs = CanonicalScene(scene=scene, offset_y=smpl_offset_y)
  File "/home/andy/threestudio/custom/threestudio-dreamwaltz/utils/smpl/smpl_prompt.py", line 1129, in __init__
    smpl = MySMPL(batch_size=self.num_person, model_type=model_type)
  File "/home/andy/threestudio/custom/threestudio-dreamwaltz/utils/smpl/smpl_prompt.py", line 575, in __init__
    self.model = smplx.create(**smpl_cfgs)
  File "/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/smplx/body_models.py", line 2400, in create
    return SMPL(model_path, **kwargs)
  File "/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/smplx/body_models.py", line 137, in __init__
    data_struct = Struct(**pickle.load(smpl_file,
  File "/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/chumpy/__init__.py", line 11, in <module>
    from numpy import bool, int, float, complex, object, unicode, str, nan, inf
ImportError: cannot import name 'bool' from 'numpy' (/home/andy/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/numpy/__init__.py)```

Error during install - pyrender

Collecting pyrender (from -r requirements.txt (line 6))
Downloading pyrender-0.1.45-py3-none-any.whl (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 3.7 MB/s eta 0:00:00
ERROR: Could not find a version that satisfies the requirement open3d==0.17.0 (from versions: 0.18.0)
ERROR: No matching distribution found for open3d==0.17.0

I'm running Ubuntu with WSL

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.