buaacyw / gaussianeditor Goto Github PK

When I use command line to add a teddy bear on bench in bicycle dataset by python launch.py --config configs/add.yaml --train --gpu 0 data.source=../autodl-tmp/dataset/bicycle system.gs_source=../autodl-tmp/dataset/bicycle/point_cloud/iteration_7000/point_cloud.ply system.inpaint_prompt="a teddy bear on bench" system.refine_prompt="make it a teddy bear" system.cache_overwrite=False system.cache_dir="add_bear" trainer.max_steps=1 system.loggers.wandb.enable=true system.loggers.wandb.name="add_bear", I have encountered this problem: FileNotFoundError: [Errno 2] No such file or directory: '.cache/dpt/omnidata_dpt_depth_v2.ckpt' and the program was interrupted.

By the way, the terminal also outputs Traceback (most recent call last): File "launch.py", line 7, in <module> if os.environ['USER']=="yiwen": File "/root/miniconda3/lib/python3.8/os.py", line 675, in __getitem__ raise KeyError(key) from None KeyError: 'USER', does it matters?

Hi, I'm trying to run the webUI but the viser produces the following output:

(viser) No client build found. Building now...
(viser) nodejs is set up!
node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by node)
node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by node)
╭─────────────── viser ───────────────╮
│             ╷                       │
│   HTTP      │ http://0.0.0.0:8084   │
│   Websocket │ ws://0.0.0.0:8084     │
│             ╵                       │
╰─────────────────────────────────────╯

and the localhost http://0.0.0.0:8084 displays nothing but 404.

It's dangerous to upgrade GLIBC to version 2.28, and I wonder if there are any other solutions. Thanks!

Hi, thanks for your impressive work firstly.

However, when I try to add an instance to the scene, the code threw an error:
FileNotFoundError: [Errno 2] No such file or directory: '.cache/dpt/omnidata_dpt_depth_v2.ckpt'

I suppose it is because the weights for depth estimation is missed. I tried to download it manually and put it under the utils/cache/dpt/ folder, but it still doesn't work. Do you know how to solve it?

This is the full logs about the error, it happened after the Wonder3D reconstruction.

Traceback (most recent call last):
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1739, in write_video
    writer.add_image(image)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1648, in add_image
    if self._proc.stdin.write(data) != len(data):
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train_from_mesh.py", line 261, in <module>
    cams, images, camera_extent = render_multiview_images_from_mesh(args.mesh,
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/threestudio/utils/render.py", line 150, in render_multiview_images_from_mesh
    write_video(save_path, frames, fps=2)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1739, in write_video
    writer.add_image(image)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1610, in __exit__
    self.close()
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1661, in close
    raise RuntimeError(f"Error writing '{self.path}': {s}")
RuntimeError: Error writing '/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/tmp_add/mesh_rendering.mp4': /home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/bin/ffmpeg: error while loading shared libraries: libopenh264.so.5: cannot open shared object file: No such file or directory

/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/timm/models/_factory.py:117: UserWarning: Mapping deprecated model name vit_base_resnet50_384 to current vit_base_r50_s16_384.orig_in21k_ft_in1k.
  model = create_fn(
Traceback (most recent call last):
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 449, in wrapped
    inner()
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 257, in <lambda>
    lambda: self._handle_incoming_message(client_id, message)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 94, in _handle_incoming_message
    cb(client_id, message)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/_gui_api.py", line 157, in _handle_gui_updates
    cb(GuiEvent(client, client_id, handle))
  File "webui.py", line 511, in _
    self.add(self.camera)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "webui.py", line 1371, in add
    depth_estimator = DPT(get_device(), mode="depth")
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/threestudio/utils/dpt.py", line 1023, in __init__
    checkpoint = torch.load(path, map_location="cpu")
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/serialization.py", line 986, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/serialization.py", line 435, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/serialization.py", line 416, in __init__
    super().__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '.cache/dpt/omnidata_dpt_depth_v2.ckpt'

This project is excellent. But which part of the code the formula (6) in the paper corresponds to，I can't find it.

Hi, encounter a problem using add_dog.sh

RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DEFAULT_TENSOR_OP)

Can you please take a look?

Hi, if I want to output a rendered video of the edited scene, how can I make it?

Has anyone else had this problem?

File "./GaussianEditor/threestudio/systems/base.py", line 108, in training_step
raise NotImplementedError
NotImplementedError

Hello!

I tried adding a car in my view, and it worked fine. The problem is that it only had that view of the car. So, if I rotated the view an inch, the car disappears.

However, in your demo with the teddy bear, it seemed it was a 3d object.

How did you do that?

I can't understand self.gaussian.apply_weights(cur_cam, weights, weights_cnt, mask) in the process of update masks.

def apply_weights(self, camera, weights, weights_cnt, image_weights):

    rasterizer = camera2rasterizer(
        camera, torch.tensor([0.0, 0.0, 0.0], dtype=torch.float32, device="cuda")
    )
    rasterizer.apply_weights(
        self.get_xyz,
        None,
        self.get_opacity,
        None,
        weights,
        self.get_scaling,
        self.get_rotation,
        None,
        weights_cnt,
        image_weights,
    )

Please tell me about what does weights_cnt mean and how this function work ?

Great work!
Hello, could you introduce how to use Delta Denoising Score in GaussianEditor?

I have tried to install it on windows and seems everything has been correctly set up. However, when I try to run the webui.py, I meet bellow errors:

`(GaussianEditor) PS D:\01_DL\GaussianEditor> python webui.py --gs_source D:\01_DL\gaussian-splatting\output\armour_museum_house_center-600F_896-20F-CF1-1K6_cyberpunck\point_cloud\iteration_50000\point_cloud.ply --colmap_dir D:\01_DL\gaussian-splatting\data\finland\armour_museum_house_center-600F_896-20F-CF1_cyberpuck-1K6
Reading camera 502/502
C:\ProgramData\Miniconda3\envs\GaussianEditor\lib\site-packages\torch\functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ..\aten\src\ATen\native\TensorShape.cpp:3484.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
final text_encoder_type: bert-base-uncased
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias']

• This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  Model loaded from C:\Users\dell.cache\huggingface\hub\models--ShilongLiu--GroundingDINO\snapshots\a94c9b567a2a374598f05c584e96798a170c56fb\groundingdino_swinb_cogcoor.pth
  => _IncompatibleKeys(missing_keys=[], unexpected_keys=['label_enc.weight'])
  (viser) No client build found. Building now...
  C:\ProgramData\Miniconda3\envs\GaussianEditor\lib\site-packages\nodeenv.py:48: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources import parse_version

• Environment already exists: D:\01_DL\GaussianEditor\extern\viser\src\viser\client.nodeenv
  Traceback (most recent call last):
  File "D:\01_DL\GaussianEditor\webui.py", line 1540, in
  webui = WebUI(args)
  File "D:\01_DL\GaussianEditor\webui.py", line 156, in init
  self.server = viser.ViserServer(port=self.port)
  File "D:\01_DL\GaussianEditor\extern\viser\src\viser_viser.py", line 326, in init
  _client_autobuild.ensure_client_is_built()
  File "D:\01_DL\GaussianEditor\extern\viser\src\viser_client_autobuild.py", line 57, in ensure_client_is_built
  env_dir = _install_sandboxed_node()
  File "D:\01_DL\GaussianEditor\extern\viser\src\viser_client_autobuild.py", line 81, in _install_sandboxed_node
  subprocess.run(
  File "C:\ProgramData\Miniconda3\envs\GaussianEditor\lib\subprocess.py", line 503, in run
  with Popen(*popenargs, **kwargs) as process:
  File "C:\ProgramData\Miniconda3\envs\GaussianEditor\lib\subprocess.py", line 971, in init
  self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\ProgramData\Miniconda3\envs\GaussianEditor\lib\subprocess.py", line 1456, in _execute_child
  hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
  FileNotFoundError: [WinError 2] 系统找不到指定的文件。`

System: Windows 10, Cuda 11.7, torch 2.0.1, Rtx 3090, Python 3.10.12

I'm encountering this issue when using "Add" mode following the instruct. Any idea of fixing this?

I'm getting an error when trying to edit a scene:
Error:

  File "webui.py", line 484, in _
    self.edit(edit_cameras, train_frames, train_frustums)
  File "webui.py", line 1124, in edit
    self.guidance = EditGuidance(
  File "/home/jonathan/GaussianEditor/GUI/EditGuidance.py", line 34, in __init__
    self.prompt_utils = StableDiffusionPromptProcessor(
  File "/home/jonathan/GaussianEditor/threestudio/utils/base.py", line 83, in __init__
    self.configure(*args, **kwargs)
  File "/home/jonathan/GaussianEditor/threestudio/models/prompt_processors/base.py", line 297, in configure
    with open(os.path.join("load/prompt_library.json"), "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'load/prompt_library.json'

The UI:

when i use webui.py
I got error,

npm ERR! code ERR_INVALID_URL
npm ERR! Invalid URL
npm ERR! A complete log of this run can be found in: /home/pengxu/.npm/_logs/2024-03-20T07_23_31_215Z-debug-0.log
npm ERR! code ERR_INVALID_URL
npm ERR! Invalid URL
npm ERR! A complete log of this run can be found in: /home/pengxu/.npm/_logs/2024-03-20T07_23_31_676Z-debug-0.log

and in browser, it display 404 in the 8084 port

I don't know how to resolve it, could you help me?

When I try to add it, this issue pops up
File "webui.py", line 587, in _
self.click_cb(pointer)
File "webui.py", line 841, in click_cb
assert hasattr(pointer, "click_pos"), "please install our forked viser"
AssertionError: please install our forked viser

I'm still a bit confused about the paper: does semantic tag annotation happen during the editing process?
I see that semantic tracking is implemented in update_mask(), but I don't see Gaussian Semantic Tracing during the local edit optimization process in function edit() or in class InstructPix2PixGuidance()\EditGuidance()

Hi, i met an input/output problem when setting the environment using conda, terminal info:

  creating build/bdist.linux-x86_64/wheel/simple_knn
  copying build/lib.linux-x86_64-cpython-38/simple_knn/_C.cpython-38-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/wheel/simple_knn
  running install_egg_info
  running egg_info
  writing simple_knn.egg-info/PKG-INFO
  writing dependency_links to simple_knn.egg-info/dependency_links.txt
  writing top-level names to simple_knn.egg-info/top_level.txt
  reading manifest file 'simple_knn.egg-info/SOURCES.txt'
  writing manifest file 'simple_knn.egg-info/SOURCES.txt'
  Copying simple_knn.egg-info to build/bdist.linux-x86_64/wheel/simple_knn-0.0.0-py3.8.egg-info
  **error: [Errno 5] Input/output error: 'build/bdist.linux-x86_64/wheel/simple_knn-0.0.0-py3.8.egg-info/SOURCES.txt'**
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for simple_knn
Running setup.py clean for simple_knn
Failed to build diff_gaussian_rasterization simple_knn
ERROR: Could not build wheels for diff_gaussian_rasterization, simple_knn, which is required to install pyproject.toml-based projects

OpenGL.error.GLError: GLError(
err = 12297,
baseOperation = eglCreateContext,
cArguments = (
<OpenGL._opaque.EGLDisplay_pointer object at 0x7f61981e5940>,
<OpenGL._opaque.EGLConfig_pointer object at 0x7f61981e59c0>,
<OpenGL._opaque.EGLContext_pointer object at 0x7f61f186df40>,
<OpenGL.arrays.lists.c_int_Array_7 object at 0x7f61981e5ac0>,
),
result = <OpenGL._opaque.EGLContext_pointer object at 0x7f61981e5bc0>
)

Hello, Dr. Chen.I have a problem.I tried to run the webui, and encountered this problem: It's said that "No module named 'simple_knn'", but when I tried to install it with pip, it's said that "Requirement already satisfied".Thank you for your help!

windows10 CUDA 11.7 RTX 8000 VS2019
<1>. Installation of basic environment (基础环境安装)
1.1 Create a virtual environment and install torch 创建虚拟环境、安装 torch）
conda create -n GaussianEditor python=3.10
conda activate GaussianEditor
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia

1.2 pip install xformers==0.0.20

1.3 remove git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch in requirements_2.txt and pip install -r requirements_2.txt

1.4 Source code compilation 源码编译 tiny-cuda-nn
git clone --recursive https://github.com/nvlabs/tiny-cuda-nn
open x64 Native Tools Command Prompt for VS 2019
conda activate GaussianEditor
cd tiny-cuda-nn\bindings\torch
then run
python setup.py install

1.5 Source code compilation 源码编译 GroundingDINO:
git clone https://github.com/IDEA-Research/GroundingDINO.git
conda activate GaussianEditor
cd GroundingDINO
pip install -e .

1.6 Source code compilation 源码编译 lang-segment-anything
git clone https://github.com/luca-medeiros/lang-segment-anything && cd lang-segment-anything
去掉 pyproject.toml 中对于 groundingdino 依赖

#groundingdino = {git = "https://github.com/IDEA-Research/GroundingDINO.git"} line 26 remove
pip install -e .
1.7 install simple-knn diff-gaussian-rasterization
cd gaussiansplatting/submodules
pip install ./diff-gaussian-rasterization
pip install ./simple-knn
pip install kornia==0.7.0

<2>. Installation of Viser environment (Viser环境安装)

mkdir extern && cd extern
git clone https://github.com/heheyas/viser
pip install -e viser
Modified Viser source code for windows builds（接下来修改Viser 源代码用于windows构建）
path\to\you\extern\viser\src\viser_viser.py line 326

2.1 install node env: path\to\you\extern\viser\src\viser_client_autobuild.py


when i run ,i find this error(运行时发现这个错误)：

Install cmd manually under administrator rights （管理员权限手动安装）

cd path\to\your\GaussianEditor\extern\viser\src\viser\client
conda activate GaussianEditor
python -m nodeenv --node=20.4.0 .nodeenv

2.2 build yarn file (构建yarn文件夹)

2.3 Source code modification(需要修改源码的地方)
path\to\you\GaussianEditor\extern\viser\src\viser_client_autobuild.py replace this script

_client_autobuild.txt

3. run webui.py, It's OK

Hello all,

I am running a win 11 pro computer with rtx 4090, using conda cmd prompt.

I have tried all 3 options for installation. Any help to resolve below issues greatly appreciated.

Step 1, option 2 pip install gets me the closest via # CUDA version 11.8 -

(base) C:\Users\Dirk\GaussianEditor>pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
NO errors.

(base) C:\Users\Dirk\GaussianEditor>pip install -r requirements.txt
Errors below-

ERROR: Exception:
Traceback (most recent call last):
File "C:\Users\Dirk\anaconda3\Lib\site-packages\pip_internal\cli\base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
^^^^^^^^^^^^^^^
File "C:\Users\Dirk\anaconda3\Lib\site-packages\pip_internal\cli\req_command.py", line 248, in wrapper
return func(self, options, args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Dirk\anaconda3\Lib\site-packages\pip_internal\commands\install.py", line 377, in run
requirement_set = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "C:\Users\Dirk\anaconda3\Lib\site-packages\pip_internal\resolution\resolvelib\resolver.py", line 92, in resolve
result = self._result = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "C:\Users\Dirk\anaconda3\Lib\site-packages\pip_vendor\resolvelib\resolvers.py", line 546, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Dirk\anaconda3\Lib\site-packages\pip_vendor\resolvelib\resolvers.py", line 457, in resolve
raise ResolutionTooDeep(max_rounds)
pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000

even if each job is masked by CUDA_VISIBLE_DEVICES

I see that I have to download omnidata_dpt_depth_v2.ckpt in GaussianEditor-master/threestudio/utils/dpt.py
Because of the server network, I had to load it locally and could not download it online，could you tell me where to download "omnidata_dpt_depth_v2.ckpt"?Is it same as https://github.com/ashawkey/stable-dreamfusion/blob/5550b91862a3af7842bb04875b7f1211e5095a63/readme.md?plain=1#L68 ？

         if self.mode == "depth":
            path = ".cache/dpt/omnidata_dpt_depth_v2.ckpt"
            self.model = DPTDepthModel(backbone="vitb_rn50_384")
            self.aug = transforms.Compose(
                [
                    transforms.Resize((384, 384)),
                    transforms.Normalize(mean=0.5, std=0.5),
                ]
            )
        elif self.mode == "normal":
            path = ".cache/dpt/omnidata_dpt_normal_v2.ckpt"
            self.model = DPTDepthModel(backbone="vitb_rn50_384", num_channels=3)
            self.aug = transforms.Compose(
                [
                    transforms.Resize((384, 384)),
                ]
            )

ERROR: Project file:///app/extern/viser has a 'pyproject.toml' and its build backend is missing the 'build_editable' hook. Since it does not have a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable mode. Consider using a build backend that supports PEP 660.

I tried this example of deleting bicycle given in the paper.

The paper said: when deleting the bike directly, there are still Gaussian points around the bike which are not deleted, resulting in the creation of artifacts, so 2D inpainting optimization is needed.

Then after the optimization of deleting the bike, just like below, the optimized scene is not ideal due to the bad 2D inpainting result. (the text I used for inpainting model is "bench",which is the background description).Is this due to the fact that the results in your paper were only optimized in fewer steps?The parameter of my optimization step is the program default value.

Same as title

I hope to add the function of specifying GPU to the source code，Thanks

my cmd args: --colmap_dir
./dataset/bicycle
--gs_source
./dataset/bicycle/point_cloud/iteration_7000/point_cloud.ply

I have tried getting the code to run on Windows and have made decent progress.
But unfortunately, I am running out of time, so instead of a full PR, here are the things I have found out:

To get this thing to run on Windows, you need to:

• hack caster.h so the C++ extensions compiles(see facebookresearch/pytorch3d#1024)
• Manually install GroundingDINO using pip(pip install git+https://github.com/IDEA-Research/GroundingDINO.git) before pip install -r requirements
• find a way to get the forked viser to build its client on Windows

good luck!

Hi

is it possible to export a new 3D Gaussian Splating scene (.ply) after object removal and integration?

Sorry for my poor knowledge about Machine Learning and 3dGS... When running this project (with test data "bicycle.ply")i always encounter CUDA OUTOFMEMORY ERROR. How should I do or set if it is possible to run with a 16G GPU ? (my cuda version is 11.8)

Hi, using add_dog.sh, get this error:

No such file or directory: '/root/szd/GaussianEditor/add_cache/add_dog/inpaint_a_dog_on_the_bench/multiview_pred_images/rgb_000_front.png'

any idea how to fix this?

when i run <pip install -r requirements.txt>,always some plugins can not be downloaded.

I tried webui in a docker （Window & WSL ） and got error as below

root@e26714615908:/app/github/GaussianEditor# nvidia-smi
Tue Nov 28 12:03:07 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.103                Driver Version: 537.13       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3090        On  | 00000000:01:00.0  On |                  N/A |
|  0%   47C    P8              36W / 350W |    971MiB / 24576MiB |      9%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        20      G   /Xwayland                                 N/A      |
|    0   N/A  N/A        20      G   /Xwayland                                 N/A      |
|    0   N/A  N/A        27      G   /Xwayland                                 N/A      |
+---------------------------------------------------------------------------------------+
root@e26714615908:/app/github/GaussianEditor# python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'2.0.1+cu118'
>>> import torchvision
>>> torchvision.__version__
'0.15.2+cu118'
>>> exit()
root@e26714615908:/app/github/GaussianEditor# python3 webui.py \
    --colmap_dir /app/github/dataset/models/bicycle \
    --gs_source /app/github/dataset/models/bicycle/point_cloud/iteration_7000/point_cloud.ply
/usr/local/lib/python3.10/dist-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_5m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_5m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/usr/local/lib/python3.10/dist-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_11m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_11m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/usr/local/lib/python3.10/dist-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/usr/local/lib/python3.10/dist-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_384 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_384. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/usr/local/lib/python3.10/dist-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_512 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_512. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/usr/local/lib/python3.10/dist-packages/groundingdino/models/GroundingDINO/ms_deform_attn.py:31: UserWarning: Failed to load custom C++ ops. Running on CPU mode Only!
  warnings.warn("Failed to load custom C++ ops. Running on CPU mode Only!")
Traceback (most recent call last):
  File "/app/github/GaussianEditor/webui.py", line 1537, in <module>
    webui = WebUI(args)
  File "/app/github/GaussianEditor/webui.py", line 127, in __init__
    scene = CamScene(self.colmap_dir, h=512, w=512)
  File "/app/github/GaussianEditor/gaussiansplatting/scene/camera_scene.py", line 39, in __init__
    assert False, "Could not recognize scene type!"
AssertionError: Could not recognize scene type!
root@e26714615908:/app/github/GaussianEditor# 

pip list :

root@e26714615908:/app/github/GaussianEditor# pip list  
Package                     Version      Editable project location
--------------------------- ------------ -----------------------------------------------------------------------------------
absl-py                     2.0.0
accelerate                  0.24.1
addict                      2.4.0
aiofiles                    23.2.1
aiohttp                     3.9.1
aiosignal                   1.3.1
albumentations              0.5.2
altair                      5.2.0
annotated-types             0.6.0
antlr4-python3-runtime      4.9.3
anyio                       3.7.1
appdirs                     1.4.4
asttokens                   2.4.1
async-timeout               4.0.3
attrs                       23.1.0
beautifulsoup4              4.12.2
bitsandbytes                0.38.1
braceexpand                 0.1.7
cachetools                  5.3.2
certifi                     2023.11.17
cffi                        1.16.0
chardet                     5.2.0
charset-normalizer          3.3.2
click                       8.1.7
clip                        1.0
cmake                       3.27.7
coloredlogs                 15.0.1
colorlog                    6.7.0
contourpy                   1.2.0
controlnet-aux              0.0.7
cycler                      0.12.1
decorator                   5.1.1
diff-gaussian-rasterization 0.0.0        /app/github/GaussianEditor/gaussiansplatting/submodules/diff-gaussian-rasterization
diffusers                   0.23.1
docker-pycreds              0.4.0
docstring-parser            0.15
easydict                    1.11
einops                      0.7.0
embreex                     2.17.7.post3
envlight                    0.1.0
exceptiongroup              1.2.0
executing                   2.0.1
fastapi                     0.104.1
ffmpy                       0.3.1
filelock                    3.13.1
flatbuffers                 23.5.26
fonttools                   4.45.1
frozenlist                  1.4.0
fsspec                      2023.10.0
ftfy                        6.1.3
gdown                       4.7.1
gitdb                       4.0.11
GitPython                   3.1.40
google-auth                 2.23.4
google-auth-oauthlib        1.1.0
gradio                      3.50.2
gradio_client               0.6.1
groundingdino               0.1.0
grpcio                      1.59.3
h11                         0.14.0
httpcore                    1.0.2
httpx                       0.25.2
huggingface-hub             0.16.4
humanfriendly               10.0
idna                        3.6
imageio                     2.33.0
imageio-ffmpeg              0.4.9
imgaug                      0.4.0
importlib-metadata          6.8.0
importlib-resources         6.1.1
ipython                     8.18.1
jaxtyping                   0.2.24
jedi                        0.19.1
Jinja2                      3.1.2
joblib                      1.3.2
jsonschema                  4.20.0
jsonschema-specifications   2023.11.1
kiui                        0.1.10
kiwisolver                  1.4.5
kornia                      0.7.0
lang-sam                    0.1.0
lazy_loader                 0.3
libigl                      2.5.0
lightning                   2.1.2
lightning-utilities         0.10.0
lit                         17.0.5
llvmlite                    0.41.1
lxml                        4.9.3
mapbox-earcut               1.0.1
Markdown                    3.5.1
markdown-it-py              3.0.0
MarkupSafe                  2.1.3
matplotlib                  3.8.2
matplotlib-inline           0.1.6
mdurl                       0.1.2
mediapipe                   0.10.8
mediapy                     1.1.9
mpmath                      1.3.0
msgpack                     1.0.7
multidict                   6.0.4
networkx                    3.2.1
ninja                       1.11.1.1
nodeenv                     1.8.0
numba                       0.58.1
numpy                       1.26.2
nvdiffrast                  0.3.1
oauthlib                    3.2.2
objprint                    0.2.3
omegaconf                   2.3.0
onnxruntime                 1.16.3
opencv-contrib-python       4.8.1.78
opencv-python               4.8.1.78
opencv-python-headless      4.8.1.78
orjson                      3.9.10
packaging                   23.2
pandas                      2.1.3
parso                       0.8.3
pexpect                     4.9.0
Pillow                      9.3.0
pip                         23.3.1
platformdirs                4.0.0
plyfile                     1.0.2
pooch                       1.8.0
prompt-toolkit              3.0.41
protobuf                    3.20.3
psutil                      5.9.6
ptyprocess                  0.7.0
pure-eval                   0.2.2
pyasn1                      0.5.1
pyasn1-modules              0.3.0
pycocotools                 2.0.7
pycollada                   0.7.2
pycparser                   2.21
pydantic                    2.5.2
pydantic_core               2.14.5
pydub                       0.25.1
Pygments                    2.17.2
pyliblzfse                  0.4.1
PyMatting                   1.1.11
PyMCubes                    0.1.4
pyparsing                   3.1.1
pysdf                       0.1.9
PySocks                     1.7.1
python-dateutil             2.8.2
python-multipart            0.0.6
pytorch-lightning           2.1.2
pytz                        2023.3.post1
PyYAML                      6.0.1
referencing                 0.31.0
regex                       2023.10.3
rembg                       2.0.52
requests                    2.31.0
requests-oauthlib           1.3.1
rich                        13.7.0
rpds-py                     0.13.1
rsa                         4.9
Rtree                       1.1.0
safetensors                 0.4.1
scikit-image                0.22.0
scikit-learn                1.3.2
scipy                       1.11.4
segment-anything            1.0
semantic-version            2.10.0
sentencepiece               0.1.99
sentry-sdk                  1.37.1
setproctitle                1.3.3
setuptools                  59.6.0
shapely                     2.0.2
shtab                       1.6.4
simple-knn                  0.0.0        /app/github/GaussianEditor/gaussiansplatting/submodules/simple-knn
six                         1.16.0
smmap                       5.0.1
sniffio                     1.3.0
sounddevice                 0.4.6
soupsieve                   2.5
stack-data                  0.6.3
starlette                   0.27.0
supervision                 0.6.0
svg.path                    6.3
sympy                       1.12
taming-transformers-rom1504 0.0.6
tensorboard                 2.15.1
tensorboard-data-server     0.7.2
threadpoolctl               3.2.0
tifffile                    2023.9.26
timm                        0.9.12
tinycudann                  1.7
tokenizers                  0.13.3
tomli                       2.0.1
toolz                       0.12.0
torch                       2.0.1+cu118
torch-efficient-distloss    0.1.3
torchmetrics                1.2.0
torchvision                 0.15.2+cu118
tqdm                        4.66.1
traitlets                   5.14.0
transformers                4.28.1
trimesh                     4.0.5
triton                      2.0.0
typeguard                   2.13.3
typing_extensions           4.8.0
tyro                        0.5.18
tzdata                      2023.3
urllib3                     2.1.0
uvicorn                     0.24.0.post1
varname                     0.12.1
viser                       0.1.10       /app/extern/viser
wandb                       0.16.0
wcwidth                     0.2.12
webdataset                  0.2.79
websockets                  11.0.3
Werkzeug                    3.0.1
wheel                       0.37.1
xatlas                      0.0.8
xxhash                      3.4.1
yapf                        0.40.2
yarl                        1.9.3
yourdfpy                    0.0.53
zipp                        3.17.0

(base) root@autodl-container-f31b4fbcca-d56efb7e:~# python autodl-tmp/GaussianEditor-master/webui.py\

--colmap_dir autodl-tmp/GaussianEditor-master/dataset/bicycle \
--gs_source autodl-tmp/GaussianEditor-master/dataset/bicycle/point_cloud/iteration_7000/point_cloud.ply

/root/miniconda3/lib/python3.8/site-packages/controlnet_aux/mediapipe_face/mediapipe_face_common.py:7: UserWarning: The module 'mediapipe' is not installed. The package will have limited functionality. Please install it using the command: pip install 'mediapipe'
warnings.warn(
/root/miniconda3/lib/python3.8/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_5m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_5m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
/root/miniconda3/lib/python3.8/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_11m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_11m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
/root/miniconda3/lib/python3.8/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
/root/miniconda3/lib/python3.8/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_384 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_384. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
/root/miniconda3/lib/python3.8/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_512 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_512. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
Traceback (most recent call last):
File "autodl-tmp/GaussianEditor-master/webui.py", line 1569, in
webui = WebUI(args)
File "autodl-tmp/GaussianEditor-master/webui.py", line 129, in init
scene = CamScene(self.colmap_dir, h=512, w=512)
File "/root/autodl-tmp/GaussianEditor-master/gaussiansplatting/scene/camera_scene.py", line 39, in init
assert False, "Could not recognize scene type!"
AssertionError: Could not recognize scene type!

When using the add option, skip the steps of text to 2D image and then to wonder3D to generate 3Dmesh, and directly import an existing model at the specified location. Is this possible and how to modify it? Looking forward to your reply.

Thanks for your inspiring work. I found you provide the .ply file for the scene "face" in instruct-N2N data. Do you know how to generate its corresponding "sparse" folder of COLMAP required by the code?

I have successfully installed viser by following the instructions below:
mkdir extern && cd extern
git clone https://github.com/heheyas/viser
pip install -e viser
cd ..
but when I run "python webui.py --gs_source --colmap_dir " it occur some questions as follow:

i also attempt upgrade pip. but it also didn't work. Can you give me some advice? Thank you!
my environment is:
torch==2.0.1+cu117
torchvision==0.15.2+cu117
pip=24.0

First of all thank you for your great work!!
I want to run this project on Window environment. Is it working???
Now I am installing requirements.txt, there are many error...

Thanks for your excellent work!!
The idea of HGS is very genius. As for the previous work GSGEN, DreamGaussian and GaussianDreamer mentioned in the paper, "GS has shown limitations when facing the randomness of generative guidance."
I would like to know what this challenge is specifically, because I don't remember any mention of this in these papers. Does it mean that their results are not so good, so speculate on the reason?

When I tried to add object, I always meet the same error. Is there any solution?
Downloading: "https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-vitj
x/jx_vit_base_resnet50_384-9fd3c705.pth" to /mnt/data/lyx23/.cache/torch/hub/checkpoints/jx
_vit_base_resnet50_384-9fd3c705.pth
Traceback (most recent call last):
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/urllib/request.py", line 1354, in
do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/http/client.py", line 1256, in req
uest
self._send_request(method, url, body, headers, encode_chunked)
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/http/client.py", line 1302, in _se
nd_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/http/client.py", line 1251, in end
headers
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/http/client.py", line 1011, in _se
nd_output
self.send(msg)
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/http/client.py", line 951, in send
self.connect()
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/http/client.py", line 1418, in con
nect
super().connect()
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/http/client.py", line 922, in conn
ect
self.sock = self._create_connection(
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/socket.py", line 808, in create_co
nnection
raise err
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/socket.py", line 796, in create_co
nnection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/lyx23/Experiments/control_GaussianEditor/extern/viser/src/viser/infra/_infra.
py", line 449, in wrapped
inner()
File "/home/lyx23/Experiments/control_GaussianEditor/extern/viser/src/viser/infra/_infra.
py", line 257, in
lambda: self._handle_incoming_message(client_id, message)
File "/home/lyx23/Experiments/control_GaussianEditor/extern/viser/src/viser/infra/_infra.
py", line 94, in _handle_incoming_message
cb(client_id, message)
File "/home/lyx23/Experiments/control_GaussianEditor/extern/viser/src/viser/_gui_api.py",
line 157, in _handle_gui_updates
cb(GuiEvent(client, client_id, handle))
File "webui.py", line 511, in _
self.add(self.camera)
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/site-packages/torch/utils/_context
lib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "webui.py", line 1371, in add
depth_estimator = DPT(get_device(), mode="depth")
File "/home/lyx23/Experiments/yyy23/GaussianEditor/threestudio/utils/dpt.py", line 1005,
in init
self.model = DPTDepthModel(backbone="vitb_rn50_384")
File "/home/lyx23/Experiments/yyy23/GaussianEditor/threestudio/utils/dpt.py", line 989, i
n init
super().init(head, **kwargs)
File "/home/lyx23/Experiments/yyy23/GaussianEditor/threestudio/utils/dpt.py", line 936, i
n init
self.pretrained, self.scratch = _make_encoder(
File "/home/lyx23/Experiments/yyy23/GaussianEditor/threestudio/utils/dpt.py", line 540, i
n _make_encoder
pretrained = _make_pretrained_vitb_rn50_384(
File "/home/lyx23/Experiments/yyy23/GaussianEditor/threestudio/utils/dpt.py", line 508, i
n _make_pretrained_vitb_rn50_384
model = timm.create_model("vit_base_resnet50_384", pretrained=pretrained)
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/site-packages/timm/models/factory.
py", line 71, in create_model
model = create_fn(pretrained=pretrained, pretrained_cfg=pretrained_cfg, **kwargs)
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/site-packages/timm/models/vision_t
ransformer_hybrid.py", line 254, in vit_base_resnet50_384
return vit_base_r50_s16_384(pretrained=pretrained, **kwargs)
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/site-packages/timm/models/vision_t
ransformer_hybrid.py", line 246, in vit_base_r50_s16_384
model = _create_vision_transformer_hybrid(
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/site-packages/timm/models/vision_t
ransformer_hybrid.py", line 155, in _create_vision_transformer_hybrid
return _create_vision_transformer(variant, pretrained=pretrained, embed_layer=embed_lay
er, **kwargs)
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/site-packages/timm/models/vision_t
ransformer.py", line 786, in _create_vision_transformer
model = build_model_with_cfg(
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/site-packages/timm/models/helpers.
py", line 551, in build_model_with_cfg
load_pretrained(
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/site-packages/timm/models/helpers.
py", line 248, in load_pretrained
state_dict = load_state_dict_from_url(
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/site-packages/torch/hub.py", line
746, in load_state_dict_from_url
download_url_to_file(url, cached_file, hash_prefix, progress=progress)
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/site-packages/torch/hub.py", line
611, in download_url_to_file
u = urlopen(req)
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/urllib/request.py", line 222, in u
rlopen
return opener.open(url, data, timeout)
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/urllib/request.py", line 525, in o
pen
response = self._open(req, data)
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/urllib/request.py", line 542, in _
open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/urllib/request.py", line 502, in _
call_chain
result = func(*args)
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/urllib/request.py", line 1397, in
https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/home/lyx23/anaconda3/envs/Geditor/lib/python3.8/urllib/request.py", line 1357, in
do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 110] Connection timed out>

If I want to use the rest of the data in in2n instead of the bear and face in it, how do I get the relevant .ply files?

OpenGL.error.GLError: GLError(
err = 12297,
baseOperation = eglCreateContext,
cArguments = (
<OpenGL._opaque.EGLDisplay_pointer object at 0x7f61981e5940>,
<OpenGL._opaque.EGLConfig_pointer object at 0x7f61981e59c0>,
<OpenGL._opaque.EGLContext_pointer object at 0x7f61f186df40>,
<OpenGL.arrays.lists.c_int_Array_7 object at 0x7f61981e5ac0>,
),
result = <OpenGL._opaque.EGLContext_pointer object at 0x7f61981e5bc0>
)

Hello,

Thanks for utilizing PyTorch Lightning! My name is Corey Strausman and I'm the Community Lead at Lightning.AI (from the creators of PyTorch Lightning). I'd love to connect to see if there would be any interest in building a Lightning Studio Template for GaussianEditor.

We'd love to support you in this by offering Lightning Studio credits for the development of the GaussianEditor Studio. Additionally, we'll showcase your work by publishing the Studio template and promoting it through our social media channels. Let me know if you're interested, and we can schedule a time to discuss it further. Feel free to email me at [email protected].

You can learn more about Studios here:

https://lightning.ai/studios

https://lightning.ai/docs/overview/studios

Best Regards,
Corey Strausman

Thanks for your work!
In the paper we see such a sentence，“Gaussian semantic tracing is achieved by unprojecting 2D segmentation masks into 3D Gaussians and assigning each Gaussian a semantic tag”. Does this indicate that multi-view masks are used to initialize the semantic information of each Gaussian point,Specifically, the weight of each semantic.
another question：Where is ”Gaussian tracing algorithm“ reflected in the code?

When I test the edit/add function, there are some checkpoints url downloading error when downloading checkpoints. As I see, threestudio/utils/wonder3D/configs/mvdiffusion-joint-ortho-6views.yaml define the checkpoints used in test_mvdiffusion_seq.py, such as the definition of pretrained_model_name_or_path and pretrained_unet_path. But only unet is download by the definition of pretrained_unet_path, which cause error when there is no threestudio/utils/wonder3D/ckpts direction exists. Actually, this direction realy do no exists in this repo.

Can you help me to access the unet pretrain model? Use cfg.pretrained_model_name_or_path not cfg.pretrained_unet_path during UNetMV2DConditionModel.from_pretrained_2d, or others?

The definition of mvdiffusion-joint-ortho-6views.yaml:

pretrained_model_name_or_path: 'lambdalabs/sd-image-variations-diffusers'
pretrained_unet_path: './ckpts/'

The pretrain ckpt download in

    image_encoder = CLIPVisionModelWithProjection.from_pretrained(cfg.pretrained_model_name_or_path, subfolder="image_encoder", revision=cfg.revision)
    feature_extractor = CLIPImageProcessor.from_pretrained(cfg.pretrained_model_name_or_path, subfolder="feature_extractor", revision=cfg.revision)
    vae = AutoencoderKL.from_pretrained(cfg.pretrained_model_name_or_path, subfolder="vae", revision=cfg.revision)
    unet = UNetMV2DConditionModel.from_pretrained_2d(cfg.pretrained_unet_path, subfolder="unet", revision=cfg.revision, **cfg.unet_from_pretrained_kwargs)

I notice that author modify the origin viser to capture the scene click pixel (actually, it is normal pixel). Here is an implemention via the origin viser, which can be used in click_cb function:

        camera = list(self.server.get_clients().values())[0].camera
        W = int(self.resolution_slider.value)
        H = int(W / self.aspect)
        WH = torch.Tensor([W, H])
        fxy = WH/2 / np.tan(camera.fov/2)
        # TODO: fxy from different fovx, fovy?
        dxyz = tf.SO3(camera.wxyz).inverse().as_matrix() @ pointer.ray_direction
        uv = fxy * torch.Tensor(dxyz[:2])/dxyz[2] + WH/2
        click_pos = uv / WH