ku-cvlab / 3dfuse Goto Github PK

it fails just says error

File not found error when generate 3D:

File "/opt/anaconda3/envs/3DFuse/lib/python3.8/site-packages/torch/serialization.py", line 1125, in load
with _open_file_like(f, "rb") as opened_file:
File "/opt/anaconda3/envs/3DFuse/lib/python3.8/site-packages/torch/serialization.py", line 543, in _open_file_like
return _open_file(name_or_buffer, mode)
File "/opt/anaconda3/envs/3DFuse/lib/python3.8/site-packages/torch/serialization.py", line 524, in init
super().init(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: './results/bed_0007/lora/merge.tmp/unet/diffusion_pytorch_model.bin'

Hi there! I'm trying to run the Huggingface demo in a duplicated space but running into an issue 'Something went wrong Connection errored out' - from the logs I don't see any errors, when I check the container log all I see is (Additionally maybe this is just a spaces thing but it seems that it re-downloads a bunch of models every time I run the demo possibly because the connection erroring out is causing some kind of reset? );

...
Collapsing Lin Lora in out_proj
Reshaping encoder.mid.attn_1.q.weight for SD format
Reshaping encoder.mid.attn_1.k.weight for SD format
Reshaping encoder.mid.attn_1.v.weight for SD format
Reshaping encoder.mid.attn_1.proj_out.weight for SD format
Reshaping decoder.mid.attn_1.q.weight for SD format
Reshaping decoder.mid.attn_1.k.weight for SD format
Reshaping decoder.mid.attn_1.v.weight for SD format
Reshaping decoder.mid.attn_1.proj_out.weight for SD format
(At this point Huggingface's log clears)
Looking in links: https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu117_pyt1131/download.html
Collecting pytorch3d
  Downloading https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu117_pyt1131/pytorch3d-0.7.3-cp38-cp38-linux_x86_64.whl (72.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 72.7/72.7 MB 131.5 MB/s eta 0:00:00
Requirement already satisfied: iopath in /home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages (from pytorch3d) (0.1.10)
Requirement already satisfied: fvcore in /home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages (from pytorch3d) (0.1.5.post20221221)
Requirement already satisfied: Pillow in /home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages (from fvcore->pytorch3d) (9.0.1)
Requirement already satisfied: pyyaml>=5.1 in /home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages (from fvcore->pytorch3d) (6.0)
Requirement already satisfied: tqdm in /home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages (from fvcore->pytorch3d) (4.64.1)
Requirement already satisfied: tabulate in /home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages (from fvcore->pytorch3d) (0.9.0)
Requirement already satisfied: yacs>=0.1.6 in /home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages (from fvcore->pytorch3d) (0.1.8)
Requirement already satisfied: termcolor>=1.1 in /home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages (from fvcore->pytorch3d) (2.3.0)
Requirement already satisfied: numpy in /home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages (from fvcore->pytorch3d) (1.22.4)
Requirement already satisfied: portalocker in /home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages (from iopath->pytorch3d) (2.7.0)
Requirement already satisfied: typing-extensions in /home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages (from iopath->pytorch3d) (4.5.0)
Installing collected packages: pytorch3d
Successfully installed pytorch3d-0.7.3
./weights/3DFuse_sparse_depth_injector.ckpt already exists.
/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/deprecation.py:43: UserWarning: You have unused kwarg parameters in Row, please remove them: {'scale': 1.0}
  warnings.warn(
Running on local URL:  http://0.0.0.0:7860

To create a public link, set `share=True` in `launch()`.

Hello, I would like to ask where the supplementary material mentioned in the following text are, because I did not find them on the project website. Moreover, could you please release the code for implementing variance metric. Thank you very sincerely.

Is it possible to use any other custom SD-based model with this repo and get a decent result? Or only the ones that are presented in the Gradio demo work properly?

When I run the run.sh file, it gives me an error. how to solve the problem?
TypeError: forward() got an unexpected keyword argument 'scale'

hello mates , i have been trying to install pytorch3d but i wasnt able to successfuly install it i used all the methods provided in their installation guide but non of them worked i get this error at the end of running the setup.py for pytorch3d

` C:\Users\Genesis\AppData\Local\Temp\tmpxft_00002460_00000000-7_ball_query.compute_75.cudafe1.cpp : fatal error C1083: Cannot open compiler generated file: '': Invalid argument
error: command 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin\nvcc.exe' failed with exit code 1
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> pytorch3d

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.`

Thank you for providing such a great job. May I ask if it is possible to save the 3D model to result? For example, obj, what should I do?

Running local on Win10 I'm getting the following error when generating 3D after creating the point cloud. This happens at some point during the lora process. I did initially have some problems getting PyTorch3D set up, and I did have to create the conda environment in 3.9, this repo suggests 3.8, in case that's a problem here. I see that ultimately it fails to find this particular image, as it doesn't exist.

...
Collapsing Lin Lora in out_proj
Collapsing Lin Lora in k_proj
Collapsing Lin Lora in v_proj
Collapsing Lin Lora in q_proj
Collapsing Lin Lora in out_proj
Reshaping encoder.mid.attn_1.q.weight for SD format
Reshaping encoder.mid.attn_1.k.weight for SD format
Reshaping encoder.mid.attn_1.v.weight for SD format
Reshaping encoder.mid.attn_1.proj_out.weight for SD format
Reshaping decoder.mid.attn_1.q.weight for SD format
Reshaping decoder.mid.attn_1.k.weight for SD format
Reshaping decoder.mid.attn_1.v.weight for SD format
Reshaping decoder.mid.attn_1.proj_out.weight for SD format
Added Token: ['<0>']
a fuzzy <0>
side view of a fuzzy <0>:   2%|█                                                   | 199/10000 [00:45<37:02,  4.41it/s]
Traceback (most recent call last):
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\gradio\routes.py", line 393, in run_predict
    output = await app.get_blocks().process_api(
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\gradio\blocks.py", line 1108, in process_api
    result = await self.call_function(
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\gradio\blocks.py", line 929, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\gradio\utils.py", line 490, in async_iteration
    return next(iterator)
  File "C:\stable-diffusion\3DFuse\gradio_app.py", line 47, in gen_3d
    yield from model.run_gradio(points,exp_instance_dir)
  File "C:\stable-diffusion\3DFuse\run_3DFuse.py", line 147, in run_gradio
    yield from fuse_3d(**cfgs, poser=poser,model=model,vox=vox,exp_instance_dir=exp_instance_dir, points=points, is_gradio=True)
  File "C:\stable-diffusion\3DFuse\run_3DFuse.py", line 248, in fuse_3d
    vis_routine(metric, y, depth,p,depth_map[0])
  File "C:\stable-diffusion\3DFuse\run_3DFuse.py", line 345, in vis_routine
    metric.put_artifact("img", ".png",prompt, lambda fn: imwrite(fn, im))
  File "C:\stable-diffusion\3DFuse\my\utils\event.py", line 110, in put_artifact
    save_func(fname)
  File "C:\stable-diffusion\3DFuse\run_3DFuse.py", line 345, in <lambda>
    metric.put_artifact("img", ".png",prompt, lambda fn: imwrite(fn, im))
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\imageio\core\functions.py", line 303, in imwrite
    writer = get_writer(uri, format, "i", **kwargs)
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\imageio\core\functions.py", line 231, in get_writer
    return format.get_writer(request)
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\imageio\core\format.py", line 185, in get_writer
    return self.Writer(self, request)
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\imageio\core\format.py", line 221, in __init__
    self._open(**self.request.kwargs.copy())
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\imageio\plugins\pillow.py", line 356, in _open
    PillowFormat.Writer._open(self)
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\imageio\plugins\pillow.py", line 195, in _open
    self._fp = self.request.get_file()
  File "C:\apps\anaconda3\envs\3DFuse\lib\site-packages\imageio\core\request.py", line 331, in get_file
    self._file = open(self.filename, "wb")
OSError: [Errno 22] Invalid argument: 'C:\\stable-diffusion\\3DFuse\\results\\cat_0000\\3d\\img\\step_199_backside_view_of_a_fuzzy_<0>.png'

1. Is ANY 3d data required for the training of sparse depth injector? I think the paper says that it doesn't, but I just want to be sure.
2. Is code for training the sparse depth injector available in your repo? If yes, then can you tell me in what directory?
3. How much time and computational power does it require to train it?
   Thanks in advance. Hope, that the questions are not getting too bothersome.

I started work on a google colab anyone got any idea how to finalize this ?
https://colab.research.google.com/drive/1Sv5Z1uVM5ZCGJdJSc67qGzEnyA3mlpeF?authuser=3#scrollTo=lzVAqpGLoESp

This line is producing an error for me:

https://github.com/KU-CVLAB/3DFuse/blob/main/gradio_app.py#L210

to fix it added

mask=mask[:,:,3]

immediately before it

First thank you for you great work.

Is there any evaluation code for using the pretrained parameters? Could you please give some advice ?

Hey guys! after a lot of effort and basically wasting the physical space on my hd, I managed to install the requirements to run gradio.

Now I face the problem of physical space on my hd because I'm at the limit.
During the generation of points from the cloud, I'm downloading 20gb+ and that way I can't operate in c:/. :(

Would it be possible to use another unit, such as d:/ to download the files needed to create the points?

Sorry if the question is silly, it's my first contact with AI and PY.

And if you can let me know. how many gb on average do you need?

And last question: How can I clear the downloads made in the generation of cloud points? Now I find myself with only 100mb lol

please see issue 16
#16

I have tried three demos，run.sh was set as follows，
PROMPT="a cute little kitten" \ "a cute pig with a pink snout and curly tail" \ "cute small corgi"
INITIAL="kitten" \ "pig" \ "corgi"
EXP_DIR="./results"
RANDOM_SEED=0
SEMANTIC_MODEL="SD"

All of them have janus problem, I use runwayml/stable-diffusion-v1-5.why did this problem happen？Is the problem due to the use of SD rather than karlo, or is it due to incorrect 3D point cloud generation？

I'm running the code on a Google colab with a graphics card T4 (15GB), but it produces the following output, the program stops and no error is reported.In the last line, what is the reason for "killed" and how can I fix it?：

Collapsing Lin Lora in 0
Collapsing Lin Lora in proj
Collapsing Lin Lora in k_proj
Collapsing Lin Lora in v_proj
Collapsing Lin Lora in q_proj
Collapsing Lin Lora in out_proj
Collapsing Lin Lora in k_proj
Collapsing Lin Lora in v_proj
Collapsing Lin Lora in q_proj
Collapsing Lin Lora in out_proj
Collapsing Lin Lora in k_proj
Collapsing Lin Lora in v_proj
Collapsing Lin Lora in q_proj
Collapsing Lin Lora in out_proj
Collapsing Lin Lora in k_proj
Collapsing Lin Lora in v_proj
Collapsing Lin Lora in q_proj
Collapsing Lin Lora in out_proj
Collapsing Lin Lora in k_proj
Collapsing Lin Lora in v_proj
Collapsing Lin Lora in q_proj
Collapsing Lin Lora in out_proj
Collapsing Lin Lora in k_proj
Collapsing Lin Lora in v_proj
Collapsing Lin Lora in q_proj
Collapsing Lin Lora in out_proj
Collapsing Lin Lora in k_proj
Collapsing Lin Lora in v_proj
Collapsing Lin Lora in q_proj
Collapsing Lin Lora in out_proj
Collapsing Lin Lora in k_proj
Collapsing Lin Lora in v_proj
Collapsing Lin Lora in q_proj
Collapsing Lin Lora in out_proj
Collapsing Lin Lora in k_proj
Collapsing Lin Lora in v_proj
Collapsing Lin Lora in q_proj
Collapsing Lin Lora in out_proj
Collapsing Lin Lora in k_proj
Collapsing Lin Lora in v_proj
Collapsing Lin Lora in q_proj
Collapsing Lin Lora in out_proj
Collapsing Lin Lora in k_proj
Collapsing Lin Lora in v_proj
Collapsing Lin Lora in q_proj
Collapsing Lin Lora in out_proj
Collapsing Lin Lora in k_proj
Collapsing Lin Lora in v_proj
Collapsing Lin Lora in q_proj
Collapsing Lin Lora in out_proj
Killed

Hi, I was trying to run your model using both the gradio demo and also using sh run.sh. In both these steps, the code cannot find the path of the unet model in the LoRA finetuning step.

Here is the stack trace -

Collapsing Lin Lora in out_proj
Traceback (most recent call last):
  File "run_3DFuse.py", line 352, in <module>
    dispatch(SJC_3DFuse)
  File "/home/aditya/Research/phd/code/3DFuse/my/config.py", line 91, in dispatch
    mod.run()
  File "run_3DFuse.py", line 111, in run
    model = getattr(self, family).make()
  File "/home/aditya/Research/phd/code/3DFuse/run_img_sampling.py", line 18, in make
    model = StableDiffusion(**args)
  File "/home/aditya/Research/phd/code/3DFuse/adapt_sd.py", line 135, in __init__
    model=load_3DFuse(self.checkpoint_root(),dir,alpha)
  File "/home/aditya/Research/phd/code/3DFuse/adapt_sd.py", line 122, in load_3DFuse
    state_dict, l = merge("runwayml/stable-diffusion-v1-5",dir,alpha)
  File "/home/aditya/Research/phd/code/3DFuse/adapt_sd.py", line 78, in merge
    state_dict = lora_convert(_tmp_output, as_half=True)
  File "/home/aditya/Research/phd/code/3DFuse/adapt_sd.py", line 33, in lora_convert
    unet_state_dict = torch.load(unet_path, map_location="cpu")
  File "/home/aditya/miniconda3/envs/3DFuse/lib/python3.8/site-packages/torch/serialization.py", line 699, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/home/aditya/miniconda3/envs/3DFuse/lib/python3.8/site-packages/torch/serialization.py", line 230, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/home/aditya/miniconda3/envs/3DFuse/lib/python3.8/site-packages/torch/serialization.py", line 211, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: './results/corgi/lora/merge.tmp/unet/diffusion_pytorch_model.bin'

I think, the code wouldn't also be find the corresponding path inside the vae folder. There are no .bin files created inside the unet and vae folders, just model.safetensors files.

It's hard to install on windows,so could you provide dockerfile to run more convenient？

It is interesting to leverage depth injector and LoRA for better 3D consistency, and I get a good result with the given prompt "cute small corgi".
However, when evaluating the prompt "a ladybug", I get rather strange results:

The generated initial images are as follows:

Is it the expected results? Can you offer some advice to address the problem? Thank you!

After generating the point cloud using the image, I encountered an error when trying to load the state_dict of the 3DFuse model. Here is the stack trace of the error. I have checked the source code but couldn't find the cause.

ERROR LOGS:

No module 'xformers'. Proceeding without it.
ControlLDM: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Loaded model config from [cldm_v15.yaml]
Loaded state_dict from [weights/3DFuse_sparse_depth_injector.ckpt]
Traceback (most recent call last):
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/gradio/routes.py", line 439, in run_predict
    output = await app.get_blocks().process_api(
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/gradio/blocks.py", line 1384, in process_api
    result = await self.call_function(
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/gradio/blocks.py", line 1103, in call_function
    prediction = await utils.async_iteration(iterator)
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/gradio/utils.py", line 343, in async_iteration
    return await iterator.__anext__()
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/gradio/utils.py", line 336, in __anext__
    return await anyio.to_thread.run_sync(
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/gradio/utils.py", line 319, in run_sync_iterator_async
    return next(iterator)
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/gradio/utils.py", line 688, in gen_wrapper
    yield from f(*args, **kwargs)
  File "gradio_app.py", line 47, in gen_3d
    yield from model.run_gradio(points,exp_instance_dir)
  File "/usr/local/apps/3DFuse/run_3DFuse.py", line 138, in run_gradio
    model = getattr(self, family).make()
  File "/usr/local/apps/3DFuse/run_img_sampling.py", line 18, in make
    model = StableDiffusion(**args)
  File "/usr/local/apps/3DFuse/adapt_sd.py", line 135, in __init__
    model=load_3DFuse(self.checkpoint_root(),dir,alpha)
  File "/usr/local/apps/3DFuse/adapt_sd.py", line 121, in load_3DFuse
    model.load_state_dict(load_state_dict(control['control_weight'], location='cuda'))
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1604, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for ControlLDM:
	Unexpected key(s) in state_dict: "cond_stage_model.transformer.text_model.embeddings.position_ids". 

ERROR HANPPEND SOURCE CODE:
Module: adapt_sd.py

def load_3DFuse(control,dir,alpha):
    ######################LOADCONTROL###########################
    model = create_model(control['control_yaml']).cpu()
    model.load_state_dict(load_state_dict(control['control_weight'], location='cuda'))
    state_dict, l = merge("runwayml/stable-diffusion-v1-5",dir,alpha)
    
    #######################OVERRIDE#############################
    model.load_state_dict(state_dict,strict=False)
    
    #######################ADDEMBBEDDING########################
    load_embedding(model,l)
    ###############################################################
    return model

OTHER INFO:

git+https://github.com/openai/point-e
git+https://github.com/huggingface/diffusers
git+https://github.com/cloneofsimo/lora.git
git+https://github.com/Ir1d/image-background-remove-tool@2b68f5276d0f2e90f607c7845c60d2bddb79d5ba
I have loosened the version constraints for the opencv-contrib-python and safetensors libraries in the requirements.txt file in order to accommodate the upgraded dependencies of the third-party Git repository's code.

Hi @j0seo,
Nice repo and thanks for sharing your code. Just one question related to the semantic code sampling. As shown in the pipeline, the generated image in semantic code sampling is directly used for the coarse 3D point cloud generation. What if the generated object is shadowed or incomplete (e.g., only contain the upper body in the generated image). Do you have to manually pick the images for concept learning and the coarse 3D generation?

Hello. Nice repo. I have some questions related to the result of Experiment 5.3 (Figure 7). How is these result image generated? Based on the current code, I can't seem to determine where the model checkpoint is located or how it's used to get the final images.

1. How much VRAM do you need to run this locally?
2. When will the Colab and HuggingFace demos be available?
3. When will the point cloud to mesh converter be ready? Are you going to use Open3D library?

seed set to 0
path: /root/.cache/carvekit/checkpoints/tracer_b7/tracer_b7.pth
path: /root/.cache/carvekit/checkpoints/fba/fba_matting.pth
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 25/25 [00:00<00:00, 110.85it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 25/25 [00:01<00:00, 16.74it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 10.95it/s]
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/miaowei/programs/3DFuse/run_3DFuse.py:353 in │
│ │
│ 350 │
│ 351 │
│ 352 if name == "main": │
│ ❱ 353 │ dispatch(SJC_3DFuse) │
│ 354 │
│ │
│ /home/miaowei/programs/3DFuse/my/config.py:91 in dispatch │
│ │
│ 88 │ os.makedirs(exp_path, exist_ok=True) │
│ 89 │ write_full_config(mod, os.path.join(exp_path,"full_config.yml")) │
│ 90 │ │
│ ❱ 91 │ mod.run() │
│ 92 │
│ 93 def dispatch_gradio(module, prompt, keyword, ti_step, pt_step, seed, exp_instance_dir): │
│ 94 │ cfg = optional_load_config("gradio_init.yml") │
│ │
│ /home/miaowei/programs/3DFuse/run_3DFuse.py:100 in run │
│ │
│ 97 │ │ image_dir=os.path.join(exp_instance_dir,'initial_image') │
│ 98 │ │ │
│ 99 │ │ if semantic_model == "Karlo": │
│ ❱ 100 │ │ │ semantic_karlo(initial_prompt,image_dir,cfgs['num_initial_image'],cfgs['bg_p │
│ 101 │ │ elif semantic_model == "SD": │
│ 102 │ │ │ semantic_sd(initial_prompt,image_dir,cfgs['num_initial_image'],cfgs['bg_prep │
│ 103 │ │ else: │
│ │
│ /home/miaowei/programs/3DFuse/semantic_coding.py:50 in semantic_karlo │
│ │
│ 47 │ │ │ img_without_background = interface([image]) │
│ 48 │ │ │ mask = np.array(img_without_background[0]) > 127 │
│ 49 │ │ │ image = np.array(image) │
│ ❱ 50 │ │ │ image[~mask] = [255., 255., 255.] │
│ 51 │ │ │ # x, y, w, h = cv2.boundingRect(mask.astype(np.uint8)) │
│ 52 │ │ │ # image = image[y:y+h, x:x+w, :] │
│ 53 │ │ │ image = Image.fromarray(np.array(image)) │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
IndexError: boolean index did not match indexed array along dimension 2; dimension is 3 but corresponding boolean dimension is 4

Hey there! Awsome work on the paper !

I dont think I could find how much generation speed this model takes per prompt...
If you could provide the time taken and the gpu used that would be amazing!

Firstly, I would like to express my gratitude for your excellent work.

However, I have noticed that when I run the code, the resulting nerf model consistently appears with a green background color, and I am unsure about the reason behind this. I would like to inquire if you have any insights or suggestions regarding this issue.

It's worth mentioning that I haven't observed this phenomenon with other methods such as SJC and Dreambooth. I would appreciate any help you can provide in resolving this matter.

Thank you.

when i set prompt is " cute small corgi" and "a ladybug",why the result is this

i changed semantic_coding.py (image[~mask] = [255., 255., 255.]) to
image[~mask[:,:,3]] = [255., 255., 255.].

I found it hard to deal with the environment for the fact that the version of 'carvekit' conflicts to toch==1.12.0 (To run 'run.sh', carvekit package needs torch==1.11.0. Torch will be automatically uninstalled when installing carvekit.)
However, it could be a problem with the server environment I'm using. Thus, using the following yaml file may help

name: 3DFuse

channels:

• pytorch
• defaults

dependencies:

• python = 3.8.0
• pip=20.3
• cudatoolkit=11.3
• pytorch=1.11.0
• torchvision=0.12.0
• numpy=1.22.4
• pip:
  • gradio
  • albumentations==1.3.0
  • opencv-contrib-python==4.3.0.36
  • imageio==2.9.0
  • imageio-ffmpeg==0.4.2
  • pytorch-lightning==1.5.0
  • omegaconf==2.1.1
  • test-tube>=0.7.5
  • streamlit==1.12.1
  • einops==0.3.0
  • transformers
  • diffusers==0.11.1
  • webdataset==0.2.5
  • kornia==0.6
  • open_clip_torch==2.0.2
  • invisible-watermark>=0.1.5
  • streamlit-drawable-canvas==0.8.0
  • torchmetrics==0.6.0
  • timm==0.6.12
  • addict==2.4.0
  • yapf==0.32.0
  • prettytable==3.6.0
  • safetensors==0.2.7
  • basicsr==1.4.2
  • pydantic
  • click
  • easydict
  • tabulate
  • plotly
  • accelerate
  • carvekit

I just found that the default point cloud model was mistakenly typed as a smaller model (Point-E 40M). I have just pushed a commit that modifies this to use the 1B model, which produces much better results.

query = attn.to_q(hidden_states, *args)
return forward_call(*input, **kwargs)

TypeError: forward() takes 2 positional arguments but 3 were given
Is this a diffusers version mismatch?