kijai / comfyui-marigold Goto Github PK

View Code? Open in Web Editor NEW

424.0 6.0 21.0 16.74 MB

Marigold depth estimation in ComfyUI

License: GNU General Public License v3.0

Python 100.00%

comfyui-marigold's Introduction

Marigold depth estimation in ComfyUI

marigold_nodes.mp4

This is a wrapper node for Marigold depth estimation: https://github.com/prs-eth/Marigold

Join us at the Banodoco Discord for discussion on the use and node development: https://discord.com/channels/1076117621407223829/1184863853096484865

What I know of the parameters so far:

denoise_steps: steps per depth map, increase for accuracy in exchange of processing time

n_repeat: amount of iterations to be ensembled into single depth map, increase for accuracy in exchange of processing time

n_repeat_batch_size: how many of the n_repeats are processed as a batch, if you have the VRAM this can match the n_repeats for faster processing

invert: marigold by default produces depth map where black is front, for controlnets etc. we want the opposite

regularizer_strength, reduction_method, max_iter, tol (tolerance) are settings for the ensembling process, don't fully know how to use them yet.

It can pretty memory hungry, and slow, fp16 halves the memory use. Marigold is meant to be run around 768p resolution so resizing is recommended, at higher res your mileage may wary. I added a remap node to see the full range better, and OpenEXR node to save the full range, works wonders compared to default png when used in VFX/3D modeling software.

Installing:

Recommended way:

Use the ComfyUI manager (search for "marigold")

Manual install:

Clone this repo to ComfyUI/custom_nodes Install requirements: pip install -r requirements.txt

Get the model:

Currently using the same diffusers pipeline as in the original implementation, so in addition to the custom node, you need the model in diffusers format.

If the model is not found, it should autodownload with hugginface_hub. Alternatively get it manually from: https://huggingface.co/Bingxin/Marigold (or do git clone https://huggingface.co/Bingxin/Marigold/) in either of these folders:

ComfyUI\custom_nodes\ComfyUI-Marigold\checkpoints or ComfyUI\models\diffusers

comfyui-marigold's People

Contributors

Stargazers

Watchers

comfyui-marigold's Issues

macOS half not implemented

I got the Marigold node now, but getting this weird error:
`Error occurred when executing MarigoldDepthEstimation: "LayerNormKernelImpl" not implemented for 'Half' File

Btw, how to get Image Resize node?

'No checkpoint directory found.'

I've followed your setup as described, unpacked the .tar in ComfyUI\custom_nodes\checkpoints but I keep getting this error when it tries to load the diffuser model?

Input type (float) and bias type (struct c10::Half) should be the same

un-checking the use fp16...

Error occurred when executing MarigoldDepthEstimation:

Input type (float) and bias type (struct c10::Half) should be the same

File "I:\ai\ComfyUI_windows_portable\ComfyUI\execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\ComfyUI\execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\ComfyUI\execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Marigold\nodes.py", line 111, in process
depth_maps_sub_batch = self.marigold_pipeline(sub_batch, num_inference_steps=denoise_steps, show_pbar=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Marigold\marigold\model\marigold_pipeline.py", line 205, in forward
rgb_latent = self.encode_rgb(rgb_in)
^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Marigold\marigold\model\marigold_pipeline.py", line 282, in encode_rgb
rgb_latent = self.rgb_encoder(rgb_in) # [B, 4, h, w]
^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Marigold\marigold\model\rgb_encoder.py", line 30, in forward
return self.encode(rgb_in)
^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Marigold\marigold\model\rgb_encoder.py", line 33, in encode
moments = self.rgb_encoder(rgb_in) # [B, 8, H/8, W/8]
^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\container.py", line 215, in forward
input = module(input)
^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\python_embeded\Lib\site-packages\diffusers\models\vae.py", line 141, in forward
sample = self.conv_in(sample)
^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\conv.py", line 460, in forward
return self._conv_forward(input, self.weight, self.bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\ai\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\conv.py", line 456, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Cannot initialize model with low cpu memory usage because `accelerate` was not found in the environment.

Hello,
I'm getting the following warning:

Cannot initialize model with low cpu memory usage because `accelerate` was not found in the environment. Defaulting to `low_cpu_mem_usage=False`. It is strongly recommended to install `accelerate` for faster and less memory-intense model loading. You can do so with:

pip install accelerate

I'm using ComfyUI portable and ran the command using:

F:\ComfyUI_windows_portable\python_embeded>python -m pip install accelerate

it installed correctly but I'm still getting the same message, it is running, but it's taking forever... any ideas? thx !

no file named diffusion_pytorch_model.bin

hi i'm already used last version of Comfyui , i tried to download the model as you mentioned , same error
could you help please

I downloaded the necessary models and nodes but Marigold is not available. What could be the cause?

Could you make a node for TilingZoeDepth?

Thank you so much for Marigold! Could you make a node for TilingZoeDepth? (https://github.com/BillFSmith/TilingZoeDepth)
This method gives simply wonderful results, but unfortunately it is only in Collab and is difficult to use

Marigold-lcm has been released

Marigold-lcm has been released, and the official demo is very fast.
This is the online test address: huggingface.co/spaces/prs-eth/marigold-lcm
This is the model address: huggingface.co/prs-eth/marigold-lcm-v1-0

Please check the contents of the error.

I'm working on using Anaconda Python 3.11.7 on Windows PowerShell, but it's not downloading properly due to the error below..
(It's my first time working on this, so I'd appreciate it if you could teach me well due to my lack of knowledge.)

from diffusers import DiffusionPipeline
pipeline = DiffusionPipeline.from_pretrained("prs-eth/marigold-v1-0")
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Administrator\anaconda3\Lib\site-packages\huggingface_hub\utils_validators.py", line 119, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\anaconda3\Lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 671, in from_pretrained
cached_folder = cls.download(
^^^^^^^^^^^^^
File "C:\Users\Administrator\anaconda3\Lib\site-packages\huggingface_hub\utils_validators.py", line 119, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\anaconda3\Lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1317, in download
pipeline_class = _get_pipeline_class(
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\anaconda3\Lib\site-packages\diffusers\pipelines\pipeline_loading_utils.py", line 343, in _get_pipeline_class
pipeline_cls = getattr(diffusers_module, class_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\anaconda3\Lib\site-packages\diffusers\utils\import_utils.py", line 711, in getattr
raise AttributeError(f"module {self.name} has no attribute {name}")
AttributeError: module diffusers has no attribute MarigoldPipeline

Prompt outputs failed validation MarigoldDepthEstimation: - Required input is missing: scheduler

A got a simple Workflow but it getting this error

After install, no idea how to use?

Would be nice to have workflow screenshot example or json? I cannot produce the depth by guessing.

Messages from Marigold authors

Thanks for creating the node, very happy that the community is interested in Marigold!

According to some users (https://www.reddit.com/r/comfyui/s/1QO2IAdUHl), the quality of the produced result is not always consistent with our online demo. We would like to recommend making the default settings of the node consistent with the default settings of our repository.

Additionally, we are working to update the source code of our pipeline to include several performance fixes, so stay tuned for updates!

Anton

Error occurred when executing MarigoldDepthEstimation: "slow_conv2d_cpu" not implemented for 'Half'

Got the following error "out of the box"

Error occurred when executing MarigoldDepthEstimation:

"slow_conv2d_cpu" not implemented for 'Half'

File "D:\apps\SD-WebUI\ComfyUI\execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "D:\apps\SD-WebUI\ComfyUI\execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "D:\apps\SD-WebUI\ComfyUI\execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "D:\apps\SD-WebUI\ComfyUI\custom_nodes\ComfyUI-Marigold\nodes.py", line 123, in process
depth_maps_sub_batch = self.marigold_pipeline(sub_batch, num_inference_steps=denoise_steps, show_pbar=False)
File "D:\apps\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "D:\apps\SD-WebUI\ComfyUI\custom_nodes\ComfyUI-Marigold\marigold\model\marigold_pipeline.py", line 211, in forward
rgb_latent = self.encode_rgb(rgb_in)
File "D:\apps\SD-WebUI\ComfyUI\custom_nodes\ComfyUI-Marigold\marigold\model\marigold_pipeline.py", line 287, in encode_rgb
rgb_latent = self.rgb_encoder(rgb_in) # [B, 4, h, w]
File "D:\apps\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "D:\apps\SD-WebUI\ComfyUI\custom_nodes\ComfyUI-Marigold\marigold\model\rgb_encoder.py", line 30, in forward
return self.encode(rgb_in)
File "D:\apps\SD-WebUI\ComfyUI\custom_nodes\ComfyUI-Marigold\marigold\model\rgb_encoder.py", line 33, in encode
moments = self.rgb_encoder(rgb_in) # [B, 8, H/8, W/8]
File "D:\apps\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "D:\apps\Python\Python310\lib\site-packages\torch\nn\modules\container.py", line 217, in forward
input = module(input)
File "D:\apps\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "D:\apps\Python\Python310\lib\site-packages\diffusers\models\autoencoders\vae.py", line 143, in forward
sample = self.conv_in(sample)
File "D:\apps\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "D:\apps\Python\Python310\lib\site-packages\torch\nn\modules\conv.py", line 463, in forward
return self._conv_forward(input, self.weight, self.bias)
File "D:\apps\Python\Python310\lib\site-packages\torch\nn\modules\conv.py", line 459, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,

Nothing else in the Workflow:

Unable to choose the spectral (coloured) output as an option?

Hi.

I can't figure out to output the spectral/coloured depth maps which are possible with the non-comfyui version of Marigold. I'd prefer to use these if possible because they contain more depth information than greyscale.

Cannot run the nodes

Installing the node from the ComfyUI Manager I get this error when restarting the server:

FileNotFoundError: [Errno 2] No such file or directory: '/run/media/minerva/SSD/ComfyUI/ComfyUI/custom_nodes/Marigold-Checkpoint/__init__.py'

Cannot import /run/media/minerva/SSD/ComfyUI/ComfyUI/custom_nodes/Marigold-Checkpoint module for custom nodes: [Errno 2] No such file or directory: '/run/media/minerva/SSD/ComfyUI/ComfyUI/custom_nodes/Marigold-Checkpoint/__init__.py'
Traceback (most recent call last):
  File "/run/media/minerva/SSD/ComfyUI/ComfyUI/nodes.py", line 1800, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/run/media/minerva/SSD/ComfyUI/ComfyUI/custom_nodes/ComfyUI-Marigold/__init__.py", line 1, in <module>
    from .nodes import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
  File "/run/media/minerva/SSD/ComfyUI/ComfyUI/custom_nodes/ComfyUI-Marigold/nodes.py", line 24, in <module>
    empty_text_embed = torch.load(os.path.join(__file__, '..', "empty_text_embed.pt"), map_location="cpu")
  File "/run/media/minerva/SSD/ComfyUI/ComfyUI/venv/lib/python3.10/site-packages/torch/serialization.py", line 986, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/run/media/minerva/SSD/ComfyUI/ComfyUI/venv/lib/python3.10/site-packages/torch/serialization.py", line 435, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/run/media/minerva/SSD/ComfyUI/ComfyUI/venv/lib/python3.10/site-packages/torch/serialization.py", line 416, in __init__
    super().__init__(open(name, mode))
NotADirectoryError: [Errno 20] Not a directory: '/run/media/minerva/SSD/ComfyUI/ComfyUI/custom_nodes/ComfyUI-Marigold/nodes.py/../empty_text_embed.pt'

Cannot import /run/media/minerva/SSD/ComfyUI/ComfyUI/custom_nodes/ComfyUI-Marigold module for custom nodes: [Errno 20] Not a directory: '/run/media/minerva/SSD/ComfyUI/ComfyUI/custom_nodes/ComfyUI-Marigold/nodes.py/../empty_text_embed.pt'

From what I understand, this error seems to be caused by a line introduced in commit 1a50170

[Request] location to insert the checkpoint

Thank you for the excellent response. Regarding the placement of the model, is it possible to change it to use this location?

ComfyUI\models\diffusers

Install failed - Cannot import name LCMScheduler from diffuser

Hi guys,

I have an issue after the install preventing me to use the nodes

`Traceback (most recent call last):
  File "D:\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1889, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Marigold\__init__.py", line 1, in <module>
    from .nodes import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
  File "D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Marigold\nodes.py", line 5, in <module>
    from .marigold.model.marigold_pipeline import MarigoldPipeline
  File "D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Marigold\marigold\model\marigold_pipeline.py", line 9, in <module>
    from diffusers import (
ImportError: cannot import name 'LCMScheduler' from 'diffusers' (D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\diffusers\__init__.py)

[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.jsonCannot import D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Marigold module for custom nodes: cannot import name 'LCMScheduler' from 'diffusers' (D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\diffusers\__init__.py)`

Any pointer to get this resolved ?

thank you !

Marigold import failed diffusers after installing new custom nodes

Adding any other custom node that uses diffusers (pip install diffusers) breaks Marigold and fails to import.

Current work around seem to be to manually uninstall & reinstall diffusers from python_embeded & main python install (only if also required)

Disabling or uninstalling other nodes added does not fix, only diffusers reinstall will stop the import failed:

comfyui log:
[2024-05-01 13:20] Traceback (most recent call last):
File "C:\ComfyUi\ComfyUI\nodes.py", line 1864, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 883, in exec_module
File "", line 241, in call_with_frames_removed
File "C:\ComfyUi\ComfyUI\custom_nodes\ComfyUI-Marigold_init.py", line 1, in
from .nodes import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
File "C:\ComfyUi\ComfyUI\custom_nodes\ComfyUI-Marigold\nodes.py", line 5, in
from .marigold.model.marigold_pipeline import MarigoldPipeline
File "C:\ComfyUi\ComfyUI\custom_nodes\ComfyUI-Marigold\marigold\model\marigold_pipeline.py", line 9, in
from diffusers import (
ImportError: cannot import name 'LCMScheduler' from 'diffusers' (C:\ComfyUi\python_embeded\lib\site-packages\diffusers_init_.py)

After reinstall, marigold will load normally with no import failed

The scheduler is missing the LCM component. How do I download it and in which path should I place it?

MPS error "Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead."

Using marigold_LCM_example_01.json

Error occurred when executing MarigoldDepthEstimation:

Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.

File "/Users/s/ComfyUI/execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/s/ComfyUI/execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/s/ComfyUI/execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/s/ComfyUI/custom_nodes/ComfyUI-Marigold/nodes.py", line 184, in process
depth_map, pred_uncert = ensemble_depths(
^^^^^^^^^^^^^^^^
File "/Users/s/ComfyUI/custom_nodes/ComfyUI-Marigold/marigold/util/ensemble.py", line 82, in ensemble_depths
s = torch.from_numpy(s).to(device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I've tried with and without use_fp16 and with and without --force-fp32 commandline flag enabled and the same error is given, also with export PYTORCH_ENABLE_MPS_FALLBACK=1 in case that made any difference and it didn't.

Is it possible to change the Marigold node to add a cache function?

I am working on a tool and I am trying to optimize it, I manage to cache all models on the VRAM except the Marigold model, there is the possibility to keep the model loaded but if I change the workflow and I don't use the marigold node it takes time to be loaded when I put another workflow with Marigold on it.

So my question is, can I cache the model into the VRAM? or if not can I separate the marigold model from the main node and instead add a socket that inserts the model on it (like the regular sd checkpoint) and then I can cache this model to the VRAM with a caching node.

This would improve in about 35% the generation speed in some workflows of my project, does anyone have an idea about how to do that?

Marigold import failed ,cannot import name 'builder' from 'google.protobuf.internal'

File "D:\ComfyUI-aki-v1.2\custom_nodes\ComfyUI-Marigold_init_.py", line 1, in
from .nodes import MarigoldDepthEstimation, MarigoldDepthEstimationVideo, ColorizeDepthmap, SaveImageOpenEXR, RemapDepth
File "D:\ComfyUI-aki-v1.2\custom_nodes\ComfyUI-Marigold\nodes.py", line 8, in
from .marigold.model.marigold_pipeline import MarigoldPipeline
File "D:\ComfyUI-aki-v1.2\custom_nodes\ComfyUI-Marigold\marigold\model\marigold_pipeline.py", line 9, in
from diffusers import (
File "", line 1075, in _handle_fromlist
File "D:\ComfyUI-aki-v1.2\python\lib\site-packages\diffusers\utils\import_utils.py", line 799, in getattr
value = getattr(module, name)
File "D:\ComfyUI-aki-v1.2\python\lib\site-packages\diffusers\utils\import_utils.py", line 798, in getattr
module = self._get_module(self._class_to_module[name])
File "D:\ComfyUI-aki-v1.2\python\lib\site-packages\diffusers\utils\import_utils.py", line 810, in get_module
raise RuntimeError(
RuntimeError: Failed to import diffusers.models.unets.unet_2d_condition because of the following error (look up to see its traceback):
Failed to import diffusers.loaders.single_file_model because of the following error (look up to see its traceback):
Failed to import transformers.models.auto.image_processing_auto because of the following error (look up to see its traceback):
cannot import name 'builder' from 'google.protobuf.internal' (D:\ComfyUI-aki-v1.2\python\lib\site-packages\google\protobuf\internal_init.py)

Cannot import D:\ComfyUI-aki-v1.2\custom_nodes\ComfyUI-Marigold module for custom nodes: Failed to import diffusers.models.unets.unet_2d_condition because of the following error (look up to see its traceback):
Failed to import diffusers.loaders.single_file_model because of the following error (look up to see its traceback):
Failed to import transformers.models.auto.image_processing_auto because of the following error (look up to see its traceback):
cannot import name 'builder' from 'google.protobuf.internal' (D:\ComfyUI-aki-v1.2\python\lib\site-packages\google\protobuf\internal_init_.py)

my env

[SD-Launcher] D:\ComfyUI-aki-v1.2>python
Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(torch.__version__)
2.1.2+cu118
>>> print(torch.cuda.is_available()) 
True
>>> print(torch.version.cuda) 
11.8

diffusers 0.29.2

[ Bug ] duplicated model downloads ( in the cmd line )

The node works.

And, despide what you see in the cmd line, all the models will be downloaded correctly.

So I opened this mostly to help the next users.

There is an issue in the way models are downloaded.

I have a portable Comy UI install, on Windows 10.
I installed this node via the Comfy UI Manager.

The first time I used the node, it started to download three 3.46 GB models, and another three 1.36 GB models, all simultaneously !
I just let it finish, because I didn't want to break things.

Thankfully, it stopped after the first model finished downloading.

It created a ComfyUI\models\diffusers\Marigold folder, with all the files in the repository.

So it worked well, in the end, but this may be stealing bandwidth, and making the downloads slower.

For AMD (Directml) users

Clone the repository and remove torch from requirements.txt (assuming you already have torch-directml installed) before pip installing it.
Delete or comment-out the line with torch.cuda.ipc_collect() from nodes.py on custom_nodes\ComfyUI-Marigold folder.

On the workflow, on the node:
reduction_method: mean, works on fp16
reduction_method: median, errors on fp16

scheduler: DEISMultistepScheduler, is the only one that doesn't freeze during the denoising step of the node.

That's how it works for me with my Rx580 (8Gb), on Windows. I only do 1 n_repeat_batch_size at 768 resolution or 2 at half that resolution, but maybe I can go higher. Half the recommended resolution is fast to compute and still gives better results than Midas, imo.

RuntimeError: Calculated padded input size per channel: (2 x 2). Kernel size: (3 x 3). Kernel size can't be greater than actual input size

How can I slove this ?

Strong parasitic noise

A good model, but unlike it gives a very strong Noise after generation (in Tiled Zoe Depth there is no such of noise)

When you run it, it tells you that the path to the model file cannot be found, but you have already downloaded the model file.

The model files are downloaded in the "/models/diffusers/marigold" directory, but the model files have not been renamed, I don't know if the model files need to be renamed, and if so, what should they be renamed to?

Black image

It only seems to put out a black image in any resolution,
no error that stops the process, and only the following warnings in console:


\diffusers\models\attention_processor.py:1231: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:253.)
  hidden_states = F.scaled_dot_product_attention(

extracting

Can it be made not to rerun when extracting the same image with the same seed?

3D side-by-side format videos

I can ask where to find projects that use depth maps to create 3D side-by-side format videos? Does the author have any thoughts on adding such functionality?

ImportError: cannot import name 'LCMScheduler' from 'diffusers'

When launching Comfy after pulling a few minutes ago, I get:

Traceback (most recent call last):
  File "/ai/ComfyUI/nodes.py", line 1889, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/ai/ComfyUI/custom_nodes/ComfyUI-Marigold/__init__.py", line 1, in <module>
    from .nodes import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
  File "/ai/ComfyUI/custom_nodes/ComfyUI-Marigold/nodes.py", line 5, in <module>
    from .marigold.model.marigold_pipeline import MarigoldPipeline
  File "/ai/ComfyUI/custom_nodes/ComfyUI-Marigold/marigold/model/marigold_pipeline.py", line 9, in <module>
    from diffusers import (
ImportError: cannot import name 'LCMScheduler' from 'diffusers' (/ai/ve/comfy/lib/python3.10/site-packages/diffusers/__init__.py)

Cannot import /ai/ComfyUI/custom_nodes/ComfyUI-Marigold module for custom nodes: cannot import name 'LCMScheduler' from 'diffusers' (/ai/ve/comfy/lib/python3.10/site-packages/diffusers/__init__.py)

Incorporate bug fix of Marigold core

Marigold has fixed an important bug that might lead to worse performance: prs-eth/Marigold@9ed49d7. Please incorporate this update.

Error occurred when executing NegiTools_DepthEstimationByMarigold

Linux Manjaro
Cuda 12.1
Python 3.12

Guys is it error related to other plugin or could be something of marigold node? I've installed the node and download manually the models using git clone (with the large files enabled) and set in location like in the documentation, but it dont work.

Error occurred when executing NegiTools_DepthEstimationByMarigold:

[Errno 2] No such file or directory: '/home/noe/Documentos/ComfyUI/custom_nodes/ComfyUI-NegiTools/dependencies/Marigold/work/output/depth_npy/image_pred.npy'

File "/home/noe/Documentos/ComfyUI/execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/Documentos/ComfyUI/execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/Documentos/ComfyUI/execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/Documentos/ComfyUI/custom_nodes/ComfyUI-NegiTools/negi/depth_estimation_by_marigold.py", line 123, in doit
im1 = np.load(os.path.join(output_dir, "depth_npy", "image_pred.npy")).astype(np.float32)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/.pyenv/versions/Comfyv4_312/lib/python3.12/site-packages/numpy/lib/npyio.py", line 427, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^

kijai / comfyui-marigold Goto Github PK

comfyui-marigold's Introduction

Marigold depth estimation in ComfyUI

Installing:

comfyui-marigold's People

Contributors

Stargazers

Watchers

Forkers

comfyui-marigold's Issues

Recommend Projects

Recommend Topics

Recommend Org