astra-vision / materialpalette Goto Github PK

First of all, this is a great project. Thanks for sharing.

I tried the given sample mansion.zip. However, it generated a texture like this for grass, it is not smooth and contains 4 tile-like areas. Do you have any suggestions to improve this?

I saw in your paper that you had several different resolutions the texture could be scaled to. What is the average size per texture (kb or mb) at each resolution of 1024, 2048, 4096, and 8192?

Facing this issue after downloading blue_tiles:

(matpal) issac@issac-KVM:~/MaterialPalette$ python concept/infer.py blue_tiles/unet/adapter_model.bin --outdir OUTPUT
Namespace(path=PosixPath('blue_tiles/unet/adapter_model.bin'), outdir=PosixPath('OUTPUT'), token=None, stitch_mode='wmean', resolution=1024, prompt='p1', seed=1, renorm=False, num_inference_steps=50)
Traceback (most recent call last):
File "/home/issac/MaterialPalette/concept/infer.py", line 412, in
main(args)
File "/home/issac/miniconda3/envs/matpal/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/issac/MaterialPalette/concept/infer.py", line 114, in main
assert args.path.is_dir()
AssertionError

As explained in the Project Structure of your README, the data directory inside the capture directory holds essential files for running the complete pipeline. Since it is missing, attempts at running pipeline.py fail with the following error while importing modules:

python pipeline.py G:\SAGNIK_Material_Palette\Material_Palette\mansion
Traceback (most recent call last):
File "G:\SAGNIK_Material_Palette\Material_Palette\MaterialPalette\pipeline.py", line 7, in
import capture
File "G:\SAGNIK_Material_Palette\Material_Palette\MaterialPalette\capture_init_.py", line 2, in
from .utils.model import get_inference_module
File "G:\SAGNIK_Material_Palette\Material_Palette\MaterialPalette\capture\utils_init_.py", line 3, in
from .cli import get_args
File "G:\SAGNIK_Material_Palette\Material_Palette\MaterialPalette\capture\utils\cli.py", line 11, in
from ..data.module import DataModule
ModuleNotFoundError: No module named 'capture.data'

Is there anything on my end that can be done to solve this or is there an update slated to the repo itself that takes care of this?

Hello, I'm a student from SJTU. I'm deeply impressed by your exceptional work on materials. I wonder if you have any plans to release the source code as open-source in the future and I am really looking forward to it.

"WhitePaint" cluster, q=39.57%
512x512 kept 0 patches -> /content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/my_images/crops/WhitePaint
256x256 kept 0 patches -> /content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/my_images/crops/WhitePaint
192x192 kept 0 patches -> /content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/my_images/crops/WhitePaint
128x128 kept 4 patches -> /content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/my_images/crops/WhitePaint
---- kept 2/3 crops.
04/18/2024 03:24:04 - INFO - concept.utils - Distributed environment: NO
Num processes: 1
Process index: 0
Local process index: 0
Device: cuda

Mixed precision type: fp16

trainable params: 589,824 || all params: 123,650,304 || trainable%: 0.4770097451600281
{'scaling_factor', 'force_upcast'} was not found in config. Values will be initialized to default values.
{'num_attention_heads', 'mid_block_only_cross_attention', 'dual_cross_attention', 'resnet_skip_time_act', 'addition_time_embed_dim', 'time_embedding_type', 'cross_attention_norm', 'class_embed_type', 'only_cross_attention', 'conv_out_kernel', 'time_embedding_dim', 'resnet_time_scale_shift', 'conv_in_kernel', 'transformer_layers_per_block', 'encoder_hid_dim_type', 'addition_embed_type', 'resnet_out_scale_factor', 'projection_class_embeddings_input_dim', 'timestep_post_act', 'time_cond_proj_dim', 'num_class_embeds', 'mid_block_type', 'encoder_hid_dim', 'class_embeddings_concat', 'time_embedding_act_fn', 'addition_embed_type_num_heads', 'upcast_attention', 'use_linear_projection'} was not found in config. Values will be initialized to default values.
trainable params: 1,594,368 || all params: 861,115,332 || trainable%: 0.18515150535027286
04/18/2024 03:24:08 - INFO - concept.utils - ***** Running training *****
04/18/2024 03:24:08 - INFO - concept.utils - Num examples = 12
04/18/2024 03:24:08 - INFO - concept.utils - Num batches each epoch = 12
04/18/2024 03:24:08 - INFO - concept.utils - Instantaneous batch size per device = 1
04/18/2024 03:24:08 - INFO - concept.utils - Total train batch size (w. parallel, distributed) = 1
04/18/2024 03:24:08 - INFO - concept.utils - Total optimization steps = 800
Steps: 100% 800/800 [05:04<00:00, 2.62it/s, loss=0.685, lr=0.0001]
loading LoRA with token azertyuiop
{'requires_safety_checker'} was not found in config. Values will be initialized to default values.
Loading pipeline components...: 0% 0/6 [00:00<?, ?it/s]Loaded feature_extractor as CLIPImageProcessor from feature_extractor subfolder of runwayml/stable-diffusion-v1-5.
Loaded text_encoder as CLIPTextModel from text_encoder subfolder of runwayml/stable-diffusion-v1-5.
Loading pipeline components...: 33% 2/6 [00:00<00:00, 7.29it/s]{'timestep_spacing', 'prediction_type'} was not found in config. Values will be initialized to default values.
Loaded scheduler as PNDMScheduler from scheduler subfolder of runwayml/stable-diffusion-v1-5.
{'scaling_factor', 'force_upcast'} was not found in config. Values will be initialized to default values.
Loaded vae as AutoencoderKL from vae subfolder of runwayml/stable-diffusion-v1-5.
Loading pipeline components...: 67% 4/6 [00:00<00:00, 8.02it/s]Loaded tokenizer as CLIPTokenizer from tokenizer subfolder of runwayml/stable-diffusion-v1-5.
{'num_attention_heads', 'mid_block_only_cross_attention', 'dual_cross_attention', 'resnet_skip_time_act', 'addition_time_embed_dim', 'time_embedding_type', 'cross_attention_norm', 'class_embed_type', 'only_cross_attention', 'conv_out_kernel', 'time_embedding_dim', 'resnet_time_scale_shift', 'conv_in_kernel', 'transformer_layers_per_block', 'encoder_hid_dim_type', 'addition_embed_type', 'resnet_out_scale_factor', 'projection_class_embeddings_input_dim', 'timestep_post_act', 'time_cond_proj_dim', 'num_class_embeds', 'mid_block_type', 'encoder_hid_dim', 'class_embeddings_concat', 'time_embedding_act_fn', 'addition_embed_type_num_heads', 'upcast_attention', 'use_linear_projection'} was not found in config. Values will be initialized to default values.
Loaded unet as UNet2DConditionModel from unet subfolder of runwayml/stable-diffusion-v1-5.
Loading pipeline components...: 100% 6/6 [00:01<00:00, 4.90it/s]
You have disabled the safety checker for <class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> by passing safety_checker=None. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at huggingface/diffusers#254 .
p1 => top view realistic texture of {}
ignoring args.outdir and using path /content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/my_images/weights/Pebbles/an_object_with_azertyuiop_texture/checkpoint-800/outputs
preparing for /content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/my_images/weights/Pebbles/an_object_with_azertyuiop_texture/checkpoint-800/outputs/azertyuiop_1K_t50_wmean_top-view-realistic-texture-of-o_1.png
100% 50/50 [00:10<00:00, 4.82it/s]
Traceback (most recent call last):
File "/content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/pipeline.py", line 21, in
concept.infer(lora, renorm=True)
File "/content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/concept/infer.py", line 398, in infer
return main(Namespace(
File "/usr/local/envs/matpal/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/concept/infer.py", line 393, in main
renorm(fname)
File "/content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/concept/renorm.py", line 40, in renorm
low_threshold = sorted_pixels[exclude_count]
IndexError: index 0 is out of bounds for dimension 0 with size 0

I got the following error when trying the mansion.zip pipeline example provided in the readme.

Traceback (most recent call last):
File "G:\SAGNIK_Material_Palette\Material_Palette\MaterialPalette\pipeline.py", line 25, in
module = capture.get_inference_module(pt='model.ckpt')
File "G:\SAGNIK_Material_Palette\Material_Palette\MaterialPalette\capture\utils\model.py", line 41, in get_inference_module
assert Path(pt).exists()
AssertionError

I cant figure out what the model.ckpt is supposed to be referencing because there is no ckpt file in the capture directory itself.