(hcpd) ➜  HCP-Diffusion git:(main) ✗ accelerate launch -m hcpdiff.train_ac_single --cfg cfgs/train/examples/DreamArtist++2.yaml
[11:10:59] WARNING  The following values were not passed to `accelerate launch` and had defaults used instead:              launch.py:890
                            `--num_processes` was set to a value of `1`
                            `--num_machines` was set to a value of `1`
                            `--mixed_precision` was set to a value of `'no'`
                            `--dynamo_backend` was set to a value of `'no'`
                    To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
tensorboard is not available
wandb is not available
2023-05-15 11:11:01.927 | INFO     | hcpdiff.loggers.cli_logger:_info:30 - world_size: 1
2023-05-15 11:11:01.927 | INFO     | hcpdiff.loggers.cli_logger:_info:30 - accumulation: 1
You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
2023-05-15 11:11:08.818 | INFO     | hcpdiff.data.bucket:build_buckets_from_images:130 - build buckets from images
/data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
2023-05-15 11:11:08.841 | INFO     | hcpdiff.data.bucket:build_buckets_from_images:159 - buckets info: size:[512 512], num:1
2023-05-15 11:11:08.842 | INFO     | __main__:build_data:57 - len(train_dataset): 4
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:02<00:00,  1.87it/s]
2023-05-15 11:11:12.413 | INFO     | hcpdiff.loggers.cli_logger:_info:30 - ***** Running training *****
2023-05-15 11:11:12.414 | INFO     | hcpdiff.loggers.cli_logger:_info:30 -   Num batches each epoch = 1
2023-05-15 11:11:12.414 | INFO     | hcpdiff.loggers.cli_logger:_info:30 -   Num Steps = 1000
2023-05-15 11:11:12.414 | INFO     | hcpdiff.loggers.cli_logger:_info:30 -   Instantaneous batch size per device = 4
2023-05-15 11:11:12.414 | INFO     | hcpdiff.loggers.cli_logger:_info:30 -   Total train batch size (w. parallel, distributed & accumulation) = 4
2023-05-15 11:11:12.414 | INFO     | hcpdiff.loggers.cli_logger:_info:30 -   Gradient Accumulation steps = 1
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/runpy.py:196 in _run_module_as_main                │
│                                                                                                  │
│   193 │   main_globals = sys.modules["__main__"].__dict__                                        │
│   194 │   if alter_argv:                                                                         │
│   195 │   │   sys.argv[0] = mod_spec.origin                                                      │
│ ❱ 196 │   return _run_code(code, main_globals, None,                                             │
│   197 │   │   │   │   │    "__main__", mod_spec)                                                 │
│   198                                                                                            │
│   199 def run_module(mod_name, init_globals=None,                                                │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/runpy.py:86 in _run_code                           │
│                                                                                                  │
│    83 │   │   │   │   │      __loader__ = loader,                                                │
│    84 │   │   │   │   │      __package__ = pkg_name,                                             │
│    85 │   │   │   │   │      __spec__ = mod_spec)                                                │
│ ❱  86 │   exec(code, run_globals)                                                                │
│    87 │   return run_globals                                                                     │
│    88                                                                                            │
│    89 def _run_module_code(code, init_globals=None,                                              │
│                                                                                                  │
│ /data/yoke/HCP-Diffusion/hcpdiff/train_ac_single.py:105 in <module>                              │
│                                                                                                  │
│   102 │                                                                                          │
│   103 │   conf = load_config_with_cli(args.cfg, args_list=sys.argv[3:]) # skip --cfg             │
│   104 │   trainer=TrainerSingleCard(conf)                                                        │
│ ❱ 105 │   trainer.train()                                                                        │
│   106                                                                                            │
│                                                                                                  │
│ /data/yoke/HCP-Diffusion/hcpdiff/train_ac.py:409 in train                                        │
│                                                                                                  │
│   406 │   │                                                                                      │
│   407 │   │   loss_sum = np.ones(30)                                                             │
│   408 │   │   for data_list in self.train_loader_group:                                          │
│ ❱ 409 │   │   │   loss = self.train_one_step(data_list)                                          │
│   410 │   │   │   loss_sum[self.global_step%len(loss_sum)] = loss                                │
│   411 │   │   │                                                                                  │
│   412 │   │   │   self.global_step += 1                                                          │
│                                                                                                  │
│ /data/yoke/HCP-Diffusion/hcpdiff/train_ac.py:501 in train_one_step                               │
│                                                                                                  │
│   498 │   │   │   │   other_datas = {k:v.to(self.device, dtype=self.weight_dtype) for k, v in    │
│   499 │   │   │   │                                                                              │
│   500 │   │   │   │   latents = self.get_latents(image, self.train_loader_group.get_dataset(id   │
│ ❱ 501 │   │   │   │   model_pred, target, timesteps = self.forward(latents, prompt_ids, **othe   │
│   502 │   │   │   │   loss = self.get_loss(model_pred, target, timesteps, att_mask) * self.tra   │
│   503 │   │   │   │   self.accelerator.backward(loss)                                            │
│   504                                                                                            │
│                                                                                                  │
│ /data/yoke/HCP-Diffusion/hcpdiff/train_ac.py:479 in forward                                      │
│                                                                                                  │
│   476 │   │                                                                                      │
│   477 │   │   # CFG context for DreamArtist                                                      │
│   478 │   │   noisy_latents, timesteps = self.cfg_context.pre(noisy_latents, timesteps)          │
│ ❱ 479 │   │   model_pred = self.encode_decode(prompt_ids, noisy_latents, timesteps, **kwargs)    │
│   480 │   │   model_pred = self.cfg_context.post(model_pred)                                     │
│   481 │   │                                                                                      │
│   482 │   │   # Get the target for loss depending on the prediction type                         │
│                                                                                                  │
│ /data/yoke/HCP-Diffusion/hcpdiff/train_ac_single.py:78 in encode_decode                          │
│                                                                                                  │
│    75 │   │   │   │   feeder(input_all)                                                          │
│    76 │   │                                                                                      │
│    77 │   │   encoder_hidden_states = self.text_encoder(prompt_ids, output_hidden_states=True)   │
│ ❱  78 │   │   model_pred = self.unet(noisy_latents, timesteps, encoder_hidden_states).sample     │
│    79 │   │   return model_pred                                                                  │
│    80 │                                                                                          │
│    81 │   def get_loss(self, model_pred, target, timesteps, att_mask):                           │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/torch/nn/modules/module.py:1130 in   │
│ _call_impl                                                                                       │
│                                                                                                  │
│   1127 │   │   # this function, and just call forward.                                           │
│   1128 │   │   if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks o  │
│   1129 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                   │
│ ❱ 1130 │   │   │   return forward_call(*input, **kwargs)                                         │
│   1131 │   │   # Do not call functions when jit is used                                          │
│   1132 │   │   full_backward_hooks, non_full_backward_hooks = [], []                             │
│   1133 │   │   if self._backward_hooks or _global_backward_hooks:                                │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/accelerate/utils/operations.py:521   │
│ in forward                                                                                       │
│                                                                                                  │
│   518 │   model_forward = ConvertOutputsToFp32(model_forward)                                    │
│   519 │                                                                                          │
│   520 │   def forward(*args, **kwargs):                                                          │
│ ❱ 521 │   │   return model_forward(*args, **kwargs)                                              │
│   522 │                                                                                          │
│   523 │   # To act like a decorator so that it can be popped when doing `extract_model_from_pa   │
│   524 │   forward.__wrapped__ = model_forward                                                    │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/accelerate/utils/operations.py:509   │
│ in __call__                                                                                      │
│                                                                                                  │
│   506 │   │   update_wrapper(self, model_forward)                                                │
│   507 │                                                                                          │
│   508 │   def __call__(self, *args, **kwargs):                                                   │
│ ❱ 509 │   │   return convert_to_fp32(self.model_forward(*args, **kwargs))                        │
│   510 │                                                                                          │
│   511 │   def __getstate__(self):                                                                │
│   512 │   │   raise pickle.PicklingError(                                                        │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/torch/amp/autocast_mode.py:12 in     │
│ decorate_autocast                                                                                │
│                                                                                                  │
│     9 │   @functools.wraps(func)                                                                 │
│    10 │   def decorate_autocast(*args, **kwargs):                                                │
│    11 │   │   with autocast_instance:                                                            │
│ ❱  12 │   │   │   return func(*args, **kwargs)                                                   │
│    13 │   decorate_autocast.__script_unsupported = '@autocast() decorator is not supported in    │
│    14 │   return decorate_autocast                                                               │
│    15                                                                                            │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.p │
│ y:724 in forward                                                                                 │
│                                                                                                  │
│   721 │   │   down_block_res_samples = (sample,)                                                 │
│   722 │   │   for downsample_block in self.down_blocks:                                          │
│   723 │   │   │   if hasattr(downsample_block, "has_cross_attention") and downsample_block.has   │
│ ❱ 724 │   │   │   │   sample, res_samples = downsample_block(                                    │
│   725 │   │   │   │   │   hidden_states=sample,                                                  │
│   726 │   │   │   │   │   temb=emb,                                                              │
│   727 │   │   │   │   │   encoder_hidden_states=encoder_hidden_states,                           │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/torch/nn/modules/module.py:1130 in   │
│ _call_impl                                                                                       │
│                                                                                                  │
│   1127 │   │   # this function, and just call forward.                                           │
│   1128 │   │   if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks o  │
│   1129 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                   │
│ ❱ 1130 │   │   │   return forward_call(*input, **kwargs)                                         │
│   1131 │   │   # Do not call functions when jit is used                                          │
│   1132 │   │   full_backward_hooks, non_full_backward_hooks = [], []                             │
│   1133 │   │   if self._backward_hooks or _global_backward_hooks:                                │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:8 │
│ 59 in forward                                                                                    │
│                                                                                                  │
│    856 │   │   │   │   │   return custom_forward                                                 │
│    857 │   │   │   │                                                                             │
│    858 │   │   │   │   hidden_states = torch.utils.checkpoint.checkpoint(create_custom_forward(  │
│ ❱  859 │   │   │   │   hidden_states = torch.utils.checkpoint.checkpoint(                        │
│    860 │   │   │   │   │   create_custom_forward(attn, return_dict=False),                       │
│    861 │   │   │   │   │   hidden_states,                                                        │
│    862 │   │   │   │   │   encoder_hidden_states,                                                │
│                                                                                                  │
│ /data/yoke/HCP-Diffusion/hcpdiff/train_ac.py:48 in checkpoint_fix                                │
│                                                                                                  │
│    45 # fix checkpoint bug for train part of model                                               │
│    46 import torch.utils.checkpoint                                                              │
│    47 def checkpoint_fix(function, *args, use_reentrant: bool = False, checkpoint_raw = torch.   │
│ ❱  48 │   return checkpoint_raw(function, *args, use_reentrant=use_reentrant, **kwargs)          │
│    49 torch.utils.checkpoint.checkpoint = checkpoint_fix                                         │
│    50                                                                                            │
│    51 class Trainer:                                                                             │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/torch/utils/checkpoint.py:237 in     │
│ checkpoint                                                                                       │
│                                                                                                  │
│   234 │   if use_reentrant:                                                                      │
│   235 │   │   return CheckpointFunction.apply(function, preserve, *args)                         │
│   236 │   else:                                                                                  │
│ ❱ 237 │   │   return _checkpoint_without_reentrant(                                              │
│   238 │   │   │   function,                                                                      │
│   239 │   │   │   preserve,                                                                      │
│   240 │   │   │   *args                                                                          │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/torch/utils/checkpoint.py:383 in     │
│ _checkpoint_without_reentrant                                                                    │
│                                                                                                  │
│   380 │   │   return storage.pop(x)                                                              │
│   381 │                                                                                          │
│   382 │   with torch.autograd.graph.saved_tensors_hooks(pack, unpack):                           │
│ ❱ 383 │   │   output = function(*args)                                                           │
│   384 │   │   if torch.cuda._initialized and preserve_rng_state and not had_cuda_in_fwd:         │
│   385 │   │   │   # Cuda was not initialized before running the forward, so we didn't            │
│   386 │   │   │   # stash the CUDA state.                                                        │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:8 │
│ 52 in custom_forward                                                                             │
│                                                                                                  │
│    849 │   │   │   │   def create_custom_forward(module, return_dict=None):                      │
│    850 │   │   │   │   │   def custom_forward(*inputs):                                          │
│    851 │   │   │   │   │   │   if return_dict is not None:                                       │
│ ❱  852 │   │   │   │   │   │   │   return module(*inputs, return_dict=return_dict)               │
│    853 │   │   │   │   │   │   else:                                                             │
│    854 │   │   │   │   │   │   │   return module(*inputs)                                        │
│    855                                                                                           │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/torch/nn/modules/module.py:1130 in   │
│ _call_impl                                                                                       │
│                                                                                                  │
│   1127 │   │   # this function, and just call forward.                                           │
│   1128 │   │   if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks o  │
│   1129 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                   │
│ ❱ 1130 │   │   │   return forward_call(*input, **kwargs)                                         │
│   1131 │   │   # Do not call functions when jit is used                                          │
│   1132 │   │   full_backward_hooks, non_full_backward_hooks = [], []                             │
│   1133 │   │   if self._backward_hooks or _global_backward_hooks:                                │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/diffusers/models/transformer_2d.py:2 │
│ 65 in forward                                                                                    │
│                                                                                                  │
│   262 │   │                                                                                      │
│   263 │   │   # 2. Blocks                                                                        │
│   264 │   │   for block in self.transformer_blocks:                                              │
│ ❱ 265 │   │   │   hidden_states = block(                                                         │
│   266 │   │   │   │   hidden_states,                                                             │
│   267 │   │   │   │   encoder_hidden_states=encoder_hidden_states,                               │
│   268 │   │   │   │   timestep=timestep,                                                         │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/torch/nn/modules/module.py:1130 in   │
│ _call_impl                                                                                       │
│                                                                                                  │
│   1127 │   │   # this function, and just call forward.                                           │
│   1128 │   │   if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks o  │
│   1129 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                   │
│ ❱ 1130 │   │   │   return forward_call(*input, **kwargs)                                         │
│   1131 │   │   # Do not call functions when jit is used                                          │
│   1132 │   │   full_backward_hooks, non_full_backward_hooks = [], []                             │
│   1133 │   │   if self._backward_hooks or _global_backward_hooks:                                │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/diffusers/models/attention.py:331 in │
│ forward                                                                                          │
│                                                                                                  │
│   328 │   │   │   # TODO (Birch-San): Here we should prepare the encoder_attention mask correc   │
│   329 │   │   │   # prepare attention mask here                                                  │
│   330 │   │   │                                                                                  │
│ ❱ 331 │   │   │   attn_output = self.attn2(                                                      │
│   332 │   │   │   │   norm_hidden_states,                                                        │
│   333 │   │   │   │   encoder_hidden_states=encoder_hidden_states,                               │
│   334 │   │   │   │   attention_mask=encoder_attention_mask,                                     │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/torch/nn/modules/module.py:1130 in   │
│ _call_impl                                                                                       │
│                                                                                                  │
│   1127 │   │   # this function, and just call forward.                                           │
│   1128 │   │   if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks o  │
│   1129 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                   │
│ ❱ 1130 │   │   │   return forward_call(*input, **kwargs)                                         │
│   1131 │   │   # Do not call functions when jit is used                                          │
│   1132 │   │   full_backward_hooks, non_full_backward_hooks = [], []                             │
│   1133 │   │   if self._backward_hooks or _global_backward_hooks:                                │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/diffusers/models/attention_processor │
│ .py:267 in forward                                                                               │
│                                                                                                  │
│    264 │   │   # The `Attention` class can call different attention processors / attention func  │
│    265 │   │   # here we simply pass along all tensors to the selected processor class           │
│    266 │   │   # For standard processors that are defined here, `**cross_attention_kwargs` is e  │
│ ❱  267 │   │   return self.processor(                                                            │
│    268 │   │   │   self,                                                                         │
│    269 │   │   │   hidden_states,                                                                │
│    270 │   │   │   encoder_hidden_states=encoder_hidden_states,                                  │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/diffusers/models/attention_processor │
│ .py:696 in __call__                                                                              │
│                                                                                                  │
│    693 │   │   key = attn.head_to_batch_dim(key).contiguous()                                    │
│    694 │   │   value = attn.head_to_batch_dim(value).contiguous()                                │
│    695 │   │                                                                                     │
│ ❱  696 │   │   hidden_states = xformers.ops.memory_efficient_attention(                          │
│    697 │   │   │   query, key, value, attn_bias=attention_mask, op=self.attention_op, scale=att  │
│    698 │   │   )                                                                                 │
│    699 │   │   hidden_states = hidden_states.to(query.dtype)                                     │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/xformers/ops/fmha/__init__.py:192 in │
│ memory_efficient_attention                                                                       │
│                                                                                                  │
│   189 │   │   and options.                                                                       │
│   190 │   :return: multi-head attention Tensor with shape ``[B, Mq, H, Kv]``                     │
│   191 │   """                                                                                    │
│ ❱ 192 │   return _memory_efficient_attention(                                                    │
│   193 │   │   Inputs(                                                                            │
│   194 │   │   │   query=query, key=key, value=value, p=p, attn_bias=attn_bias, scale=scale       │
│   195 │   │   ),                                                                                 │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/xformers/ops/fmha/__init__.py:295 in │
│ _memory_efficient_attention                                                                      │
│                                                                                                  │
│   292 │   │   )                                                                                  │
│   293 │                                                                                          │
│   294 │   output_shape = inp.normalize_bmhk()                                                    │
│ ❱ 295 │   return _fMHA.apply(                                                                    │
│   296 │   │   op, inp.query, inp.key, inp.value, inp.attn_bias, inp.p, inp.scale                 │
│   297 │   ).reshape(output_shape)                                                                │
│   298                                                                                            │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/xformers/ops/fmha/__init__.py:41 in  │
│ forward                                                                                          │
│                                                                                                  │
│    38 │   │   op_fw = op[0] if op is not None else None                                          │
│    39 │   │   op_bw = op[1] if op is not None else None                                          │
│    40 │   │                                                                                      │
│ ❱  41 │   │   out, op_ctx = _memory_efficient_attention_forward_requires_grad(                   │
│    42 │   │   │   inp=inp, op=op_fw                                                              │
│    43 │   │   )                                                                                  │
│    44                                                                                            │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/xformers/ops/fmha/__init__.py:323 in │
│ _memory_efficient_attention_forward_requires_grad                                                │
│                                                                                                  │
│   320 │   │   op = _dispatch_fw(inp)                                                             │
│   321 │   else:                                                                                  │
│   322 │   │   _ensure_op_supports_or_raise(ValueError, "memory_efficient_attention", op, inp)    │
│ ❱ 323 │   out = op.apply(inp, needs_gradient=True)                                               │
│   324 │   assert out[1] is not None                                                              │
│   325 │   return (out[0].reshape(output_shape), out[1])                                          │
│   326                                                                                            │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/xformers/ops/fmha/cutlass.py:175 in  │
│ apply                                                                                            │
│                                                                                                  │
│   172 │   │   if type(inp.attn_bias) not in FwOp.SUPPORTED_ATTN_BIAS_TYPES:                      │
│   173 │   │   │   raise NotImplementedError("Unsupported attn_bias type")                        │
│   174 │   │   seqstart_k, seqstart_q, max_seqlen_q, _ = _get_seqlen_info(inp)                    │
│ ❱ 175 │   │   out, lse, rng_seed, rng_offset = cls.OPERATOR(                                     │
│   176 │   │   │   query=inp.query,                                                               │
│   177 │   │   │   key=inp.key,                                                                   │
│   178 │   │   │   value=inp.value,                                                               │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/torch/_ops.py:143 in __call__        │
│                                                                                                  │
│   140 │   │   # is still callable from JIT                                                       │
│   141 │   │   # We save the function ptr as the `op` attribute on                                │
│   142 │   │   # OpOverloadPacket to access it here.                                              │
│ ❱ 143 │   │   return self._op(*args, **kwargs or {})                                             │
│   144 │                                                                                          │
│   145 │   # TODO: use this to make a __dir__                                                     │
│   146 │   def overloads(self):                                                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Expected query.size(0) == key.size(0) to be true, but got false.  (Could this error message be improved?  If so, please
report an enhancement request to PyTorch.)
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /data/yoke/anaconda3/envs/hcpd/bin/accelerate:8 in <module>                                      │
│                                                                                                  │
│   5 from accelerate.commands.accelerate_cli import main                                          │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(main())                                                                         │
│   9                                                                                              │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.p │
│ y:45 in main                                                                                     │
│                                                                                                  │
│   42 │   │   exit(1)                                                                             │
│   43 │                                                                                           │
│   44 │   # Run                                                                                   │
│ ❱ 45 │   args.func(args)                                                                         │
│   46                                                                                             │
│   47                                                                                             │
│   48 if __name__ == "__main__":                                                                  │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/accelerate/commands/launch.py:918 in │
│ launch_command                                                                                   │
│                                                                                                  │
│   915 │   elif defaults is not None and defaults.compute_environment == ComputeEnvironment.AMA   │
│   916 │   │   sagemaker_launcher(defaults, args)                                                 │
│   917 │   else:                                                                                  │
│ ❱ 918 │   │   simple_launcher(args)                                                              │
│   919                                                                                            │
│   920                                                                                            │
│   921 def main():                                                                                │
│                                                                                                  │
│ /data/yoke/anaconda3/envs/hcpd/lib/python3.10/site-packages/accelerate/commands/launch.py:580 in │
│ simple_launcher                                                                                  │
│                                                                                                  │
│   577 │   process.wait()                                                                         │
│   578 │   if process.returncode != 0:                                                            │
│   579 │   │   if not args.quiet:                                                                 │
│ ❱ 580 │   │   │   raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)    │
│   581 │   │   else:                                                                              │
│   582 │   │   │   sys.exit(1)                                                                    │
│   583                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CalledProcessError: Command '['/data/yoke/anaconda3/envs/hcpd/bin/python', '-m', 'hcpdiff.train_ac_single', '--cfg',
'cfgs/train/examples/DreamArtist++2.yaml']' returned non-zero exit status 1.

      unet:
        -
          lr: 1e-6
          layers:
            - '' # fine-tuning all layers in unet
      
      ## fine-tuning text-encoder
      text_encoder:
       - lr: 1e-6
         layers:
           - ''
      
      tokenizer_pt:
        train: null
      
      train:
        gradient_accumulation_steps: 1
        save_step: 100
      
        scheduler:
          name: 'constant_with_warmup'
          num_warmup_steps: 50
          num_training_steps: 600
      
      model:
        pretrained_model_name_or_path: '/home/jovyan/data-vol-polefs-1/sd-webui/model/stable-diffusion-v1-5'
        tokenizer_repeats: 1
        ema_unet: 0
        ema_text_encoder: 0
      
      data:
        batch_size: 4
        prompt_template: 'prompt_tuning_template/object.txt'
        caption_file: null
        cache_latents: True
        tag_transforms:
          transforms:
            - _target_: hcpdiff.utils.caption_tools.TagShuffle
            - _target_: hcpdiff.utils.caption_tools.TagDropout
              p: 0.1
            - _target_: hcpdiff.utils.caption_tools.TemplateFill
              word_names: {}
        bucket:
          _target_: hcpdiff.data.bucket.RatioBucket.from_files # aspect ratio bucket
          img_root: '/home/jovyan/data-vol-polefs-1/sd-webui/Data/GOODONES/'
          target_area: {_target_: "builtins.eval", _args_: ['512*512']}
          num_bucket: 10
      
      data_class: null

      [E ProcessGroupNCCL.cpp:821] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=106, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1801455 milliseconds before timing out.
      [E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
      [E ProcessGroupNCCL.cpp:461] To avoid data inconsistency, we are taking the entire process down.
      terminate called after throwing an instance of 'std::runtime_error'
        what():  [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=106, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1801455 milliseconds before timing out.
      WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102175 closing signal SIGTERM
      WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102176 closing signal SIGTERM
      WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102177 closing signal SIGTERM
      WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102178 closing signal SIGTERM
      WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102179 closing signal SIGTERM
      WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102180 closing signal SIGTERM
      WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102181 closing signal SIGTERM
      ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 102174) of binary: /home/jovyan/miniconda3/envs/sd-webui2/bin/python
      Traceback (most recent call last):
        File "/home/jovyan/miniconda3/envs/sd-webui2/bin/torchrun", line 8, in <module>
          sys.exit(main())
        File "/home/jovyan/miniconda3/envs/sd-webui2/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
          return f(*args, **kwargs)
        File "/home/jovyan/miniconda3/envs/sd-webui2/lib/python3.10/site-packages/torch/distributed/run.py", line 762, in main
          run(args)
        File "/home/jovyan/miniconda3/envs/sd-webui2/lib/python3.10/site-packages/torch/distributed/run.py", line 753, in run
          elastic_launch(
        File "/home/jovyan/miniconda3/envs/sd-webui2/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
          return launch_agent(self._config, self._entrypoint, list(args))
        File "/home/jovyan/miniconda3/envs/sd-webui2/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
          raise ChildFailedError(
      torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
      =======================================================
      hcpdiff.train_colo FAILED
      -------------------------------------------------------
      Failures:
        <NO_OTHER_FAILURES>
      -------------------------------------------------------

2023-04-17 16:36:29.428 | INFO     | __main__:build_data:283 - len(train_dataset): 40
100%|████████████████████████████████████████████████████████████████████████| 40/40 [00:08<00:00,  4.65it/s]
2023-04-17 16:36:41.156 | INFO     | __main__:train:355 - ***** Running training *****
2023-04-17 16:36:41.157 | INFO     | __main__:train:356 -   Num batches each epoch = 2
2023-04-17 16:36:41.157 | INFO     | __main__:train:357 -   Num Steps = 20000
2023-04-17 16:36:41.157 | INFO     | __main__:train:358 -   Instantaneous batch size per device = 1
2023-04-17 16:36:41.157 | INFO     | __main__:train:359 -   Total train batch size (w. parallel, distributed & accumulation) = 5
2023-04-17 16:36:41.157 | INFO     | __main__:train:360 -   Gradient Accumulation steps = 1
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/yabin/miniconda3/envs/hcp/lib/python3.10/runpy.py:196 in _run_module_as_main               │
│                                                                                                  │
│   193 │   main_globals = sys.modules["__main__"].__dict__                                        │
│   194 │   if alter_argv:                                                                         │
│   195 │   │   sys.argv[0] = mod_spec.origin                                                      │
│ ❱ 196 │   return _run_code(code, main_globals, None,                                             │
│   197 │   │   │   │   │    "__main__", mod_spec)                                                 │
│   198                                                                                            │
│   199 def run_module(mod_name, init_globals=None,                                                │
│                                                                                                  │
│ /home/yabin/miniconda3/envs/hcp/lib/python3.10/runpy.py:86 in _run_code                          │
│                                                                                                  │
│    83 │   │   │   │   │      __loader__ = loader,                                                │
│    84 │   │   │   │   │      __package__ = pkg_name,                                             │
│    85 │   │   │   │   │      __spec__ = mod_spec)                                                │
│ ❱  86 │   exec(code, run_globals)                                                                │
│    87 │   return run_globals                                                                     │
│    88                                                                                            │
│    89 def _run_module_code(code, init_globals=None,                                              │
│                                                                                                  │
│ /home/yabin/HCP-Diffusion/hcpdiff/train_ac.py:532 in <module>                                    │
│                                                                                                  │
│   529 │                                                                                          │
│   530 │   conf = load_config_with_cli(args.cfg, args_list=sys.argv[3:]) # skip --cfg             │
│   531 │   trainer=Trainer(conf)                                                                  │
│ ❱ 532 │   trainer.train()                                                                        │
│   533                                                                                            │
│                                                                                                  │
│ /home/yabin/HCP-Diffusion/hcpdiff/train_ac.py:370 in train                                       │
│                                                                                                  │
│   367 │   │                                                                                      │
│   368 │   │   loss_sum=0                                                                         │
│   369 │   │   for image, att_mask, prompt_ids in cycle_data(self.train_loader, arb=self.arb_is   │
│ ❱ 370 │   │   │   loss=self.train_one_step(image, att_mask, prompt_ids)                          │
│   371 │   │   │   loss_sum+=loss                                                                 │
│   372 │   │   │                                                                                  │
│   373 │   │   │   self.global_step += 1                                                          │
│                                                                                                  │
│ /home/yabin/HCP-Diffusion/hcpdiff/train_ac.py:463 in train_one_step                              │
│                                                                                                  │
│   460 │   │   │   else:                                                                          │
│   461 │   │   │   │   loss = self.get_loss(model_pred, target, att_mask)                         │
│   462 │   │   │                                                                                  │
│ ❱ 463 │   │   │   self.accelerator.backward(loss)                                                │
│   464 │   │   │                                                                                  │
│   465 │   │   │   if hasattr(self, 'optimizer'):                                                 │
│   466 │   │   │   │   if self.accelerator.sync_gradients: # fine-tuning                          │
│                                                                                                  │
│ /home/yabin/miniconda3/envs/hcp/lib/python3.10/site-packages/accelerate/accelerator.py:1681 in   │
│ backward                                                                                         │
│                                                                                                  │
│   1678 │   │   elif self.distributed_type == DistributedType.MEGATRON_LM:                        │
│   1679 │   │   │   return                                                                        │
│   1680 │   │   elif self.scaler is not None:                                                     │
│ ❱ 1681 │   │   │   self.scaler.scale(loss).backward(**kwargs)                                    │
│   1682 │   │   else:                                                                             │
│   1683 │   │   │   loss.backward(**kwargs)                                                       │
│   1684                                                                                           │
│                                                                                                  │
│ /home/yabin/miniconda3/envs/hcp/lib/python3.10/site-packages/torch/_tensor.py:487 in backward    │
│                                                                                                  │
│    484 │   │   │   │   create_graph=create_graph,                                                │
│    485 │   │   │   │   inputs=inputs,                                                            │
│    486 │   │   │   )                                                                             │
│ ❱  487 │   │   torch.autograd.backward(                                                          │
│    488 │   │   │   self, gradient, retain_graph, create_graph, inputs=inputs                     │
│    489 │   │   )                                                                                 │
│    490                                                                                           │
│                                                                                                  │
│ /home/yabin/miniconda3/envs/hcp/lib/python3.10/site-packages/torch/autograd/__init__.py:200 in   │
│ backward                                                                                         │
│                                                                                                  │
│   197 │   # The reason we repeat same the comment below is that                                  │
│   198 │   # some Python versions print out the first line of a multi-line function               │
│   199 │   # calls in the traceback and some print out the last line                              │
│ ❱ 200 │   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the bac   │
│   201 │   │   tensors, grad_tensors_, retain_graph, create_graph, inputs,                        │
│   202 │   │   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to ru   │
│   203                                                                                            │
│                                                                                                  │
│ /home/yabin/miniconda3/envs/hcp/lib/python3.10/site-packages/torch/autograd/function.py:274 in   │
│ apply                                                                                            │
│                                                                                                  │
│   271 │   │   │   │   │   │   │      "Function is not allowed. You should only implement one "   │
│   272 │   │   │   │   │   │   │      "of them.")                                                 │
│   273 │   │   user_fn = vjp_fn if vjp_fn is not Function.vjp else backward_fn                    │
│ ❱ 274 │   │   return user_fn(self, *args)                                                        │
│   275 │                                                                                          │
│   276 │   def apply_jvp(self, *args):                                                            │
│   277 │   │   # _forward_cls is defined by derived class                                         │
│                                                                                                  │
│ /home/yabin/miniconda3/envs/hcp/lib/python3.10/site-packages/torch/utils/checkpoint.py:157 in    │
│ backward                                                                                         │
│                                                                                                  │
│   154 │   │   │   raise RuntimeError(                                                            │
│   155 │   │   │   │   "none of output has requires_grad=True,"                                   │
│   156 │   │   │   │   " this checkpoint() is not necessary")                                     │
│ ❱ 157 │   │   torch.autograd.backward(outputs_with_grad, args_with_grad)                         │
│   158 │   │   grads = tuple(inp.grad if isinstance(inp, torch.Tensor) else None                  │
│   159 │   │   │   │   │     for inp in detached_inputs)                                          │
│   160                                                                                            │
│                                                                                                  │
│ /home/yabin/miniconda3/envs/hcp/lib/python3.10/site-packages/torch/autograd/__init__.py:200 in   │
│ backward                                                                                         │
│                                                                                                  │
│   197 │   # The reason we repeat same the comment below is that                                  │
│   198 │   # some Python versions print out the first line of a multi-line function               │
│   199 │   # calls in the traceback and some print out the last line                              │
│ ❱ 200 │   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the bac   │
│   201 │   │   tensors, grad_tensors_, retain_graph, create_graph, inputs,                        │
│   202 │   │   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to ru   │
│   203                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following
reasons: 1) Use of a module parameter outside the `forward` function. Please make sure model parameters are
not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a
workaround if this module graph does not change during training loop.2) Reused parameters in multiple
reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of
your model, it would result in the same set of parameters been used by different reentrant backward passes
multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in
default. You can try to use _set_static_graph() as a workaround if your module graph does not change over
iterations.
Parameter at index 598 has been marked as ready twice. This means that multiple autograd engine  hooks have
fired for this particular parameter during this iteration. You can set the environment variable
TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print parameter names for further debugging.

_base_: [cfgs/train/train_base.yaml, cfgs/train/tuning_base.yaml]

unet:
  -
    lr: 1e-6
    layers:
      - ''

text_encoder:
  - lr: 1e-6
    layers:
      - ''

lora_unet: null
lora_text_encoder: null

tokenizer_pt:
  train: null

train:
  gradient_accumulation_steps: 1
  save_step: 100

  scheduler:
    name: 'constant_with_warmup'
    num_warmup_steps: 50
    num_training_steps: 600

model:
  pretrained_model_name_or_path: 'runwayml/stable-diffusion-v1-5'
  tokenizer_repeats: 1
  ema_unet: 0
  ema_text_encoder: 0
  enable_xformers: False

data:
  batch_size: 1
  prompt_template: 'prompt_tuning_template/object.txt'
  caption_file: null
  cache_latents: True
  tag_transforms:
    transforms:
      - _target_: hcpdiff.utils.caption_tools.TagShuffle
      - _target_: hcpdiff.utils.caption_tools.TagDropout
        p: 0.1
      - _target_: hcpdiff.utils.caption_tools.TemplateFill
        word_names:
          pt1: sks
          class: dog
  bucket:
    _target_: hcpdiff.data.bucket.RatioBucket.from_files
    img_root: '/home/yabin/datasets/custom/enma_ai/'
    target_area: {_target_: "builtins.eval", _args_: ['512*512']}
    num_bucket: 1

data_class:
  null

_base_: [cfgs/train/train_base.yaml, cfgs/train/tuning_base.yaml]

unet:
  - lr: 1e-6
    layers:
      - '' # fine-tuning all layers in unet

# fine-tuning text-encoder
text_encoder:
  - lr: 1e-6
    layers:
      - ''

tokenizer_pt:
  train: null

train:
  gradient_accumulation_steps: 1
  save_step: 100

  scheduler:
    name: 'constant_with_warmup'
    num_warmup_steps: 500
    num_training_steps: 20000

model:
  pretrained_model_name_or_path: 'stabilityai/stable-diffusion-2-1'
#  pretrained_model_name_or_path: '/home/yabin/HCP-Diffusion/converted_models/realismengine'
  tokenizer_repeats: 1
  ema_unet: 0
  ema_text_encoder: 0
  enable_xformers: False

data:
  batch_size: 1
  prompt_template: 'prompt_tuning_template/object.txt'
  caption_file: null
  cache_latents: True
  tag_transforms:
    transforms:
      - _target_: hcpdiff.utils.caption_tools.TagShuffle
      - _target_: hcpdiff.utils.caption_tools.TagDropout
        p: 0.1
      - _target_: hcpdiff.utils.caption_tools.TemplateFill
        word_names: {}
  bucket:
    _target_: hcpdiff.data.bucket.RatioBucket.from_files # aspect ratio bucket
    img_root: '/home/yabin/datasets/custom/enma_ai/'
    target_area: {_target_: "builtins.eval", _args_: ['1024*1024']}
    num_bucket: 1

data_class: null

magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '\xa0'.

(venv) E:\AI\HCP-diffusion\HCP-Diffusion-webui\HCP-Diffusion>python -m hcpdiff.tools.sd2diffusers ^
More? --checkpoint_path "E:\AI\HCP-diffusion\erwin-demo\SD_to_Diffuse_Model\CKPT\meinamix_meinaV11.safetensors" ^
More? --original_config_file "E:\AI\HCP-diffusion\erwin-demo\SD_to_Diffuse_Model\SD1.5-v1-inference.yaml" ^
More? --dump_path "E:\AI\HCP-diffusion\erwin-demo\SD_to_Diffuse_Model\Diffuse_Output"
A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
wandb is not available
Traceback (most recent call last):
  File "C:\Users\erwin\.pyenv\pyenv-win\versions\3.10.9\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\erwin\.pyenv\pyenv-win\versions\3.10.9\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "E:\AI\HCP-diffusion\HCP-Diffusion-webui\HCP-Diffusion\hcpdiff\tools\sd2diffusers.py", line 385, in <module>
    convert_ckpt(args)
  File "E:\AI\HCP-diffusion\HCP-Diffusion-webui\HCP-Diffusion\hcpdiff\tools\sd2diffusers.py", line 212, in convert_ckpt
    pipe = load_sd_ckpt(
  File "C:\Users\erwin\.pyenv\pyenv-win\versions\3.10.9\lib\site-packages\diffusers\pipelines\stable_diffusion\convert_from_ckpt.py", line 1258, in download_from_original_stable_diffusion_ckpt
    checkpoint = torch.load(checkpoint_path_or_dict, map_location=device)
  File "C:\Users\erwin\.pyenv\pyenv-win\versions\3.10.9\lib\site-packages\torch\serialization.py", line 1028, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\erwin\.pyenv\pyenv-win\versions\3.10.9\lib\site-packages\torch\serialization.py", line 1246, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '\xa0'.

[Model Training Tutorial](https://github.com/7eu7d7/HCP-Diffusion/blob/main/doc/guide_train.md) -> [Model Training Tutorial](https://github.com/7eu7d7/HCP-Diffusion/blob/main/doc/en/user_guides/train.md)
[DreamArtist++ Tutorial](https://github.com/7eu7d7/HCP-Diffusion/blob/main/doc/guide_DA.md) -> [DreamArtist++ Tutorial](https://github.com/7eu7d7/HCP-Diffusion/blob/main/doc/en/tutorial/DA.md)
[Model Inference Tutorial](https://github.com/7eu7d7/HCP-Diffusion/blob/main/doc/guide_infer.md) -> [Model Inference Tutorial](https://github.com/7eu7d7/HCP-Diffusion/blob/main/doc/en/user_guides/infer.md)
[Configuration File Explanation](https://github.com/7eu7d7/HCP-Diffusion/blob/main/doc/guide_cfg.md) -> [Configuration File Explanation](https://github.com/7eu7d7/HCP-Diffusion/blob/main/doc/en/user_guides/cfg.md)
[webui Model Conversion Tutorial](https://github.com/7eu7d7/HCP-Diffusion/blob/main/doc/guide_webui_lora.md) -> [webui Model Conversion Tutorial](https://github.com/7eu7d7/HCP-Diffusion/blob/main/doc/en/user_guides/model_convert.md)

Traceback (most recent call last):
  File "/home/image1325_user/anaconda3/envs/yudongjian_23_ldm/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/image1325_user/anaconda3/envs/yudongjian_23_ldm/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/image1325_user/ssd_disk1/yudongjian_23/HCP-Diffusion-main/hcpdiff/train_ac_single.py", line 44, in <module>
    trainer = TrainerSingleCard(conf)
  File "/home/image1325_user/ssd_disk1/yudongjian_23/HCP-Diffusion-main/hcpdiff/train_ac.py", line 78, in __init__
    self.build_optimizer_scheduler()
  File "/home/image1325_user/ssd_disk1/yudongjian_23/HCP-Diffusion-main/hcpdiff/train_ac.py", line 366, in build_optimizer_scheduler
    parameters, parameters_pt = self.get_param_group_train()
  File "/home/image1325_user/ssd_disk1/yudongjian_23/HCP-Diffusion-main/hcpdiff/train_ac.py", line 356, in get_param_group_train
    word_emb = self.ex_words_emb[v.name]
KeyError: 'pt-catgirl1'

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <cell line: 16>:16                                                                            │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py:115 in decorate_context       │
│                                                                                                  │
│   112 │   @functools.wraps(func)                                                                 │
│   113 │   def decorate_context(*args, **kwargs):                                                 │
│   114 │   │   with ctx_factory():                                                                │
│ ❱ 115 │   │   │   return func(*args, **kwargs)                                                   │
│   116 │                                                                                          │
│   117 │   return decorate_context                                                                │
│   118                                                                                            │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/hcpdiff/visualizer.py:202 in vis_images                  │
│                                                                                                  │
│   199 │   │   │   │   for feeder in self.pipe.unet.input_feeder:                                 │
│   200 │   │   │   │   │   feeder(ex_input_dict)                                                  │
│   201 │   │   │                                                                                  │
│ ❱ 202 │   │   │   images = self.pipe(prompt_embeds=emb_p, negative_prompt_embeds=emb_n, **kwar   │
│   203 │   │   return images                                                                      │
│   204 │                                                                                          │
│   205 │   def save_images(self, images, root, prompt, negative_prompt='', save_cfg=True):        │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py:115 in decorate_context       │
│                                                                                                  │
│   112 │   @functools.wraps(func)                                                                 │
│   113 │   def decorate_context(*args, **kwargs):                                                 │
│   114 │   │   with ctx_factory():                                                                │
│ ❱ 115 │   │   │   return func(*args, **kwargs)                                                   │
│   116 │                                                                                          │
│   117 │   return decorate_context                                                                │
│   118                                                                                            │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_dif │
│ fusion.py:755 in __call__                                                                        │
│                                                                                                  │
│   752 │   │   │   │   │   │   callback(i, t, latents)                                            │
│   753 │   │                                                                                      │
│   754 │   │   if not output_type == "latent":                                                    │
│ ❱ 755 │   │   │   image = self.vae.decode(latents / self.vae.config.scaling_factor, return_dic   │
│   756 │   │   │   image, has_nsfw_concept = self.run_safety_checker(image, device, prompt_embe   │
│   757 │   │   else:                                                                              │
│   758 │   │   │   image = latents                                                                │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/diffusers/utils/accelerate_utils.py:46 in wrapper        │
│                                                                                                  │
│   43 │   def wrapper(self, *args, **kwargs):                                                     │
│   44 │   │   if hasattr(self, "_hf_hook") and hasattr(self._hf_hook, "pre_forward"):             │
│   45 │   │   │   self._hf_hook.pre_forward(self)                                                 │
│ ❱ 46 │   │   return method(self, *args, **kwargs)                                                │
│   47 │                                                                                           │
│   48 │   return wrapper                                                                          │
│   49                                                                                             │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/diffusers/models/autoencoder_kl.py:191 in decode         │
│                                                                                                  │
│   188 │   │   │   decoded_slices = [self._decode(z_slice).sample for z_slice in z.split(1)]      │
│   189 │   │   │   decoded = torch.cat(decoded_slices)                                            │
│   190 │   │   else:                                                                              │
│ ❱ 191 │   │   │   decoded = self._decode(z).sample                                               │
│   192 │   │                                                                                      │
│   193 │   │   if not return_dict:                                                                │
│   194 │   │   │   return (decoded,)                                                              │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/diffusers/models/autoencoder_kl.py:177 in _decode        │
│                                                                                                  │
│   174 │   │   if self.use_tiling and (z.shape[-1] > self.tile_latent_min_size or z.shape[-2] >   │
│   175 │   │   │   return self.tiled_decode(z, return_dict=return_dict)                           │
│   176 │   │                                                                                      │
│ ❱ 177 │   │   z = self.post_quant_conv(z)                                                        │
│   178 │   │   dec = self.decoder(z)                                                              │
│   179 │   │                                                                                      │
│   180 │   │   if not return_dict:                                                                │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1501 in _call_impl            │
│                                                                                                  │
│   1498 │   │   if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks   │
│   1499 │   │   │   │   or _global_backward_pre_hooks or _global_backward_hooks                   │
│   1500 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                   │
│ ❱ 1501 │   │   │   return forward_call(*args, **kwargs)                                          │
│   1502 │   │   # Do not call functions when jit is used                                          │
│   1503 │   │   full_backward_hooks, non_full_backward_hooks = [], []                             │
│   1504 │   │   backward_pre_hooks = []                                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py:463 in forward                  │
│                                                                                                  │
│    460 │   │   │   │   │   │   self.padding, self.dilation, self.groups)                         │
│    461 │                                                                                         │
│    462 │   def forward(self, input: Tensor) -> Tensor:                                           │
│ ❱  463 │   │   return self._conv_forward(input, self.weight, self.bias)                          │
│    464                                                                                           │
│    465 class Conv3d(_ConvNd):                                                                    │
│    466 │   __doc__ = r"""Applies a 3D convolution over an input signal composed of several inpu  │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py:459 in _conv_forward            │
│                                                                                                  │
│    456 │   │   │   return F.conv2d(F.pad(input, self._reversed_padding_repeated_twice, mode=sel  │
│    457 │   │   │   │   │   │   │   weight, bias, self.stride,                                    │
│    458 │   │   │   │   │   │   │   _pair(0), self.dilation, self.groups)                         │
│ ❱  459 │   │   return F.conv2d(input, weight, bias, self.stride,                                 │
│    460 │   │   │   │   │   │   self.padding, self.dilation, self.groups)                         │
│    461 │                                                                                         │
│    462 │   def forward(self, input: Tensor) -> Tensor:                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.HalfTensor) should be the same

pretrained_model: /home/yabin/HCP-Diffusion/converted_models/Acertain
prompt: enma_ai, 1girl, tree, solo, plant, bush, outdoors, grass, garden, nature,
  branch, day, sky,
neg_prompt: lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit,
  fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts,
  signature, watermark, username, blurry
out_dir: output/
emb_dir: embs/
N_repeats: 1
bs: 4
num: 1
seed: null
fp16: true
clip_skip: 2
save:
  save_cfg: true
  image_type: png
  quality: 95
infer_args:
  width: 512
  height: 768
  guidance_scale: 7.5
new_components: {}
merge:
  exp_dir: 2023-04-17-21-59-53
  alpha: 0.8
  group1:
    type: unet
    base_model_alpha: 1.0
    lora:
    - path: exps/${....exp_dir}/ckpts/unet-2000.safetensors
      alpha: ${....alpha}
      layers: all
      mask: null
    part: null
  group2:
    type: TE
    base_model_alpha: 1.0
    lora:
    - path: exps/${....exp_dir}/ckpts/text_encoder-2000.safetensors
      alpha: ${....alpha}
      layers: all
      mask: null
    part: null

Traceback (most recent call last):
  File "/root/git/hcp_proj/env/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/git/hcp_proj/env/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/root/git/hcp_proj/HCP-Diffusion/hcpdiff/tools/sd2diffusers.py", line 372, in <module>
    convert_ckpt(args)
  File "/root/git/hcp_proj/HCP-Diffusion/hcpdiff/tools/sd2diffusers.py", line 210, in convert_ckpt
    pipe = load_sd_ckpt(
TypeError: download_from_original_stable_diffusion_ckpt() got an unexpected keyword argument 'checkpoint_path'