zeleni9 / pytorch-wgan Goto Github PK

THANK YOU!!!

2022-04-13 16:47:09.310913: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 2: out of memory

how to run the code in os terminal?

python main.py --model WGAN-GP
--is_train True
--download True
--dataroot datasets/cifar
--dataset cifar
--generator_iters 40000
--cuda True
--batch_size 64
I tried many times. thanks in advance

Would you provide some reference for what you did here? I have never passed an argument to loss.backward() and why do you pass one and mone.

In WGAN，the task of discriminator is to predict the Wasserstein-Distance of real_data and fake_data. In the paper, the formula is maximizing Ereal[f(x)] - Efake[f(x)]，in the code it should be errD = err_fake - err_real and calculate the gradient descent parameter update for errD. But I don't see this in the code， just err_real.backward() and err_fake.backward()

Hello,

Thanks for posting your code, I'm finding it hard to find a WGAN-GP implementation in pytorch.

I've tried your implementation work but I'm completely stuck. I did get past the error below but got hit with many others, so I think my fix was wrong.

I'm running this command
main.py --model WGAN-GP --is_train True --download True --dataroot datasets/fashion-mnist --dataset fashion-mnist --epochs 30 --cuda True --batch_size 64

Traceback (most recent call last):
File "C:/Users/Guillaume/PycharmProjects/pytorch-wgan2/main.py", line 41, in
main(args)
File "C:/Users/Guillaume/PycharmProjects/pytorch-wgan2/main.py", line 30, in main
model.train(train_loader)
File "C:\Users\Guillaume\PycharmProjects\pytorch-wgan2\models\wgan_gradient_penalty.py", line 170, in train
d_loss_real.backward(mone)
File "C:\Program Files\Python36\lib\site-packages\torch\tensor.py", line 93, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "C:\Program Files\Python36\lib\site-packages\torch\autograd_init_.py", line 90, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: invalid gradient at index 0 - expected shape [] but got [1]

Basically, I think it's some backward incompatibility with newer versions of pytorch (I'm using 0.4.1).

Do you have any idea how to fix? I'd be very grateful.

Hello,

Thanks for your great implementation!

Is there some tricks for setting the weight of gp? I noticed that the original paper adopted weight of 10, but when I used the same weight on my own data(super-resolution task), it was hard for critic to distinguish real/fake data, that means d_real_loss and d_fake_loss are nearly the same. After I lower the weight of gp, the training process seems have a better behavior.

So, the weight of gradient penalty can only be set empirically? Do you have some good experience or tricks for it?

Thanks in advance!

Hi, Zeleni9, I have a problem with your code, can you check https://stackoverflow.com/questions/69302763/difference-between-wgan-and-wgan-gp-gradient-penalty when you are free, thx a lot!

赶紧学习一哈

Hello, great job that helps me a lot.
I wonder why did you use alpha+=alpha in all models to increase alpha, why double alpha each time instead of using linear increasing?

hi, bro, I am new to GAN so sorry about the bother, when I train the WGAN-GP, the results seem quite grey. You may check the detail here when you are free, thx a lot!

Seems it's caused by too big dimensionality, I will check it when I am free. I just close it now.

I'm sorry I am a newer for WGAN, I have trouble understanding the code below. I think it should be d_loss.backward() directly.

Thank you for your answer :)

https://github.com/Zeleni9/pytorch-wgan/blob/master/models/wgan_gradient_penalty.py#L324

grad_penalty = ((gradients.norm(2, dim=1) - 1) ** 2).mean() * self.lambda_term

The shape of gradients is [batch_size, 1, 32, 32], the above norm(2, dim=1) is actually not behaving as you may want (and returns a tensor with shape [batch_size, 32, 32]). This is actually enforcing gradients to be closer to 1 element-wise. So the gradient and the lipschitz are much bigger than 1.

A potential fix is
grad_penalty = (((gradients.view(gradients.shape[0], -1) ** 2).sum(dim=1).sqrt() - 1) ** 2).mean() * self.lambda_term

Correct me if I am wrong. Thx.

Traceback (most recent call last):
File "E:/_BACKUP/lbc/pytorch-wgan-master/main.py", line 45, in
main(args)
File "E:/BACKUP/lbc/pytorch-wgan-master/main.py", line 33, in main
model.train(train_loader)
File "E:_BACKUP\lbc\pytorch-wgan-master\models\dcgan.py", line 226, in train
z = Variable(torch.randn(self.batch_size, 100, 1, 1).cuda(self.cuda_index))
File "E:\anaconda\envs\pytorch-wgan-master\lib\site-packages\torch\cuda_init.py", line 164, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

Process finished with exit code 1

To me the fact that your DCGAN is performing better then WGAN-GP is very interesting and surprising. Can you elaborate as to why this happens and what is special about the model/implementation and architecture?

First I defined my own DataLoader.

class myDataset(Dataset):

    def __init__(self, file_path):
        if not os.path.isdir(file_path):
            raise ValueError("input file_path is not a dir")
        self.file_path = file_path
        self.image_list = os.listdir(file_path)
        self.transforms = transforms.Compose([transforms.Resize((96, 96)), transforms.ToTensor()])

    def __getitem__(self, index):
        image_path = os.path.join(self.file_path, self.image_list[index])
        image = self._read_convert_image(image_path)
        return image

    def _read_convert_image(self, image_name):
        image = Image.open(image_name)
        image = self.transforms(image).float()
        return image

    def __len__(self):
        return len(self.image_list)

dataset = myDataset('../data/fing/Real')
dataloader = DataLoader(dataset, batch_size=opt.batch_size, shuffle=True)

Then I modified the real_images and generate_img functions in the code.

def real_images(self, images, number_of_images):
      if (self.C == 3):
          return self.to_np(images.view(-1, self.C, 96, 96)[:self.number_of_images])
      else:
          return self.to_np(images.view(-1, 96, 96)[:self.number_of_images])

  def generate_img(self, z, number_of_images):
      samples = self.G(z).data.cpu().numpy()[:number_of_images]
      generated_images = []
      for sample in samples:
          if self.C == 3:
              generated_images.append(sample.reshape(self.C, 96, 96))
          else:
              generated_images.append(sample.reshape(96, 96))
      return generated_images

Then, the train function is called.

model.train(dataloader)

And then it says something wrong.

It shows that:

File "<ipython-input-27-9292b3c8f357>", line 389, in <module> model.train(dataloader)
File "<ipython-input-27-9292b3c8f357>", line 231, in train d_loss_real = d_loss_real.mean(0).view(1)
RuntimeError: shape '[1]' is invalid for input of size 81

However, if I transform the images in the data set to 32*32, there will be no error

How do I fix this bug? Thank you!

I noticed that in the implementation of WGAN with gradient penalty, the weights are not initialized from a normal distribution as is detailed here - https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html#weight-initialization

The same initialization is also done in the original WGAN implementation - https://github.com/martinarjovsky/WassersteinGAN/blob/f7a01e82007ea408647c451b9e1c8f1932a3db67/main.py#L108

May I know if this was done on purpose? Or was it just an unfortunate omission?

RuntimeError: Mismatch in shape: grad_output[0] has a shape of torch.Size([1]) and output[0] has a shape of torch.Size([]).

I'm trying to fix the bug.

Please correct me if I'm wrong, but it looks like you're using TensorFlow only for TensorBoard. If the same features you're using are available in https://pytorch.org/docs/stable/tensorboard.html, it may be possible to remove the TF dependency.

The version of tensorflow cause issues.

Traceback (most recent call last):
File "main.py", line 42, in
main(args)
File "main.py", line 30, in main
model.train(train_loader)
File "/data1/mxx/projects/pytorch-wgan/models/wgan_gradient_penalty.py", line 274, in train
self.logger.image_summary(tag, images, g_iter + 1)
File "/data1/mxx/projects/pytorch-wgan/utils/tensorboard_logger.py", line 35, in image_summary
img_sum = tf.Summary.Image(encoded_image_string=s.getvalue(),
AttributeError: module 'tensorflow' has no attribute 'Summary'

I wonder if the version of tensorflow is 2.5.0?

I'm sorry I am a newer for WGAN, I think it should be "d_loss .backward()" in the optimization of D instead of "d_loss_real.backward(one) " and "d_loss_fake.backward(mone) ". Wasserstein_D and g_cost don't seem to work.

https://github.com/Zeleni9/pytorch-wgan/blob/master/models/wgan_clipping.py
166
d_loss_real = d_loss_real.mean(0).view(1)
d_loss_real.backward(one)
z = self.get_torch_variable(torch.randn(self.batch_size, 100, 1, 1))
fake_images = self.G(z)
d_loss_fake = self.D(fake_images)
d_loss_fake = d_loss_fake.mean(0).view(1)
d_loss_fake.backward(mone)
d_loss = d_loss_fake - d_loss_real
Wasserstein_D = d_loss_real - d_loss_fake
self.d_optimizer.step()

191
z = self.get_torch_variable(torch.randn(self.batch_size, 100, 1, 1))
fake_images = self.G(z)
g_loss = self.D(fake_images)
g_loss = g_loss.mean().mean(0).view(1)
g_loss.backward(one)
g_cost = -g_loss
self.g_optimizer.step()
print(f'Generator iteration: {g_iter}/{self.generator_iters}, g_loss: {g_loss.data}')

Hello, bro, may I know that for DCGAN, why the output dimension of the Generator = Cx32x32 while in the comment it's C. The same problem also exists in the Discriminator, the dimensionality of the input is Cx32x32 instead of Cx64x64 mentioned in the comment. Did you skip a Fractional-Strided Conv and a Conv?

why mone=-1 was passed to backward in fake_loss while one=1 passed to backward in real_loss? for maximize(fake_loss) and minimize(real_loss)?

Hello.

I run your DCGAN, dataset is Fashion-mnist, but this error occur.

Traceback (most recent call last):
File "/content/drive/MyDrive/pytorch-wgan/main.py", line 42, in
main(args)
File "/content/drive/MyDrive/pytorch-wgan/main.py", line 30, in main
model.train(train_loader)
File "/content/drive/MyDrive/pytorch-wgan/models/dcgan.py", line 125, in train
for i, (images, _) in enumerate(train_loader):
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 517, in next
data = self._next_data()
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 557, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/content/drive/MyDrive/pytorch-wgan/utils/fashion_mnist.py", line 75, in getitem
img = self.transform(img)
File "/usr/local/lib/python3.7/dist-packages/torchvision/transforms/transforms.py", line 60, in call
img = t(img)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 889, in call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torchvision/transforms/transforms.py", line 221, in forward
return F.normalize(tensor, self.mean, self.std, self.inplace)
File "/usr/local/lib/python3.7/dist-packages/torchvision/transforms/functional.py", line 336, in normalize
tensor.sub(mean).div(std)
RuntimeError: output with shape [1, 32, 32] doesn't match the broadcast shape [3, 32, 32]

After i search your codes, in utils/data_loader.py, transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)) exists.

Fashion-mnist is gray-scale dataset, so it need to set transforms.Normalize((0.5,), (0.5)).

I thought I should write like
"D_cost = D_fake - D_real + gradient_penalty
D_cost.backward()"
but I don't know why you use backward like that.