nvlabs / munit Goto Github PK

Can you add tensorboard-monitoring for losses please ?

I think there may be a problem with the code's handling of the new_size field. I tried training a model with crop_image_height = 256, crop_image_width = 256 and new_size = 286. However, this code crashes with the following error:

File "PATH/utils.py", line 114, in __write_images image_tensor = torch.cat([images[:display_image_num] for images in image_outputs], 0) RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 284 and 286 in dimension 2 at /opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THC/generic/THCTensorMath.cu:111

Note that this error message triggers on the write_2images call in train.py--but it only crashes when writing test images (when I commented that line out, it did not trigger when writing the training images). When I re-ran the experiment with new_size = 256, training does not crash and it saves images properly.

Looking into it, it appears that the issue may be related to the parameters being supplied to the get_data_loader_folder function when called for test data, but I have not looked into it closely yet. If it matters, I ran using the folder setup, not the list. In particular, it appears that test images are resized to new_size and not cropped at all. Seeing as training is done using images of size crop_image_height x crop_image_width, shouldn't the test loader resize images to new_size and then center crop to crop_image_height x crop_image_width?

I am a new guy in DL, and I am trying to reproduce results of training on SYNTHIA-Cityscape dataset. How many images do you use as a training set? Which subset should I use to get the same results in the paper? Should I set the max iteration as 1,000,000 as in the example?

Thanks a lot!

Thank you very much for the awesome implementation. I have successfully trained MUNIT with my own data.

I am trying training with --trainer UNIT, with exactly the same dataset and yaml config. However, the following error shown up. I am guessing that either x_aba or x_a is NoneType because of some reasons that I haven't found out. It would be nice if you can provide some help. Thank you very much!

`Traceback (most recent call last):
File "train.py", line 67, in
trainer.gen_update(images_a, images_b, config)
File "/media/tai/6TB/Projects/MUNIT/trainer.py", line 282, in gen_update
self.loss_gen_cyc_x_a = self.recon_criterion(x_aba, x_a)
File "/media/tai/6TB/Projects/MUNIT/trainer.py", line 238, in recon_criterion
return torch.mean(torch.abs(input - target))
File "/home/tai/anaconda3/envs/pytorch_p3/lib/python3.6/site-packages/torch/tensor.py", line 322, in rsub
return -self + other
TypeError: add() received an invalid combination of arguments - got (NoneType), but expected one of:

• (Tensor other, Number alpha)
• (Number other, Number alpha)`

We discover a couple of issues (slower training speed and degraded output image quality) when migrating our code from pytorch 0.3 to 0.4. We are working on fixing the issues. For now, we recommend that using munit_pytorch0.3.

Hi, I have some questions about the layer normalization used on the model.
why do you use the LN on a decoder?

I guess the reason is for making training stable.
Am I right..?

Additionally,
following lots of papers concerning a 'style transfer', the effect of the IN is flattening the style of the image. (That's why some networks of yours don't take normalization. )
But I don't know the high level view of the LN. Different from the IN, LN keeps the difference between channels. It means we can get a normalized feature without a loss of style through LN.

Is it right interpretation?
I wish your insightful comments.

Thanks.

While I am running the test code:
python3 test.py --config configs/edges2shoes_folder.yaml --input inputs/edge.jpg --output_folder outputs --checkpoint models/edges2shoes.pt --a2b 1 --style inputs/shoe.jpg

I get the following error info:

Traceback (most recent call last):
File "test.py", line 48, in
trainer = MUNIT_Trainer(config)
File "/wrk/pengwei1/DONOTREMOVE/codes/MUNIT/trainer.py", line 17, in init
self.gen_a = AdaINGen(hyperparameters['input_dim_a'], hyperparameters['gen']) # auto-encoder for domain a
File "/wrk/pengwei1/DONOTREMOVE/codes/MUNIT/networks.py", line 105, in init
self.enc_content = ContentEncoder(n_downsample, n_res, input_dim, dim, 'in', activ, pad_type=pad_type)
File "/wrk/pengwei1/DONOTREMOVE/codes/MUNIT/networks.py", line 210, in init
self.model += [Conv2dBlock(input_dim, dim, 7, 1, 3, norm=norm, activation=activ, pad_type=pad_type)]
File "/wrk/pengwei1/DONOTREMOVE/codes/MUNIT/networks.py", line 308, in init
self.norm = nn.InstanceNorm2d(norm_dim, track_running_stats=True)
TypeError: init() got an unexpected keyword argument 'track_running_stats'
srun: error: g114: task 0: Exited with exit code 1
srun: Terminating job step 22792784.0

My pytorch version is: 0.3.1 CUDA8.0 installed by pip3.

Hi.
I have a question regarding the implementation of the ResBlock here. I figured out that the output of the ResBlock does not have any activation and it is input directly to the next block. I would like to confirm if it is a bug or the authors' intention. Thanks in advance!

In discriminator, style encoder and content encoder, i find 4x4 conv filters. Where this idea came from or i missed something.

Hi NVlabs.
I used edges2handbags dataset according to Github.
After 1,000,000 training Iterations, I Execute this command again, An error message displayed like this.

root@bc5b90e77883:~/model# python train.py --config configs/edges2handbags_folder.yaml
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524580938250/work/aten/src/THC/THCTensorRandom.cu line=25 error=2 : out of memory
Traceback (most recent call last):
File "train.py", line 5, in
from utils import get_all_data_loaders, prepare_sub_folder,
File "/root/model/utils.py", line 5, in
from torch.utils.serialization import load_lua
File "/root/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/init.py", line 2, in
from .read_lua_file import load_lua, T7Reader
File "/root/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 184, in
register_torch_class('Storage', make_storage_reader)
File "/root/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 181, in register_torch_class
reader_registry[cls_name] = reader_factory(cls_name)
File "/root/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 160, in make_storage_reader
element_size = python_class().element_size()
File "/root/anaconda2/lib/python2.7/site-packages/torch/cuda/init.py", line 492, in _lazy_new
_lazy_init()
File "/root/anaconda2/lib/python2.7/site-packages/torch/cuda/init.py", line 161, in _lazy_init
torch._C._cuda_init()
RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda bld/pytorch_1524580938250/work/aten/src/THC/THCTensorRandom.cu:25

I changed max-iteration 1,000,000 to 100,000 / 10,000 / 1,000 but this error is same.
Anyone know how to solve this problem? please answer to my question. Thank You!

Can you share the code that generates the youtube video?

I am trying to reproducing the results of SYNTHIA-City and set hyperparameter vgg_w=1, there is an error as below.

Exception NameError: "global name 'FileNotFoundError' is not defined" in <bound method _DataLoaderIter.del of <torch.utils.data.dataloader._DataLoaderIter object at 0x7f7bf31bbb10>> ignored
Exception NameError: "global name 'FileNotFoundError' is not defined" in <bound method _DataLoaderIter.del of <torch.utils.data.dataloader._DataLoaderIter object at 0x7f7bf31a7ed0>> ignored
Traceback (most recent call last):
File "train.py", line 66, in
trainer.gen_update(images_a, images_b, config)
File "/workspace/trainer.py", line 100, in gen_update
self.loss_gen_vgg_a = self.compute_vgg_loss(self.vgg, x_ba, x_b) if hyperparameters['vgg_w'] > 0 else 0
File "/workspace/trainer.py", line 119, in compute_vgg_loss
img_vgg = vgg_preprocess(img)
File "/workspace/utils.py", line 245, in vgg_preprocess
batch = batch.sub(Variable(mean)) # subtract mean
RuntimeError: Expected object of type torch.cuda.FloatTensor but found type torch.FloatTensor for argument #3 'other'

There is no error when I set vgg_w=0, should I set other hyperparameters?
Thank you!

hello

I'm trying to get animal face from imagenet dataset

https://github.com/mingyuliutw/UNIT/issues/27

A little more clarification is needed on the template matching algorithm described in the above issue.

1. Is template created by manually cropping the portrait and converting it to vgg conv5 feature? Or is the truncated image itself a template?

2. How was the clustering algorithm used?

thank you

code

c_b_recon, s_a_recon = self.gen_a.encode(x_ba)

-> c_a_recon, s_a_recon = self.gen_a.encode(x_ba)

I think this is right.

What do you think ?

Hi

1. If you look at configs, there is no information about animal translation.
   When will you upload it?
   Is it ok to just run as hyper-parameter like edge2shoes?

2. The paper used a common gan loss
   But you used lsgan in your code. What was the difference in the results when comparing the two?

3. How many datasets did you use for edges2shoes or edges2handbags?
   In these cases, the amount is much larger than the other datasets. But did you use it all?

Hi, thanks for this marvelous work!
I want to run your project in yosemite data, can you provide some detail information(batch size, epoch) in training process?and how long do you finish your traing process?

Thanks for publishing your beautiful code!

I have a question about the implementation of AdaptiveInstanceNorm2d.

As shown in below, the instance normalization is achieved by F.batch_norm. Should we use F.instance_norm instead?

        out = F.batch_norm(
            x_reshaped, running_mean, running_var, self.weight, self.bias,
            True, self.momentum, self.eps)```

Hi NVlabs
Can you please explain the dataset arrangement.
1). The folders name TrainA - TrainB - TestA - TestB. What type of images are we supposed to place.
2). How do we specify the style images. ?

Hi,

Could you please provide the instructions and config files for HD (like in the paper) datasets ?
Could you also please provide the translated HD images in the paper in full resolution ? So one can see the fine details/artifacts.

Thank you !!

tensorboardX package is not mentioned in USAGE.md but it's needed.

Thanks for the great code!

I'm trying to do some interpolations and latent space exploration. The slerp code from the progressive GAN repo is slightly different than your implementation.

You are not setting the low value to the previous high value in the loop. Is there a certain reason why you changed the code?

I'm not really deep into this topic, so I appreciate your answer.

should be changed to trainer = MUNIT_Trainer(config)

otherwise it causes
TypeError: init() takes 2 positional arguments but 3 were given

with the fix it works fine

Hi there!
I added os.environ['CUDA_VISIBLE_DEVICES'] = '1' to train.py, but it turned out to make no effect in select a GPU device. When running train.py, gpu:0 was always selected even if several processes have been run on gpu0 and other gpus on my server were not uesd.
So my question is How to select which GPU to run for training?

Hi NVlabs.
Please tell me!
When I want to use multiple GPUs with this code, how should I do?

Hello

I am trained MUNIT to make conversion of from 14 bit per pixel infrared data to colour images.
The training run without any particular issue.
Then I try to evaluate my model by tuning the test.py scripty but I encounter that message:

Traceback (most recent call last):
File "test.py", line 55, in
trainer.gen_a.load_state_dict(state_dict['a'])
KeyError: 'a'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "test.py", line 58, in
state_dict = pytorch03_to_pytorch04(torch.load(opts.checkpoint), opts.trainer)
File "/hdd1/prog/MUNIT/utils.py", line 372, in pytorch03_to_pytorch04
state_dict['a'] = __conversion_core(state_dict_base['a'], trainer_name)
KeyError: 'a'

If I well understand it does not find the weights ?
After the training I used the file "optimizer.pt" as model because I did not have a more explicit filename regarding the name of my dataset.

Hi guys,
I collect some data image about winter and summer, and resize to 256*256. When I train these image, I find the vgg_w parameter is hard to tune. If this value is large, the output image quality is OK, but I can only see little translation effect, the output image is almost the same as input. However, when this value is small, the output image is blurry after 1,000,000 iterations. What should I do, enlarging mlp_dim or somthing else?

Hi,
Could you share the pretrained model to the BaiDuYun Link, since I could not open the google link. Much appreciated.

Thanks for sharing the code of this amazing work with us. A quick question: In your house cats <-> big cats example, are those cat faces properly cropped and aligned before training?

Hi NVlabs.
I used the datasets "summer2winter_yosemite" according to "https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix" which is mentioned in "USAGE.md". I used "edges2shoes_folder.yaml" and the scale of images are (256, 256).
After 200000 training iterations, the result images are also so blurry.
Could you please tell me why did that happen? Is it the reason that the max_iter is still not enough?
I also see that you metioned "Yosemite summer to winter (HD). We collect a new high-resolution dataset containing 3253 summer photos and 2385 winter photos of Yosemite. The images are downsampled such that the shortest side of each image is 1024 pixels." in the paper. So could you please release the yaml file or other changes to the code so that I can do the translation to large images.
Thank you!!!

Can I get Animal Image Translation dataset?
Can not find resource url in MUNIT user manual, paper and pytorch-CycleGAN-and-pix2pix project.

Nice paper!
Could you please describe how you process the shoes using HED algorithm in more details?
I follow the instruction in here but the edge images I create are not as smooth as yours, e.g.

There are two type of image size(ut-zap50k-images,ut-zap50k-images-square) in the ut-zap50k data set, which one do you use? Or could you release the edge images you create？
Thanks in advance~

Hi
Very awesome works
I implemented MUNIT in tensorflow, very thank you share it !!!

Can you share the data you used to animal image translation ?
I am also interested in image-to-image translation, and I wish I could get a dataset.

Hi all,
After updating to pytorch 0.4.0, I try to train MUNIT on the summer2winter_yosemite dataset from Cyclegan (https://github.com/junyanz/CycleGAN) yet the results are of ill quality. I tried the initial released version of MUNIT before and it worked perfectly well. Is anyone facing the same issue after switching to pytorch 0.4.0?

Here is the snapshot after 150000 iterations:

I use the default configuration.

Thank you.

some questions while training:

1. which batch-size is better in training?
2. how to trade-off batch-size and image-size?

i found out that it's hard to increase both batch-size and image-size in 1080ti, i had to crop the image to 128*128 and the batch-size is still only 8.

Hi authors,

I failed to run the test code and it crashes at the following code:
trainer.gen_a.load_state_dict(state_dict['a']) -- (Line 44 in test.py)

The error message is:
RuntimeError: Error(s) in loading state_dict for AdaINGen:
Unexpected running stats buffer(s) "enc_content.model.0.norm.running_mean" and "enc_content.model.0.norm.running_var" for InstanceNorm2d with track_running_stats=False. If state_dict is a checkpoint saved before 0.4.0, this may be expected because InstanceNorm2d does not track running stats by default since 0.4.0. Please remove these keys from state_dict. If the running stats are actually needed, instead set track_running_stats=True in InstanceNorm2d to enable them. See the documentation of InstanceNorm2d for details.

After tracing the codes, I guess the problem comes from incorrectly load the pretrained model. Does anyone have any thoughts to solve the problem?

My system configuration:
cuda9.0, python 3.6 and/or 2.7

Thanks for your help!

Hello, I would like to design a logo for your project. I can make some ideas if you are interested? Do you have something on your mind for your logo?
Greetings
Nuno

Hi, there, i'm trying to train dataset and stucked with some learning problems

Example of 10000 iterations day 2 night (train)

Example of 10000 iterations night 2 day (train)

I think that it's unit_day2night.yaml configuration problem. Can you share yours config file?

Another option, that it's dataset's split issue.
I used 50/50 for train and test. And both divided into two equal domains A and B.

Loss, parameters, and weight of networks became Nan after some time of training MUNIT. I don't know how it happened.

I have tried to adjust some hyperparameters, but it seemed of no use. I wanna know how I can make adjust to solve this problem.

I'm not really deep into this topic, so really appreciate your answer.

Hi
code
In Decoder, you used the layer_norm

have you tried another normalization ? like instance, batch, group

Thank you for the open-sourced codes!
I would really appreciate it if you could share the configuration file (.yaml) for Cityscapes <-> Synthia experiment.
Although I guess that the most of the parameters are the same as edges2handbags.yaml, I could not guess the reasonable choice especially for new_size, crop_image_height, and crop_image_height.
Thanks again in advance.

I have trained this model in my own datasets. In the checkpoint folders some .pt files are stored. Now I would like to test with this model. The model stored this files (dis_00100000.pt and gen_00100000.pt, optimizer.pt) So which .pt file is needed to test the model? Thanks in advance.

C:\Users\caocao\Anaconda3\python.exe D:/work/gan-pix2pix/MUNIT-master/train.py --config configs/edges2shoes_folder.yaml
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\caocao\Anaconda3\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "C:\Users\caocao\Anaconda3\lib\multiprocessing\spawn.py", line 115, in _main
Traceback (most recent call last):
File "D:/work/gan-pix2pix/MUNIT-master/train.py", line 60, in
prepare(preparation_data)
File "C:\Users\caocao\Anaconda3\lib\multiprocessing\spawn.py", line 226, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\caocao\Anaconda3\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
run_name="mp_main")
File "C:\Users\caocao\Anaconda3\lib\runpy.py", line 254, in run_path
for it, (images_a, images_b) in enumerate(zip(train_loader_a, train_loader_b)):
File "C:\Users\caocao\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 451, in iter
pkg_name=pkg_name, script_name=fname)
File "C:\Users\caocao\Anaconda3\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\caocao\Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "D:\work\gan-pix2pix\MUNIT-master\train.py", line 60, in
for it, (images_a, images_b) in enumerate(zip(train_loader_a, train_loader_b)):
File "C:\Users\caocao\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 451, in iter
return _DataLoaderIter(self)return _DataLoaderIter(self)

File "C:\Users\caocao\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 239, in init
File "C:\Users\caocao\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 239, in init
w.start()
w.start() File "C:\Users\caocao\Anaconda3\lib\multiprocessing\process.py", line 105, in start

File "C:\Users\caocao\Anaconda3\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
self._popen = self._Popen(self)
File "C:\Users\caocao\Anaconda3\lib\multiprocessing\context.py", line 212, in _Popen
File "C:\Users\caocao\Anaconda3\lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)return _default_context.get_context().Process._Popen(process_obj)

File "C:\Users\caocao\Anaconda3\lib\multiprocessing\context.py", line 313, in _Popen
File "C:\Users\caocao\Anaconda3\lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "C:\Users\caocao\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 34, in init
return Popen(process_obj)
File "C:\Users\caocao\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 66, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\caocao\Anaconda3\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
reduction.dump(process_obj, to_child)
File "C:\Users\caocao\Anaconda3\lib\multiprocessing\reduction.py", line 59, in dump
_check_not_importing_main()
File "C:\Users\caocao\Anaconda3\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.
ForkingPickler(file, protocol).dump(obj)

BrokenPipeError: [Errno 32] Broken pipe

Hi,

Thank you so much for the code and paper. My dataset contains 4 domains. The dataset structure is: trainA, trainB, trainC, trainD, testA, testB, testC, testD. How can I use this code for generating images images if 4 domains are exist in the dataset? Thanks in advanced.

Your work gets surprising results and I expect to reproduce the data of LPIPS distance that you list in Table1. Given one input image, you sample 19 outputs. For every input(edges), do you calculate the LPIPS distance between the given image(edges) and corresponding 19 samples(handbags) ? After that, you sum those 19 groups of data and have a average? Is it the same to other 99 input images in your experiment ? I'm confused about this and looking forward to your reply, thank you!

Thanks for your beautiful work.
I have a question after reading the paper.
The AdaIN is used here and its' affine parameters are fixed (computed from input y ) when it was proposed in Huang's paper. But here it is learned from a MLP module.
However the affine parameters of Instance Norm are also learned from data during the training, so what the difference between IN and AdaIN in your paper? Will it be ok if we replace the AdaIN with IN and do not use the MLP ?

Nice paper! Could you please release the code for calculating CIS and IS score?

There are losts of .pt well trained model in the Google Driver like housecat2dog.pt, but I can't find the related .yaml config file in the /config directory.
Plz help!

Hey,

when following the USAGE.md guide on Ubuntu 16 LTS, it was required to specify pytorch=0.3 since 0.4 is now the latest version and the default. Also, conda install -y -c anaconda yaml doesn't work for me ("couldn't import yaml" etc.), maybe you meant pyyaml? I assume the same goes for the dockerfile.

My full installation that worked with the Testing section of USAGE.md:

conda install pytorch=0.3 torchvision -c pytorch
conda install -y -c anaconda pip
conda install -y -c anaconda pyyaml
pip install tensorboard tensorboardX

PyTorch 0.3 defaults to cuda 8.0

Could you please share the pre-trained model for Yosemite? It is not in the google drive folder.
Thanks!