get_real_stat.py error about gan-compression HOT 23 CLOSED

For the horse2zebra, we do not have aligned data. So you may need to specify the dataset mode with an additional argument --dataset_mode single. The whole command may be like this

python get_real_stat.py --dataroot database/horse2zebra/valB --output_path real_stat/horse2zebra_B.npz --direction AtoB --dataset_mode single

from gan-compression.

by aligned data you mean each input image have it's own output image?
if so, then i have an aligned data, should i use --dataset_mode aligned?
How can i prepare and train a dataset that have for each image it's own output image?

from gan-compression.

We follow the dataset structure of junyanz/pytorch-CycleGAN-and-pix2pix.

The aligned dataset means each input image have an explicit target output image just like pix2pix. So if you need to prepare it, just specify the dataset_mode as aligned. For example, for the edges2shoe-r dataset

python get_real_stat.py \
--dataroot database/edges2shoes-r \
--output_path real_stat/edges2shoes-r_B.npz \
--direction AtoB

For the unaligned dataset, which means the input image doesn't have an explicit target image, just like horse2zebra. You could use the command I mentioned above. (dataset mode single means only contains the horse images or the zebra images, since you only need to compute the FID stat for only one side)

There's also an option --dataset_mode option for training. Just make it consistent.

from gan-compression.

i will test the aligned mode after i finish from the single mode, so keep this issue open.
but mean-while, for the Select the Best Model step, by default it chooses latest_net_G.pth, whil according to the log.txt from the previous step "Once-for-all" Network Training, the fid_largest is at epoch 260 / 400.
should i set it to --restore_G_path logs/cycle_gan/horse2zebra_lite/supernet/checkpoints/260_net_G.pth?

from gan-compression.

Of course you could. In our experiment, we use the lastest checkpoint, which means the last one.

from gan-compression.

The single mode is working great, i was able to search and find a good model within 3 hours, for horse2zebra.

Now i want to train using the facades dataset which is aligned, i get the same error as before:

python get_real_stat.py --dataset_mode aligned --dataroot database/facades/ --output_path real_stat/facades_B.npz --direction AtoB
Traceback (most recent call last):
  File "get_real_stat.py", line 84, in <module>
    main(opt)
  File "get_real_stat.py", line 15, in main
    dataloader = create_dataloader(opt)
  File "/home/home/programs/level 2/gan-compression/data/__init__.py", line 45, in create_dataloader
    dataloader = CustomDatasetDataLoader(opt, verbose)
  File "/home/home/programs/level 2/gan-compression/data/__init__.py", line 97, in __init__
    self.dataset = dataset_class(opt)
  File "/home/home/programs/level 2/gan-compression/data/aligned_dataset.py", line 24, in __init__
    self.AB_paths = sorted(make_dataset(self.dir_AB))  # get image paths
  File "/home/home/programs/level 2/gan-compression/data/image_folder.py", line 45, in make_dataset
    assert os.path.isdir(dir) or os.path.islink(dir), '%s is not a valid directory' % dir
AssertionError: database/facades/val is not a valid directory

from gan-compression.

Could you please provide the directory structure of database/facades?
Does the directory database/facades/val exist? If not, you could create it with a symlink

cd database/facades/
ln -s test val

from gan-compression.

The facades dataset have valA and valB, how to solve

home@home-lnx:~/programs/level 2/gan-compression$ bash datasets/download_cyclegan_dataset.sh facades
home@home-lnx:~/programs/level 2/gan-compression$ cd database/facades/
home@home-lnx:~/programs/level 2/gan-compression/database/facades$ tree -d
.
├── testA
├── testB
├── trainA
├── trainB
├── valA -> testA
└── valB -> testB

from gan-compression.

Oh, you have downloaded the unaligned version of the dataset (since you used datasets/download_cyclegan_dataset.sh). Just download the dataset in this way

bash datasets/download_pix2pix_dataset.sh facades

from gan-compression.

So i donwloaded the pix2pix version of facades as you suggested, and i was able to create the real_stat files.

Are you sure that i can train cyclegan using aligned dataset, because i get an error:

train.py --dataroot database/facades --dataset_mode aligned --model cycle_gan --netG resnet_9blocks --log_dir logs/cycle_gan/facades/full --real_stat_A_path real_stat/facades_A.npz --real_stat_B_path real_stat/facades_B.npz
dataset [AlignedDataset] was created
The number of training images = 400
Traceback (most recent call last):
  File "train.py", line 4, in <module>
    trainer = Trainer('train')
  File "/home/home/programs/level 2/gan-compression/trainer.py", line 51, in __init__
    model = create_model(opt)  # create a model given opt.model and other options
  File "/home/home/programs/level 2/gan-compression/models/__init__.py", line 47, in create_model
    instance = model(opt)
  File "/home/home/programs/level 2/gan-compression/models/cycle_gan_model.py", line 88, in __init__
    assert opt.dataset_mode == 'unaligned'
AssertionError

So i decided to change unaligned to aligned in cycle_gan_model.py, i am currently training and will report back later

from gan-compression.

I do not think you could change unaligned to aligned in cycle_gan_model.py. If you did that, you would need to modify many codes regarding to data preprocess.

The best way to launch the experiment is as follows:

• First, you download the cycleGAN version of the dataset with

bash datasets/download_cyclegan_dataset.sh facades

• Compute the FID statistics with single mode like:

python get_real_stat.py --dataroot database/facades/valB --output_path real_stat/facades_B.npz --direction AtoB --dataset_mode single
python get_real_stat.py --dataroot database/facades/valA --output_path real_stat/facades_A.npz --direction BtoA --dataset_mode single

• Launch your experiment like:

train.py --dataroot database/facades --dataset_mode unaligned --model cycle_gan --netG resnet_9blocks --log_dir logs/cycle_gan/facades/full --real_stat_A_path real_stat/facades_A.npz --real_stat_B_path real_stat/facades_B.npz

from gan-compression.

My dataset have for each input image a corresponding output image, so it's an exact input and it's exact output.
if i use the commands that you posted, will i be able to train using my dataset.
or should i switch to pix2pix?

from gan-compression.

Both are okay. It depends on which model are you going to compress.

from gan-compression.

• so i can't train cyclegan with aligned data?
• my dataset have the resolution of 768*768 how should i prepare, train and search using pix2pix? what options to choose?
  In my dataset, i have for each input image an explicit target output image. how can i combine the input and output images to create aligned images?

from gan-compression.

• You do can train cyclegan with aligned data, but you need modify the directory structure and preprocess like the unaligned data (like facades, the directory structures of the aligned (download with scripts/download_pix2pix_dataset.sh) and unaligned (download with scripts/download_cyclegan_dataset.sh) are different, but the content are the same).
• Yes, you could use pix2pix. Just use aligned dataset is ok. You could use combine_A_and_B.py to combine your images.

from gan-compression.

• My images are 768*768, i created the combined images resulting 1536*7681.
• Now what setting should i choose to prepare, train and search using 768 resolution?

from gan-compression.

You could use the argument '--load_size' and '--crop_size' to change the resolution during training, searching or other things.

from gan-compression.

For get_real_stat i should change also yes?

from gan-compression.

You do not have to change it in get_real_stat.py since the inception model for computing FID has already rescaled the images to a fixed size. But remember to change it while training, searching and testing.

from gan-compression.

How can i resume training? lets say i stopped at epoch 32 and iteration 100000 which is the latest

from gan-compression.

You could use --restore_G_path, --restore_D_path to specify the checkpoints you would like to resume.
By changing the your starting epoch and iteration with --epoch_base and --iter_base, you can go on training.

from gan-compression.

I will close the issue since it seems that your problem is not regarding to get_real_stat.py. If you have any other problems, feel free to open another issue.

from gan-compression.

haha, i know right,
i asked you questions on how to train from scratch
Thanks!

from gan-compression.