Hi,

Thank you for this nice work! Is there a way that can run the code without providing the segmentation masks? Thanks!

Cheers,

In your paper you mention that 'We use the encoder-decoder architecture with fixed VGG
encoder weights' and you only train the decoder for COCO dataset. However, since you have changed the max pooling/unpooling layers from the original VGG-19 architecture to the wavelet pooling/unpooling layers, you need to train the entire encoder-decoder VGG-19 architecture right? Since these layers are changed, the VGG-19 weights will also get affected as a result of which we cannot use the original ImageNet weights. Can you please provide some clarification on this.

torch and numpy svd are different

u, e, v = torch.svd(conv, some=False)
ut, et, vt = [ x.data.cpu().numpy() for x in (u,e,v) ]
un, en, vn = np.linalg.svd ( conv.data.cpu().numpy() )
print ("diff=", np.sum(np.abs(ut-un)))
print ("diff=", np.sum(np.abs(et-en)))
print ("diff=", np.sum(np.abs(vt-vn)))

result:

diff= 10380.845
diff= 0.007629454
diff= 12652.448

scipy.linalg.svd - same wrong result

diff with tensorflow:

import tensorflow as tf
    tf_sess = tf.Session(config=tf.ConfigProto(device_count={'GPU': 0}) )
    tf_inp = tf.placeholder(tf.float32, shape=conv.shape )

    tfe, tfu, tfv = tf.linalg.svd( tf_inp, full_matrices=True ) #conv.data.cpu().numpy() )

    tfe, tfu, tfv =  tf_sess.run ([tfe, tfu, tfv], feed_dict={tf_inp: conv.data.cpu().numpy() } )
    print ("diff=", np.sum(np.abs(ut-tfu)))
    print ("diff=", np.sum(np.abs(et-tfe)))
    print ("diff=", np.sum(np.abs(vt-tfv)))

diff= 10259.318
diff= 0.03329885
diff= 10259.33

so is your net trained with torch.svd , and there is no way to use your method in other frameworks?

found similar issue pytorch/pytorch#16076
but it is closed

Is there a recommended way to perform video stylisation? For now, Im converting the original video file and style video file into frames, performing photorealistic style transfer and then making a video out of the output frames. However there seems to be problem while encoding the frames to video leading to blurry video output. Any suggestions are welcome!

Hi @jaejun-yoo , thank you for your great work!

I have some original images and their corresponding depth maps. I wanna ask that after getting the style-transfered images, could the original depth maps still match them? Namely, will the style-transfer process induces a minor depth offset?

Big thanks!

I have an indoor dataset where the goal is to take the texture of one furniture and overlay on a similar furniture in another room setup. When I directly run this code, the color gets transferred but not the exact texture. Is there any paramater I should tweak to get the desired output? Can you shed some light on this?

Hi, thank you for your great work. I have a question about the loss function in the training phase.
In session 5.1, you have mentioned:

minimizing the L2 reconstruction loss and the additional feature Gram matching loss with the encoder.

So you input to network a single image x (from MS-COCO) and calculate the loss with the output is image x' as

Loss = recontruction_loss + gram_matching_loss
with:
+ recontruction_loss = L2(x - x')
+ gram_matching_loss = gram_matrix(encoder(x)) - gram_matrix(encoder(x'))

Do I miss something? Thank you.

@kiranscaria hi, In the paper, a encoder use some pretrained layers of vgg19, but vgg19 does not have that layer conv0, so i want to know why in the model, it has conv0 layer. thank u.

Thank you for your good job ! I have a problem that how to computer the value of SSIM and Gram loss of Figure 9?

Hello. Thank you for sharing your good research.
Can I try to train with my data? If I can, please explain the command and method .
Thank you in advance.

Hi , I want to transfer color for the video data. I was wondering how to add temporal consistency constraint such as optical flow on the model.

Is there any scripts for SSIM metrics ?

Did you train the decoder with cat5 settings, using only COCO dataset? And caculated feature Gram loss between reconstructed images and input images?

I'm suprised that the training process involve no style images, only COCO dataset as content images. And the Gram loss is not between stylized images and style images. Still, the network works well for style transfer

Great tutorial! I was wondering is there a recommended method to generate the segmentation masks used for stylising?

Thanks!

On google colab it's failing to install when I do pip3 install -r requirements.txt

Could not find a version that satisfies the requirement torch==0.4.1.post2 (from -r requirements.txt (line 1)) 
(from versions: 0.1.2, 0.1.2.post1, 0.3.1, 0.4.0, 0.4.1, 1.0.0, 1.0.1, 1.0.1.post2)
No matching distribution found for torch==0.4.1.post2 (from -r requirements.txt (line 1))

Hi!

I followed your instruction step by step, and after typing

!python3 transfer.py --option_unpool cat5 -a --content ./examples/content --style ./examples/style --content_segment ./examples/content_segment --style_segment ./examples/style_segment/ --output ./outputs/ --verbose --image_size 512

I got

Namespace(alpha=1, content='./examples/content', content_segment='./examples/content_segment', cpu=False, image_size=512, option_unpool='cat5', output='./outputs/', style='./examples/style', style_segment='./examples/style_segment/', transfer_all=True, transfer_at_decoder=False, transfer_at_encoder=False, transfer_at_skip=False, verbose=True)
0% 0/1 [00:00<?, ?it/s]------ transfer: 1.png
Elapsed time in whole WCT: 0:00:02.926612

Traceback (most recent call last):
File "transfer.py", line 205, in
run_bulk(config)
File "transfer.py", line 175, in run_bulk
img = wct2.transfer(content, style, content_segment, style_segment, alpha=config.alpha)
File "transfer.py", line 79, in transfer
style_feats, style_skips = self.get_all_feature(style)
File "transfer.py", line 64, in get_all_feature
x = self.encode(x, skips, level)
File "transfer.py", line 55, in encode
return self.encoder.encode(x, skips, level)
File "/content/drive/WCT2/model.py", line 163, in encode
out = self.conv0(x)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size 3 3 1 1, expected input[1, 4, 512, 512] to have 3 channels, but got 4 channels instead

Could you help me please? Thank you!

Sincerely,

Amber
@jaejun-yoo

In section 3 of your paper, where you discuss the model architecture, there is this paragraph -

"The max-pooling layers are replaced with wavelet pooling where high frequency components (LH,HL, HH) are skipped to the decoder directly. Thus, only the low frequency component (LL) is passed to the next encoding layer"

Is there a particular reason for this?

Hello, it's so lucky for me to know you've publish your source code and training data. But I don't find the test data of video. Can you publish the video data? or, you can tell me where to find the video data.
Thank you very much!

Hi,

in the original paper equivalence between avg. pooling and the LL-component of the decomposition are claimed. However, in your code the L-component evaluates to [[0.5, 0.5], [0.5, 0.5]], which contradicts this statement. Am I missing something?

Could you release the pretrained model? We could test by ourself.

Thank you for your good job ! I have a problem that the photowct-network had not used the raw picture and it yield some error points. what it will be if we add some raw information like unet structure? Does the 'unet' idea success or the haar wavelet works?

Could you please explain the reason for using the "conv0" layer, as it is not part of the original VGG19? Also, during training, are the weights of this particular layer set as trainable or fixed?

Do you have any plan to share your training source ?

I took some photos with a SLR camera.
The size is up to 5000×5000 px.

Does it support large-size photos style transfer?

I'm trying to use the net to transfer the style of a mountain landscape photo to another.
I tried building a coarse segmentation map for the 2 photos, but the result is off (see attachment).
I would like to know if there is a way to set the net to use the segmentation map as a "hint" and not as a strict constraint (from the attachment you can clearly tell where are the bounds of the segmentation map).

Thank you.

Thank you for your great work. In the skip decoder network, I try to test just one layer with wct and find only the last layer matters. The other layers seem to contribute little to the final result. It seems the skip structure just bypass the normal flow, am I right?