Details about IR/RGB conversion about unit HOT 17 CLOSED

@xelmirage I haven't tried IR/RGB conversion with this implementation and do not have an yaml file to share. But I am wondering about your setting.

1> How many IR and RGB images do you have? The fundamental assumption of the work is that there exists a joint distribution and we have a reasonable amount of images to approximate the joint distribution. If these do not hold, then the algorithm may fail.

2> IR and RGB images have different number of channels and different range values. Their reconstruction losses may not be comparable. You may need to balance them by playing with the parameter.

3> How many iterations have you trained. It takes several days in my computer to start seeing some reasonable results.

Finally, it would be great if you could share some images if possible.

from unit.

@mingyuliutw Thank you for your attention.
1>At the beginning I wanted to do an exact repeat the IR/RGB conversion in the first edition of your paper, in which you mentioned that 31386 RGB images and 31191 IR images from KAIST dataset were used to train the Net. I took a wild guess that you may used set 2,5,7,10,11 to provide RGB images and set 1,3,6,9 to provide LWIR images. But set 5,10,11 are nighttime datasets, in my understanding, all the RGB images should be daytime images, so I used RGB images in set 1,2,7,8 instead of set 2,5,7,10,11.
2>I directly used the settings from synthia2cityscape.yaml, because in all the yamls provided, I think this conversion is the most similar one to IR/RGB conversion. The modified yaml is attached. The IR images are treated as RGB images with three same channels.
kaist.zip

3>The training has about 250000 iterations so far and is not showing satisfying results. The gen_total_loss, dis_true_acc_0, and dis_fake_acc_0 are shown in the figures below:

The gen_total_loss increased after 8k iterations. because I'm not experienced in training networks, I don't know whether it's normal, in the training of CelebA dataset this also happens. So far the Net doesn't have a meaningful result, here are some results:
247500

225000

210000

165000

130000

1000000

85000

65000

30000

10000

2500

Thank you again for you precious time!

from unit.

BTW, in the first version of the work, we used a context image idea. Basically, the input image is RGB+normalized y-coordinate. We also used the UNet architecture. I am not sure if this could be the difference. But these are something that I could thought of.

from unit.

can you share the trained weights for this conversion please? thank you

from unit.

Hi @mingyuliutw , I have tried the version01 of UNIT. Since the v01 branch doesn't have a yaml for TIR-to-RGB conversion either, I modified the kaist_day_and_night.yaml. The modifications are replacing the class_name for TIR image to "dataset_thermal_image" as luckily there is a class for TIR images in the dataset_image.py, and the channel for TIR images is set to 1. However, there is also no satisyfing results after a really long training. So I have following thoughts:

1. As @xjoshramos commented, is there any pre-trained weights for TIR-RGB conversion available for evaluating the potential of UNIT?
2. Does the selection of training images have effects on the results? Because KAIST dataset has 12 sets and each set also have several subsets, Is it possible for me to know what sets and subsets (such as setXX-V00X) are used in training?
   3.Is the kaist_day_and_night.yaml has the correct settings for TIR-RGB conversion? If not, which item needs to be modified?

Thank you for your attention!

Some results of training using the version01 code:
gen_00055000

gen_00112500

gen_00150000

gen_00200000

from unit.

@xelmirage

If you check the first version of the paper (I think you can find it in the arXiv website), you would see that in the implementation section, there is a paragraph about using context. Basically, we added one channel to both the RGB (3 channels to 4 channels) and thermal ( 1 channel to 2 channels) images where the additional channel is the normalized y-coordinates [0 1] of the pixels. I pointed out this in the early reply. I don't know if you noticed it. As I recall the version 1 code in the branch does not have this context image component implemented. You might need to implement the component to get better results.

@xelmirage and @xjoshramos

I do not have any pre-trained models (The true first version of the code was implemented in Caffe because PyTorch was not yet born. I reimplemented the algorithm using PyTorch and release the code for sharing.) from the first version anymore as I have shifted to my latest version and storing these models takes a lot of disk space.

from unit.

BTW, if the goal is to make a comparison to the UNIT algorithm. I would recommend that you use the Cityscape and SYNTHIA datasets as images in the two datasets have high quality.

from unit.

@mingyuliutw Is this the paper?
https://arxiv.org/pdf/1703.00848v1.pdf

from unit.

Yes

from unit.

@mingyuliutw Thank you for your reply, as you first pointed out the context image component in your former replies, I checked the version01 and founded that version01 did have context image component. The related codes exist in dataset_image.py at line 26-34, and the yaml file also has context option switched to 1. So I'm pretty sure that I used the context image component as you mentioned.

from unit.

@xelmirage
I see. I forgot I reimplemented the context part. I guess in this case you need to explore various network design choices or various GAN training tricks.

from unit.

@mingyuliutw Thank you very much for your help. Just one minor question, I am still wondering the details about the tranning dataset. Which part of KAIST images were chosen to make the 31386 IR images and 31191 RGB images... is this detail convenient to share?

from unit.

@xelmirage , did you obtain any good results of the colourization from thermal images? If yes, can you please share how to reproduce your results!

from unit.

Just to point there was a bug in the xy image creation as described in issues 39.

from unit.

I see that the issue#39 is related to the context image component which was not part of the master branch at the time this issue was created. Does it mean I do not need to revert to version_01 for the image context part, and that the master branch would work just fine?

from unit.

Sorry for the confusion. What’s I meant is the xy-image supposed to be useful for IR2RGB but it was not the case. I suspected it was due to the bug. That’s all. I will leave version one as it is.

from unit.

Hello,
I have the same concern, I need to convert IR images to RGB.
Have you achieved acceptable result? @xelmirage
I appreciate if you share your experience about this issue.
Thx

from unit.