First, thank you for the great work, really inspiring!

To the point:
I'm trying to use EPE on my own data (Carla as source/fake domain, A set of real images as real domain).
I created fake_gbuffers, created patches, matched them, and all is working correctly.

For some reason, at iteration a little above 5000, the function clip_gradient_norm throws Error/Warning, and from that point on the reconstructed images are black, and all outputs are 0/nan.
I checked, and clip_gradient_norm results in a NAN value, hence the error.

Looking at the tensor itself, it seems that most values(weights) are indeed very close to 0.

My question is what do you think can cause this?
a few notes that might be relevant:

1. source domain is RGB, target is grayscale (I don't see why would that be a problem actually)
2. I have (currently, just as a test) 100 images from each domain. In general, I have a total of 100k images from each domain so that won't be a problem...

Thanks.

Hi,

This is an Amazing Work!! Future of gaming is here.
I have a query about about G-Buffer Encoder.
In the paper it is mentioned "The G-buffer encoder consists of multiple network streams that process the same set of G-buffers."
Does this mean different classes have different networks, one for each class?

Thanks for any help.

Hi, I'm wondering how many GPUs did you use during training phase?

Hey there,

Thanks for your work, it looks amazing.

When will the code and the checkpoints be released?

I could not find how to contact you direclty, and issues are not the place for this, but I was really curious if we could reverse the process. What I mean by that is if we used photorealistic images as the input, could convert them into much cartoony or fictional(?) ones?

Hi!

Which dimensions does the perceptual discriminator expect for the robust label map?
Three channel colored segmentation or a single channel ID mapping?

While preparing to train on my custom dataset, I stumbled upon the requirements for the robust label map.  To my understanding, the labels generated by MSeg can be replaced by any arbitrary method of labeling the images. Since I'm training on CityScapes as the target dataset, I thought about replacing the MSeg labels with the City Scapes labels, since they are also available on my synthetic dataset.

I was able to train the model on the three channel segmentation map, but the network seems to generate weird artifacts from time to time (See attached images). I assume they are caused by some error in the robust label map.  Could this be an error on my side, or did you experience similar artifacts during training?

Thank you in advance for any help & insights!

Crops from validation results at step 470000.

Hi, what type of files should robust label maps be?

Hi Guys,

Firstly love your great work, I'm fascinated by this subject . I would like to experiment with this using some of my own data and curious to see what results I can get. But task1 is to just get this working. I'm a CGI artist and Python developer but I use Windows. To get this working with all the prerequisites is Linux required? I am willing to switch over to Lunix to work on this but would like to know if that's necessary.

Thank you for your time.
Mark.

I have no affiliation with Luke Ross, I'm just a GTA player.

But the reason I suggest getting in contact with him, is because he has managed to get deep in to the GTA engine, to create a VR mod.

I read a load of information regarding how he was able to achieve this, and from my limited understanding, the things he understands about the game engine, could help you get deeper into the game's engine, allowing faster rendering with those g-buffers. (Which I believe Luke Ross has experience with, to get the VR system to work).

His GitHub name is @LukeRoss00 (ok looks like I cant tag him)... https://github.com/LukeRoss00/ (I hope I'm allowed to to that, forgive me if links aren't allowed.

But yeah, I love the video demonstration, its got me excited to see how this system can be implemented in loads of games.

Like, imagine flight simulator games, paired with a satalite imagery dataset... Slow moving gameplay allows for plenty of time to render stuff...

Truck Simulator games, where the driving style is realistic, and less busy than gta, I'd love that myself since I love just driving around in VR, with a wheel.

Please, I know it's hard, but...... Keep VR in mind.....
Too many developers are brushing off VR, but.... if this became stable, and then VR on top, that would literally be a break through in technological development.... possibly Nobel prise winning, since it really would be a milestone in realtime computer generated photorealism....

Also, can we keep multi gpu support in mind? Because I would quite happily fork out for x4 3090 cards, if the software could benefit from multiple cards... Would multiple cards be able to get that half a second render time, down to a quarter, or an 8th?

Hi,
Below is the snippet of the training log from training on a similar dataset I put together. It featues a real dataset from Houzz and an artificial dataset of GTAV buildings. Could you explain why I get a get loss update at each iteration for the generator but only occasional updates for the discriminator? I can't tell if this is intentional or if I made a mistake along the way.

I've also noticed that for the .mat files saved in /out
store ['i_fake'] and ['i_real] correctly, but ['i_rec_fake] is filled with NaN. Testing the network produces entirely black images.

Sorry, this is probably too wide to format correctly...
2022-04-19 22:53:09,737 346880 rdf0 ds0 rdf1 ds1 rdf2 ds2 rdf3 ds3 rdf4 ds4 rdf5 ds5 rdf6 ds6 rdf7 ds7 rdf8 ds8 rdf9 ds9 rdr0 rdr1 rdr2 rdr3 rdr4 rdr5 rdr6 rdr7 rdr8 rdr9 reg gs0 gs1 gs2 gs3 gs4 gs5 gs6 gs7 gs8 gs9 vgg 2022-04-19 22:53:09,737 346880 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:10,071 346881 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.02 0.99 0.99 1.04 0.99 0.99 1.03 0.98 0.97 1.05 0.72 2022-04-19 22:53:11,022 346882 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:11,034 346883 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.01 1.02 1.00 1.06 1.03 1.01 1.08 0.98 1.00 1.04 0.64 2022-04-19 22:53:11,983 346884 0.00 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:12,328 346885 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.04 1.03 0.99 1.04 1.02 1.02 1.07 0.97 0.97 1.03 0.24 2022-04-19 22:53:13,278 346886 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:13,625 346887 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.03 1.02 0.99 1.05 1.03 1.01 1.07 0.97 0.98 1.03 0.50 2022-04-19 22:53:14,576 346888 ---- ---- ---- ---- ---- ---- ---- ---- 0.00 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 ---- ---- ---- ---- ---- 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:14,588 346889 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 1.01 0.99 1.06 1.04 1.02 1.07 0.98 1.00 1.04 0.70 2022-04-19 22:53:15,540 346890 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 0.00 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 ---- ---- ---- ---- 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:15,551 346891 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.01 1.02 1.00 1.06 1.05 0.93 1.09 0.97 1.00 1.04 0.79 2022-04-19 22:53:16,497 346892 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:16,509 346893 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 0.99 1.02 1.00 1.06 1.05 0.92 1.09 0.98 1.01 1.04 0.80 2022-04-19 22:53:17,461 346894 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:17,472 346895 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 1.02 1.00 1.06 1.04 0.93 1.09 0.98 1.01 1.04 0.73 2022-04-19 22:53:18,424 346896 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:18,436 346897 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.02 1.02 1.00 1.05 1.03 0.94 1.08 0.98 1.00 1.03 0.58 2022-04-19 22:53:19,387 346898 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:19,399 346899 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 1.02 1.00 1.06 1.05 0.93 1.09 0.98 1.01 1.04 0.73

Hey guys :)
The evaluation code is missing - including the sKVD metric.
When you be able to publish it and elaborate how to use it?
Thanks!
Shahaf

Genuinely would pay money for someone to send me precompiled code

I am using my data to train the discriminator, I kept the code as it is and my data images are 960x540, I used your code to simulate the Gbuffers and your code to generate the crops and matching. When training, the data is loaded correctly and starts showing values, but soon after (about 1 hour), all values were replaced by nan, can you help me?

2022-05-08 16:27:05,927 17940 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:06,367 17941 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:07,162 17942 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:07,604 17943 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:08,380 17944 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:08,827 17945 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:09,623 17946 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:10,044 17947 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:10,852 17948 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:11,263 17949 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:12,092 17950 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:12,505 17951 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:13,318 17952 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:13,728 17953 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:14,549 17954 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:14,972 17955 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:15,841 17956 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:16,254 17957 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:17,068 17958 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:17,484 17959 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:18,317
2022-05-08 16:27:18,318 17960 rdf0 ds0  rdf1 ds1  rdf2 ds2  rdf3 ds3  rdf4 ds4  rdf5 ds5  rdf6 ds6  rdf7 ds7  rdf8 ds8  rdf9 ds9  rdr0 rdr1 rdr2 rdr3 rdr4 rdr5 rdr6 rdr7 rdr8 rdr9 reg  gs0  gs1  gs2  gs3  gs4  gs5  gs6  gs7  gs8  gs9  vgg
2022-05-08 16:27:18,319 17960 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:18,755 17961 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:19,542 17962 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:19,970 17963 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:20,772 17964 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:21,219 17965 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:22,026 17966 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:22,444 17967 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:23,269 17968 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:23,728 17969 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:24,528 17970 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:24,976 17971 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
^[[B^[[B^[[B^[[B^[[B^[[B2022-05-08 16:27:25,746 17972 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:26,180 17973 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:26,992 17974 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:27,433 17975 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:28,269 17976 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:28,729 17977 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:29,522 17978 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:29,948 17979 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan

Hi, very interesting work!
I have a question.
In epe/dataset/pfd.py, it says that number of channels G-buffers contain is {'fake':32, 'all':26, 'img':0, 'no_light':17, 'geometry':8}.
Where do these numbers come from?
I looked Playing for Data but couldn't find them.
Thank you for any help.

this is probably going way far from everyones expectations but i actually do want to try this with gtav
is there any guides that i can use to set it up?
im thinking either a mod or a filter that i put over top of everything on the screen

Hi, since my own dataset is still in process, recently I tried to train the model with pfd data using fake gbuffers extracted from vgg.
And I got the following results when achieved 4k and 8k iterations.


Is there any idea about the wrong results? Such a large difference between gbuffer and features from vgg?
By the way, I trained the model about 2 iterations per minute. I'm not sure the reason is the increase of number of channels or other things. The generator took too long to forward.
Thanks for any help

Hi,
Can you please explain me why there is a filed "max_iterations" in the test config (.yaml) file?

Hi Guys,

I'm a 3d artist, I'd love to test this out on some of my work, is there going to be a beta version released? Cheers.

Hello, I've noticed that when using the hr_new network the rec_fake (output of the generator) seems to have an unbounded value range. Compared to the loaded fake and real image which have pixels values bounded between [0, 1]

The 'hr' option uses the ResidualGenerator which uses a sigmoid on the output to bound the output between [0, 1]

However the PassthruGenerator which 'hr_new' uses doesn't have any sigmoid on the output.

Is this a bug? since it seems that the output image should be bounded [0,1] just as the real image is

Hi, thanks for the nice work! I noticed that you have mentioned the supplementary materials which contain further details on the architecture of the discriminator networks in your paper, but I cannot find them on GitHub or on the Project Website, could you please tell me where I can find them?

The README says the following:

For the ienet2.py (hr_new in config) we take inspiration from superresolution works: we remove all batch/group normalization and initialize the residual paths such they have a lower initial contribution. Further, we add 2 stages to the HRNet. This further improves final quality as well as training stability.

The training config provided in train.yaml seems to only work for hr (ienet.py) – simply changing hr to hr_new didn't seem to work for me (generated images were really bad). Would you mind submitting the config used that gave the improved final quality and training stability? Thanks!

Hi,
When I run the inference on the same test images with the following command

python epe/EPEExperiment.py test ./config/test_pfd2cs.yaml

I get always different looking images as output.

I don't change anything in the configuration (.yaml) files and I trained once before executing the inferences.
So, I train once, and then if I run several times the inference on the same images, then every time I get a different output.

Do you know why this happen? And how do I know that when I run the inference I am using the last trained model? Is it because of the field "name_save" specified in the configuration file?

Also, I tried to delete every checkpoint in the folder where checkpoints should be stored and still I can run the above command and still I get different looking outputs for the same images.
But how is it possible that I can still run the above command without any saved checkpoint?

Thanks for your amazing work,
Please let me know if you have some suggestions

Thank you for the great work!

I get some models(see pictures below) after training.

I want to test them, but I do not know what is "name_load" in test_pfd2cs.yaml.

name_load: # <name at train/name_save>-<number of iterations>

In this case should I input "pfd2cs-10000"?

As above

I followed through the whole workflow with little trouble and after setting everything up I'm getting this error upon saving the model before training starts.

2022-11-09 17:15:04,408 Saving model to savegames/pfd2cs_ie2-0-break.
2022-11-09 17:15:07,098 Unexpected error: <class 'ValueError'>
Traceback (most recent call last):
  File "code/epe/EPEExperiment.py", line 390, in <module>
    experiment.run()
  File "/mnt/b/Dev/cv-gta/code/epe/experiment/BaseExperiment.py", line 636, in run
    self.__getattribute__(self.action)()
  File "/mnt/b/Dev/cv-gta/code/epe/experiment/BaseExperiment.py", line 543, in train
    for batch in self.loader:
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
    return self._process_data(data)
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
    data.reraise()
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/_utils.py", line 543, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/mnt/b/Dev/cv-gta/code/epe/matching/paired.py", line 108, in __getitem__
    idx = np.min(np.nonzero(p<self._cumsum)[0])
  File "<__array_function__ internals>", line 180, in amin
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 2918, in amin
    return _wrapreduction(a, np.minimum, 'min', axis, None, out,
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation minimum which has no identity

Help would be appreciated.

I would love to see this tested for games like metroid prime and pikmin

Hi, my name is Luan I would like to know how I can use this AI Code to turn CGI Video or Photo to
Photorealism Enhancement with subjects like Humans and Animals with and without fur, to reach a more realistic video.Thank You Sir,Best, Luan Dal Orto Vaz.

This is amazing <3 i would like to have it as a UI software that can import videos and then turn them into realistic. I would like to be able to record my own footage from games like GTA, FLight simulator & Unreal engine demos.

DELETED COMMENT

Hello, thank you for the provided code!
Could you clarify, please, how much gpu memory the training of your model requires?
And what was the size of the synthetic dataset you used?

I'm wondering could you please provide more details on how to prepare my own dataset. Specifically, how to generate "robust label map, gbuffer file, and ground truth label map"? I also could not find "dataset/generate_fake_gbuffers.py" in your repo, so I'm stuck at this point.

At the bottom of the readme you wrote More coming soon!. Will there be a mod released? I would love to play GTA that way!

Hello,

You have done really a great work!
I really appreciate your contribution!
And, I want to know if there is any promotion in code and data sharing?

Thanks

Hello. First, congratulations on the great work and the amazing results. Will the dataset used for training be released ? If yes, do you have a time window ?

Hi, thanks for the interesting work.
I am confused about the batch size in paper. It says "We train with batch size 1 for 1M iterations".
Is there any reason not to increase the batch size? Maybe the GPU memory is the limit but I saw your cards in another issue.
And if it did 1, how long did you train your 1M iterations?
Thank you for any help.

RuntimeError: Input and output sizes should be greater than 0, but got input (H: 196, W: 0) output (H: 13, W: 13)

any idea what could be the possible reason ?

When the batch_size is set for any value >1, the following error is triggered:

" File "D:\Photorealism\PhotorealismEnhancement\code\epe\network\gb_encoder.py", line 137, in forward
features += classmap[:,c,:,:] * self.class_encoders[c](
RuntimeError: The size of tensor a (2) must match the size of tensor b (128) at non-singleton dimension 1"

The error takes place in the following line. As seen in the second screenshot, the shapes dont match. Even when I try to reshape the classmap[:,c,:,:] so that the multiplication works, an assert is thrown in the discriminators.py file as shown in the 3rd screenshot. Could anyone help in this regards?

I just want to make my gta 5 video have photorealism do I need to code stuff or what wheres the program

Thank you for the great work, really inspiring!

In readme, the only limitation of input is the matched_crops_PfD-Cityscapes.csv should contain at least 200K pairs (lines), more is better. I used 2000 images from Playing for Data dataset and 1703 images from Cityscapes dataset, and I get 230k pairs, which is enough.

My question is:
Should I use same number of images from Playing for Data dataset and Cityscapes dataset?

I hope to run a well-trained model to see the results，can you provide it?

i dont undertand about val.txt ,test.txt and train.txt ,How to create this txt file ?
Which paths do i need to change in the yaml file?

Does anyone either know an alternative to mseg or how one might get mseg to work on windows 10? I tried installing it but it is giving me an error saying that it can only be installed on linux or mac osx. Thanks!

Hello,

Thanks for the great work. Inside matching/feature_based/sample_matches.py the images are loaded by calling get_by_path method of the datasets.

However, the ImageDataset class does not have such an attribute, so this ends up with an error.
Replacing that with the _load_img method and applying ToTensor() on its output solved the issue for me.

For your notice.
Cheers!

python epe/matching/feature_based/collect_crops.py PfD pfd_files.csv

So I've been trying to setup the code for a while now, and so far so good (looks pretty well put-together by the way) and I am getting a recurring error when running pip. It is as follows: "(epe) C:\Users\johns>pip install -e ./downloads/photorealism/code
Obtaining file:///C:/Users/johns/downloads/photorealism/code
Preparing metadata (setup.py) ... done
ERROR: No .egg-info directory found in C:\Users\johns\AppData\Local\Temp\pip-pip-egg-info-fh1b71r"
Would you happen to know any possible fixes? Thanks!

Hi, is there a way to load an existing trained model (trained with this same repository) to not restart training from scratch?