Code & Data for Enhancing Photorealism Enhancement

Comments
  • Training Error

    Training Error

    First, thank you for the great work, really inspiring!

    To the point: I'm trying to use EPE on my own data (Carla as source/fake domain, A set of real images as real domain). I created fake_gbuffers, created patches, matched them, and all is working correctly.

    For some reason, at iteration a little above 5000, the function clip_gradient_norm throws Error/Warning, and from that point on the reconstructed images are black, and all outputs are 0/nan. I checked, and clip_gradient_norm results in a NAN value, hence the error.

    Looking at the tensor itself, it seems that most values(weights) are indeed very close to 0.

    My question is what do you think can cause this? a few notes that might be relevant:

    1. source domain is RGB, target is grayscale (I don't see why would that be a problem actually)
    2. I have (currently, just as a test) 100 images from each domain. In general, I have a total of 100k images from each domain so that won't be a problem...

    Thanks.

    opened by EyalMichaeli 16
  • MSeg difficulties

    MSeg difficulties

    Does anyone either know an alternative to mseg or how one might get mseg to work on windows 10? I tried installing it but it is giving me an error saying that it can only be installed on linux or mac osx. Thanks!

    opened by char119 9
  • Code and data release

    Code and data release

    Hello,

    You have done really a great work! I really appreciate your contribution! And, I want to know if there is any promotion in code and data sharing?

    Thanks

    opened by Mrsirovo 9
  • Problem in training

    Problem in training

    I am using my data to train the discriminator, I kept the code as it is and my data images are 960x540, I used your code to simulate the Gbuffers and your code to generate the crops and matching. When training, the data is loaded correctly and starts showing values, but soon after (about 1 hour), all values were replaced by nan, can you help me?

    2022-05-08 16:27:05,927 17940 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:06,367 17941 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:07,162 17942 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:07,604 17943 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:08,380 17944 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:08,827 17945 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:09,623 17946 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:10,044 17947 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:10,852 17948 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:11,263 17949 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:12,092 17950 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:12,505 17951 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:13,318 17952 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:13,728 17953 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:14,549 17954 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:14,972 17955 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:15,841 17956 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:16,254 17957 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:17,068 17958 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:17,484 17959 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:18,317
    2022-05-08 16:27:18,318 17960 rdf0 ds0  rdf1 ds1  rdf2 ds2  rdf3 ds3  rdf4 ds4  rdf5 ds5  rdf6 ds6  rdf7 ds7  rdf8 ds8  rdf9 ds9  rdr0 rdr1 rdr2 rdr3 rdr4 rdr5 rdr6 rdr7 rdr8 rdr9 reg  gs0  gs1  gs2  gs3  gs4  gs5  gs6  gs7  gs8  gs9  vgg
    2022-05-08 16:27:18,319 17960 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:18,755 17961 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:19,542 17962 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:19,970 17963 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:20,772 17964 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:21,219 17965 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:22,026 17966 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:22,444 17967 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:23,269 17968 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:23,728 17969 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:24,528 17970 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:24,976 17971 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    ^[[B^[[B^[[B^[[B^[[B^[[B2022-05-08 16:27:25,746 17972 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:26,180 17973 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:26,992 17974 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:27,433 17975 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:28,269 17976 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:28,729 17977 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    2022-05-08 16:27:29,522 17978 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
    2022-05-08 16:27:29,948 17979 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
    
    
    opened by ZouaghiHoussem 5
  • Is discriminator training properly?

    Is discriminator training properly?

    Hi, Below is the snippet of the training log from training on a similar dataset I put together. It featues a real dataset from Houzz and an artificial dataset of GTAV buildings. Could you explain why I get a get loss update at each iteration for the generator but only occasional updates for the discriminator? I can't tell if this is intentional or if I made a mistake along the way.

    I've also noticed that for the .mat files saved in /out store ['i_fake'] and ['i_real] correctly, but ['i_rec_fake] is filled with NaN. Testing the network produces entirely black images.

    Sorry, this is probably too wide to format correctly... 2022-04-19 22:53:09,737 346880 rdf0 ds0 rdf1 ds1 rdf2 ds2 rdf3 ds3 rdf4 ds4 rdf5 ds5 rdf6 ds6 rdf7 ds7 rdf8 ds8 rdf9 ds9 rdr0 rdr1 rdr2 rdr3 rdr4 rdr5 rdr6 rdr7 rdr8 rdr9 reg gs0 gs1 gs2 gs3 gs4 gs5 gs6 gs7 gs8 gs9 vgg 2022-04-19 22:53:09,737 346880 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:10,071 346881 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.02 0.99 0.99 1.04 0.99 0.99 1.03 0.98 0.97 1.05 0.72 2022-04-19 22:53:11,022 346882 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:11,034 346883 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.01 1.02 1.00 1.06 1.03 1.01 1.08 0.98 1.00 1.04 0.64 2022-04-19 22:53:11,983 346884 0.00 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:12,328 346885 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.04 1.03 0.99 1.04 1.02 1.02 1.07 0.97 0.97 1.03 0.24 2022-04-19 22:53:13,278 346886 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:13,625 346887 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.03 1.02 0.99 1.05 1.03 1.01 1.07 0.97 0.98 1.03 0.50 2022-04-19 22:53:14,576 346888 ---- ---- ---- ---- ---- ---- ---- ---- 0.00 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 ---- ---- ---- ---- ---- 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:14,588 346889 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 1.01 0.99 1.06 1.04 1.02 1.07 0.98 1.00 1.04 0.70 2022-04-19 22:53:15,540 346890 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 0.00 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 ---- ---- ---- ---- 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:15,551 346891 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.01 1.02 1.00 1.06 1.05 0.93 1.09 0.97 1.00 1.04 0.79 2022-04-19 22:53:16,497 346892 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:16,509 346893 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 0.99 1.02 1.00 1.06 1.05 0.92 1.09 0.98 1.01 1.04 0.80 2022-04-19 22:53:17,461 346894 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:17,472 346895 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 1.02 1.00 1.06 1.04 0.93 1.09 0.98 1.01 1.04 0.73 2022-04-19 22:53:18,424 346896 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:18,436 346897 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.02 1.02 1.00 1.05 1.03 0.94 1.08 0.98 1.00 1.03 0.58 2022-04-19 22:53:19,387 346898 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:19,399 346899 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 1.02 1.00 1.06 1.05 0.93 1.09 0.98 1.01 1.04 0.73

    opened by TiPEX360 4
  • Reversing the input and output

    Reversing the input and output

    I could not find how to contact you direclty, and issues are not the place for this, but I was really curious if we could reverse the process. What I mean by that is if we used photorealistic images as the input, could convert them into much cartoony or fictional(?) ones?

    opened by alpkabac 4
  • You Should Hire Luke Ross... No Idea? Read Why, Here....

    You Should Hire Luke Ross... No Idea? Read Why, Here....

    I have no affiliation with Luke Ross, I'm just a GTA player.

    But the reason I suggest getting in contact with him, is because he has managed to get deep in to the GTA engine, to create a VR mod.

    I read a load of information regarding how he was able to achieve this, and from my limited understanding, the things he understands about the game engine, could help you get deeper into the game's engine, allowing faster rendering with those g-buffers. (Which I believe Luke Ross has experience with, to get the VR system to work).

    His GitHub name is @LukeRoss00 (ok looks like I cant tag him)... https://github.com/LukeRoss00/ (I hope I'm allowed to to that, forgive me if links aren't allowed.

    But yeah, I love the video demonstration, its got me excited to see how this system can be implemented in loads of games.

    Like, imagine flight simulator games, paired with a satalite imagery dataset... Slow moving gameplay allows for plenty of time to render stuff...

    Truck Simulator games, where the driving style is realistic, and less busy than gta, I'd love that myself since I love just driving around in VR, with a wheel.

    Please, I know it's hard, but...... Keep VR in mind..... Too many developers are brushing off VR, but.... if this became stable, and then VR on top, that would literally be a break through in technological development.... possibly Nobel prise winning, since it really would be a milestone in realtime computer generated photorealism....

    Also, can we keep multi gpu support in mind? Because I would quite happily fork out for x4 3090 cards, if the software could benefit from multiple cards... Would multiple cards be able to get that half a second render time, down to a quarter, or an 8th?

    opened by dailafing 4
  • Number of image channels the provided G-buffers contain

    Number of image channels the provided G-buffers contain

    Hi, very interesting work! I have a question. In epe/dataset/pfd.py, it says that number of channels G-buffers contain is {'fake':32, 'all':26, 'img':0, 'no_light':17, 'geometry':8}. Where do these numbers come from? I looked Playing for Data but couldn't find them. Thank you for any help.

    opened by hodakagoto 3
  • Sigmoid missing in hr_new?

    Sigmoid missing in hr_new?

    Hello, I've noticed that when using the hr_new network the rec_fake (output of the generator) seems to have an unbounded value range. Compared to the loaded fake and real image which have pixels values bounded between [0, 1]

    https://github.com/isl-org/PhotorealismEnhancement/blob/6936e62a96fc2d913cdd66463d113f7d591d09aa/code/epe/EPEExperiment.py#L204-L209

    The 'hr' option uses the ResidualGenerator which uses a sigmoid on the output to bound the output between [0, 1] https://github.com/isl-org/PhotorealismEnhancement/blob/6936e62a96fc2d913cdd66463d113f7d591d09aa/code/epe/network/generator.py#L4-L6

    However the PassthruGenerator which 'hr_new' uses doesn't have any sigmoid on the output.

    Is this a bug? since it seems that the output image should be bounded [0,1] just as the real image is

    opened by KacperKazan 2
  • Question

    Question

    Hi, my name is Luan I would like to know how I can use this AI Code to turn CGI Video or Photo to Photorealism Enhancement with subjects like Humans and Animals with and without fur, to reach a more realistic video.Thank You Sir,Best, Luan Dal Orto Vaz.

    opened by LuanDalOrto 2
  • Getting this working is Linux required?

    Getting this working is Linux required?

    Hi Guys,

    Firstly love your great work, I'm fascinated by this subject . I would like to experiment with this using some of my own data and curious to see what results I can get. But task1 is to just get this working. I'm a CGI artist and Python developer but I use Windows. To get this working with all the prerequisites is Linux required? I am willing to switch over to Lunix to work on this but would like to know if that's necessary.

    Thank you for your time. Mark.

    opened by mrwal3 2
  • RuntimeError while training

    RuntimeError while training

    RuntimeError: Input and output sizes should be greater than 0, but got input (H: 196, W: 0) output (H: 13, W: 13)

    any idea what could be the possible reason ?

    opened by sandip824 0
  • ValueError when saving model

    ValueError when saving model

    I followed through the whole workflow with little trouble and after setting everything up I'm getting this error upon saving the model before training starts.

    2022-11-09 17:15:04,408 Saving model to savegames/pfd2cs_ie2-0-break.
    2022-11-09 17:15:07,098 Unexpected error: <class 'ValueError'>
    Traceback (most recent call last):
      File "code/epe/EPEExperiment.py", line 390, in <module>
        experiment.run()
      File "/mnt/b/Dev/cv-gta/code/epe/experiment/BaseExperiment.py", line 636, in run
        self.__getattribute__(self.action)()
      File "/mnt/b/Dev/cv-gta/code/epe/experiment/BaseExperiment.py", line 543, in train
        for batch in self.loader:
      File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
        data = self._next_data()
      File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
        return self._process_data(data)
      File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
        data.reraise()
      File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/_utils.py", line 543, in reraise
        raise exception
    ValueError: Caught ValueError in DataLoader worker process 0.
    Original Traceback (most recent call last):
      File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
        data = fetcher.fetch(index)
      File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
        data = [self.dataset[idx] for idx in possibly_batched_index]
      File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
        data = [self.dataset[idx] for idx in possibly_batched_index]
      File "/mnt/b/Dev/cv-gta/code/epe/matching/paired.py", line 108, in __getitem__
        idx = np.min(np.nonzero(p<self._cumsum)[0])
      File "<__array_function__ internals>", line 180, in amin
      File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 2918, in amin
        return _wrapreduction(a, np.minimum, 'min', axis, None, out,
      File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
        return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
    ValueError: zero-size array to reduction operation minimum which has no identity
    

    Help would be appreciated.

    opened by Chanka0 2
  • Attribute error in sample_matches.py

    Attribute error in sample_matches.py

    Hello,

    Thanks for the great work. Inside matching/feature_based/sample_matches.py the images are loaded by calling get_by_path method of the datasets.

    Screen Shot 2022-04-08 at 19 23 12

    However, the ImageDataset class does not have such an attribute, so this ends up with an error. Replacing that with the _load_img method and applying ToTensor() on its output solved the issue for me.

    For your notice. Cheers!

    opened by acerdur 2
Releases(v0.1.0)
  • v0.1.0(Jan 12, 2022)

    What's Changed

    • Initial code release. by @srrichter in https://github.com/isl-org/PhotorealismEnhancement/pull/14

    Full Changelog: https://github.com/isl-org/PhotorealismEnhancement/commits/v0.1.0

    Source code(tar.gz)
    Source code(zip)
Owner
Intelligent Systems Lab Org
Intelligent Systems Lab Org
Source code for paper "ATP: AMRize Than Parse! Enhancing AMR Parsing with PseudoAMRs" @NAACL-2022

ATP: AMRize Then Parse! Enhancing AMR Parsing with PseudoAMRs Hi this is the source code of our paper "ATP: AMRize Then Parse! Enhancing AMR Parsing w

Chen Liang 13 Nov 23, 2022
Implementation of the paper All Labels Are Not Created Equal: Enhancing Semi-supervision via Label Grouping and Co-training

SemCo The official pytorch implementation of the paper All Labels Are Not Created Equal: Enhancing Semi-supervision via Label Grouping and Co-training

null 42 Nov 14, 2022
ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration

ROSITA News & Updates (24/08/2021) Release the demo to perform fine-grained semantic alignments using the pretrained ROSITA model. (15/08/2021) Releas

Vision and Language Group@ MIL 48 Dec 23, 2022
MEDS: Enhancing Memory Error Detection for Large-Scale Applications

MEDS: Enhancing Memory Error Detection for Large-Scale Applications Prerequisites cmake and clang Build MEDS supporting compiler $ make Build Using Do

Secomp Lab at Purdue University 34 Dec 14, 2022
Enhancing Knowledge Tracing via Adversarial Training

Enhancing Knowledge Tracing via Adversarial Training This repository contains source code for the paper "Enhancing Knowledge Tracing via Adversarial T

Xiaopeng Guo 14 Oct 24, 2022
Enhancing Aspect-Based Sentiment Analysis with Supervised Contrastive Learning.

Enhancing Aspect-Based Sentiment Analysis with Supervised Contrastive Learning. Enhancing Aspect-Based Sentiment Analysis with Supervised Contrastive

HLT@HIT(SZ) 7 Dec 16, 2021
Pytorch implementation of the paper "Enhancing Content Preservation in Text Style Transfer Using Reverse Attention and Conditional Layer Normalization"

Pytorch implementation of the paper "Enhancing Content Preservation in Text Style Transfer Using Reverse Attention and Conditional Layer Normalization"

Dongkyu Lee 4 Sep 18, 2022
Pytorch Implementations of large number classical backbone CNNs, data enhancement, torch loss, attention, visualization and some common algorithms.

Torch-template-for-deep-learning Pytorch implementations of some **classical backbone CNNs, data enhancement, torch loss, attention, visualization and

Li Shengyan 270 Dec 31, 2022
Official code of "R2RNet: Low-light Image Enhancement via Real-low to Real-normal Network."

R2RNet Official code of "R2RNet: Low-light Image Enhancement via Real-low to Real-normal Network." Jiang Hai, Zhu Xuan, Ren Yang, Yutong Hao, Fengzhu

null 77 Dec 24, 2022
The code release of paper Low-Light Image Enhancement with Normalizing Flow

[AAAI 2022] Low-Light Image Enhancement with Normalizing Flow Paper | Project Page Low-Light Image Enhancement with Normalizing Flow Yufei Wang, Renji

Yufei Wang 176 Jan 6, 2023
The code of Zero-shot learning for low-light image enhancement based on dual iteration

Zero-shot-dual-iter-LLE The code of Zero-shot learning for low-light image enhancement based on dual iteration. You can get the real night image tests

null 1 Mar 18, 2022
PyTorch code for the paper "Complementarity is the King: Multi-modal and Multi-grained Hierarchical Semantic Enhancement Network for Cross-modal Retrieval".

Complementarity is the King: Multi-modal and Multi-grained Hierarchical Semantic Enhancement Network for Cross-modal Retrieval (M2HSE) PyTorch code fo

Xinlei-Pei 6 Dec 23, 2022
PyTorch implementation of "A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement."

FullSubNet This Git repository for the official PyTorch implementation of "A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech E

郝翔 357 Jan 4, 2023
Speech Enhancement Generative Adversarial Network Based on Asymmetric AutoEncoder

ASEGAN: Speech Enhancement Generative Adversarial Network Based on Asymmetric AutoEncoder 中文版简介 Readme with English Version 介绍 基于SEGAN模型的改进版本,使用自主设计的非

Nitin 53 Nov 17, 2022
EDCNN: Edge enhancement-based Densely Connected Network with Compound Loss for Low-Dose CT Denoising

EDCNN: Edge enhancement-based Densely Connected Network with Compound Loss for Low-Dose CT Denoising By Tengfei Liang, Yi Jin, Yidong Li, Tao Wang. Th

workingcoder 115 Jan 5, 2023
Adversarial Color Enhancement: Generating Unrestricted Adversarial Images by Optimizing a Color Filter

ACE Please find the preliminary version published at BMVC 2020 in the folder BMVC_version, and its extended journal version in Journal_version. Datase

null 28 Dec 25, 2022
From Fidelity to Perceptual Quality: A Semi-Supervised Approach for Low-Light Image Enhancement (CVPR'2020)

Under-exposure introduces a series of visual degradation, i.e. decreased visibility, intensive noise, and biased color, etc. To address these problems, we propose a novel semi-supervised learning approach for low-light image enhancement.

Yang Wenhan 117 Jan 3, 2023
Python codes for Lite Audio-Visual Speech Enhancement.

Lite Audio-Visual Speech Enhancement (Interspeech 2020) Introduction This is the PyTorch implementation of Lite Audio-Visual Speech Enhancement (LAVSE

Shang-Yi Chuang 85 Dec 1, 2022
Tensorflow implementation of MIRNet for Low-light image enhancement

MIRNet Tensorflow implementation of the MIRNet architecture as proposed by Learning Enriched Features for Real Image Restoration and Enhancement. Lanu

Soumik Rakshit 91 Jan 6, 2023