caiyuanhao1998 / pngan Goto Github PK

View Code? Open in Web Editor NEW

119.0 8.0 22.0 45.91 MB

"Learning to Generate Realistic Noisy Images via Pixel-level Noise-aware Adversarial Training" (NeurIPS 2021)

Home Page: https://arxiv.org/abs/2204.02844

License: MIT License

MATLAB 2.54% Python 97.46%

image-generation denoising noise-modelling image-denoising image-processing image-restoration

pngan's Introduction

Learning to Generate Realistic Noisy Images via Pixel-level Noise-aware Adversarial Training (NeurIPS 2021)

News

2022.04.17 : Testing codes, pre-trained denoising models, and results have been released. 🚀
2021.09.30 : Our paper has been accepted by NeurIPS 2021. 🔥

Gaussian Noise	PNGAN Noise

Abstract: Existing deep learning real denoising methods require a large amount of noisy-clean image pairs for supervision. Nonetheless, capturing a real noisy-clean dataset is an unacceptable expensive and cumbersome procedure. To alleviate this problem, this work investigates how to generate realistic noisy images. Firstly, we formulate a simple yet reasonable noise model that treats each real noisy pixel as a random variable. This model splits the noisy image generation problem into two sub-problems: image domain alignment and noise domain alignment. Subsequently, we propose a novel framework, namely Pixel-level Noise-aware Generative Adversarial Network (PNGAN). PNGAN employs a pre-trained real denoiser to map the fake and real noisy images into a nearly noise-free solution space to perform image domain alignment. Simultaneously, PNGAN establishes a pixel-level adversarial training to conduct noise domain alignment. Additionally, for better noise fitting, we present an efficient architecture Simple Multi-scale Network (SMNet) as the generator. Qualitative validation shows that noise generated by PNGAN is highly similar to real noise in terms of intensity and distribution. Quantitative experiments demonstrate that a series of denoisers trained with the generated noisy images achieve state-of-the-art (SOTA) results on four real denoising benchmarks.

PNGAN Framework

Main Results

Noise Generation : We adopt the two metrics used in DANet to measuere the performance of our generated noise, i.e., PSNR Gap (PGap) and Average KL Divergence (AKLD). The results are shown in the following table.

Methods	CBDNet	ULRD	GRDN	DANet	PNGAN
PGap	8.30	4.90	2.28	2.06	0.84
AKLD	0.728	0.545	0.443	0.212	0.153

Noise Removal : The quantitative results of models finetuned with the image pairs generated by our PNGAN are listed in the following table. Our models achieve state-of-the-art results.

Create Environment

The model is built in PyTorch 1.1.0 and tested on Ubuntu 16.04 environment (Python3.7, CUDA9.0, cuDNN7.5).

For installing, follow these intructions

sudo apt-get install cmake build-essential libjpeg-dev libpng-dev
conda create -n pytorch1 python=3.7
conda activate pytorch1
conda install pytorch=1.7 torchvision=0.8.0 cudatoolkit=9.0 -c pytorch
pip install matplotlib scikit-image opencv-python yacs joblib natsort h5py tqdm

Or you can directly install

conda env create -f PNGAN.yaml

Noise Removal

Download our pre-trained models (Google Drive | Baidu Disk, code: 65m5) and place them into folder ./pre-trained following these instructions

# For SIDD denoising
python test_denoiser.py --method mirnet --input_dir ./datasets/SIDD/ --result_dir ./results/SIDD/noise_removal/ --weights ./pre-trained/MIRNet_sidd.pth

# For PolyU denoising
python test_denoiser.py --method mirnet --input_dir ./datasets/PolyU/ --result_dir ./results/PolyU/noise_removal/ --weights ./pre-trained/MIRNet_polyu.pth

# For Nam denoising
python test_denoiser.py --method ridnet --input_dir ./datasets/Nam/ --result_dir ./results/Nam/noise_removal/ --weights ./pre-trained/RIDNet_nam.pth

For DND denoising, we provide the denoised results on DND in folder ./results/DND/noise_removal/. The file is named in terms of id_PSNR_SSIM.png. Our method ranks the 4th place on the public leaderboard. We have also provided denoised results on SIDD, PolyU, and Nam.

for instance, on PolyU

the denoised results are in ./results/PolyU/noise_removal/

You can also refer to this repo for the application of PNGAN on Reformer.

Noise Generation

We provide the noisy images generated by PNGAN on SIDD, DND, PolyU, Nam, DF2K, Urban100, Kodak24, BSD68 and Gaussian noisy images (X50) for other works to compare their performance with PNGAN.

for instance, on SIDD

the generated noisy images are in ./results/SIDD/noise_modeling/

the Gaussian noisy images are in ./results/SIDD/Gaussian_noise/

And you can find the clean image counterparts in ./datasets/SIDD/groundtruth/, feel free to use your method to generate noisy images from the clean counterparts and make a convenient comparison with our PNGAN.

If you want to compare Gaussian noisy images at other level (e.g., X30, X70), run the matlab script:

run('Generate_TrainData_HQ_LQ_Denoising_RGB.m')

You can also refer to this repo for the application of PNGAN on Reformer.

Citation

@article{cai2021learning,
  title={Learning to generate realistic noisy images via pixel-level noise-aware adversarial training},
  author={Cai, Yuanhao and Hu, Xiaowan and Wang, Haoqian and Zhang, Yulun and Pfister, Hanspeter and Wei, Donglai},
  journal={Advances in Neural Information Processing Systems},
  volume={34},
  pages={3259--3270},
  year={2021}
}

@inproceedings{pngan, 
  title={Learning to Generate Realistic Noisy Images via Pixel-level Noise-aware Adversarial Training}, 
  author={Cai, Yuanhao and Hu, Xiaowan and Wang, Haoqian and Zhang, Yulun and Pfister, Hanspeter and Wei, Donglai}, 
  booktitle={NeurIPS}, 
  year={2021}
}

Acknowledgement

The authors want to thank the following repo for their application of PNGAN on Reformer.

https://github.com/GarrickZ2/Image-Denoising

pngan's People

Contributors

Stargazers

Watchers

Forkers

linjing7 cd678 dengzhuo-ai misaka-10079 chenyang1999 btkforever linyiyang98 fentaniao shahriar1990 teacupboy hansonbee reallqk tsunamiblue 9vivian88 dabinishere hebychen1218 cyh1998jhu chris-han bitcg

pngan's Issues

Hello, I noticed that your pre-training model does not include the dnd dataset. Are you using polyu, nam, sidd models directly?

Log Files from Training

Thank you for your awesome code!

I am hoping you might open-source the log files you have from training. Maybe the training and validation loss as a function of epoch
(and/or batch) with an estimate of the runtime?

Some questions about training PNGAN

I trained PNGAN（https://github.com/GarrickZ2/Image-Denoising/tree/master/PNGAN）, and the result of the generated noisy image is very poor. I don't know where I neglected or made a mistake. Please help me to find out what went wrong. Among them, the loss values of both the generator and the discriminator are high.

Can I take a look at your training log? I will refer to it. Because I want to reproduce it first, all parameters and data sets are used by default (such as the default optimal lambda_p=6e-3, lambda_ra=8e-4 in the article and code). Once I reproduce the effect in the schematic you gave, I will tune the hyperparameters to get better results in other applications.

This is the noisy image I generated

Here is my training log
Configuration:
Namespace(act='relu', batch_size=8, benchmark_noise=False, beta1=0.9, beta2=0.9999, chop=False, cpu=True, data_test='SIDD', data_train='SIDD', debug=False, decay_type='step', dir_data='/home/shendi_mcj/datasets/SIDD_128/Datasets', epochs=100, epsilon=1e-08, ext='sep_reset', extend='.', gamma=0.5, gan_k=1, generate=False, load='.', load_best=False, load_dir='.', load_epoch=27, load_models=True, loss='1*L1', lr=0.0002, lr_decay_step=100000.0, lr_min=1e-07, model='RIDNET', momentum=0.9, n_GPUs=1, n_colors=3, n_feats=64, n_threads=8, n_train=20000, n_val=2000, noise=50, noise_g=[1], optimizer='ADAM', partial_data=True, patch_size=128, pre_train='experiment/ridnet.pt', precision='single', predict_patch_size=800, print_every=100, print_model=False, reduction=16, res_scale=1, reset=False, resume=0, rgb_range=255, save='./', save_models=True, save_results=False, savepath='./save', seed=1, self_ensemble=False, shift_mean=True, skip_threshold=1000000.0, split_batch=1, template='.', test_every=1000, test_only=False, testpath='./test', timestamp='1665173019', weight_decay=0.8)
Making model...
Loading model from experiment/ridnet.pt
Load Model from epoch: 27
Epoch 28:g_trn_l=4260.0166,d_trn_l=-8.6347: 100%|█| 2500/2500 [18:04<00:00, 2.3
val_d_loss=-16553.590091705322, val_g_loss=1456043.0610351562: 100%|█| 250/250 [
Epoch 29: val_d_loss=-8.276795045852662, val_g_loss=728.0215305175781
Epoch 29:g_trn_l=612.9119,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:27<00:00, 2.39
val_d_loss=-16508.084072113037, val_g_loss=1441245.7890625: 100%|█| 250/250 [00:
Epoch 30: val_d_loss=-8.254042036056518, val_g_loss=720.62289453125
Epoch 30:g_trn_l=569.982,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:29<00:00, 2.38i
val_d_loss=-16534.407222747803, val_g_loss=1445840.658203125: 100%|█| 250/250 [0
Epoch 31: val_d_loss=-8.267203611373901, val_g_loss=722.9203291015625
Epoch 31:g_trn_l=642.1012,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:27<00:00, 2.39
val_d_loss=-16530.938148498535, val_g_loss=1451748.1240234375: 100%|█| 250/250 [
Epoch 32: val_d_loss=-8.265469074249268, val_g_loss=725.8740620117187
Epoch 32:g_trn_l=1922.5544,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:29<00:00, 2.3
val_d_loss=-16461.000728607178, val_g_loss=1447958.69140625: 100%|█| 250/250 [00
Epoch 33: val_d_loss=-8.23050036430359, val_g_loss=723.979345703125
Epoch 33:g_trn_l=596.3593,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:28<00:00, 2.38
val_d_loss=-16396.410007476807, val_g_loss=1444418.5522460938: 100%|█| 250/250 [
Epoch 34: val_d_loss=-8.198205003738403, val_g_loss=722.2092761230468
Epoch 34:g_trn_l=4094.3552,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:33<00:00, 2.3
val_d_loss=-16325.490299224854, val_g_loss=1430286.7797851562: 100%|█| 250/250 [
Epoch 35: val_d_loss=-8.162745149612427, val_g_loss=715.1433898925782
Epoch 35:g_trn_l=562.7994,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:31<00:00, 2.38
val_d_loss=-16497.081832885742, val_g_loss=1441190.5700683594: 100%|█| 250/250 [
Epoch 36: val_d_loss=-8.24854091644287, val_g_loss=720.5952850341797
Epoch 36:g_trn_l=549.0154,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:32<00:00, 2.38
val_d_loss=-16452.982189178467, val_g_loss=1446414.1437988281: 100%|█| 250/250 [
Epoch 37: val_d_loss=-8.226491094589234, val_g_loss=723.207071899414
Epoch 37:g_trn_l=1389.9574,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:27<00:00, 2.3
val_d_loss=-16494.09930419922, val_g_loss=1442426.9399414062: 100%|█| 250/250 [0
Epoch 38: val_d_loss=-8.247049652099609, val_g_loss=721.2134699707032
Epoch 38:g_trn_l=1731.5358,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:22<00:00, 2.4
val_d_loss=-16414.167991638184, val_g_loss=1442114.4387207031: 100%|█| 250/250 [
Epoch 39: val_d_loss=-8.207083995819092, val_g_loss=721.0572193603516
Epoch 39:g_trn_l=1161.1187,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:25<00:00, 2.3
val_d_loss=-16374.823741912842, val_g_loss=1435748.1032714844: 100%|█| 250/250 [
Epoch 40: val_d_loss=-8.18741187095642, val_g_loss=717.8740516357421
Epoch 40:g_trn_l=1218.5015,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:33<00:00, 2.3
val_d_loss=-16411.774070739746, val_g_loss=1437521.4143066406: 100%|█| 250/250 [
Epoch 41: val_d_loss=-8.205887035369873, val_g_loss=718.7607071533204
Epoch 41:g_trn_l=585.713,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:30<00:00, 2.38i
val_d_loss=-16421.3918800354, val_g_loss=1435851.421875: 100%|█| 250/250 [00:51<
Epoch 42: val_d_loss=-8.2106959400177, val_g_loss=717.9257109375
Epoch 42:g_trn_l=3450.4092,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:28<00:00, 2.3
val_d_loss=-16342.977878570557, val_g_loss=1436181.6176757812: 100%|█| 250/250 [
Epoch 43: val_d_loss=-8.171488939285279, val_g_loss=718.0908088378907
Epoch 43:g_trn_l=1209.9535,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:36<00:00, 2.3
val_d_loss=-16354.145488739014, val_g_loss=1442767.9860839844: 100%|█| 250/250 [
Epoch 44: val_d_loss=-8.177072744369507, val_g_loss=721.3839930419922
Epoch 44:g_trn_l=1001.378,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:33<00:00, 2.37
val_d_loss=-16319.988140106201, val_g_loss=1442307.275390625: 100%|█| 250/250 [0
Epoch 45: val_d_loss=-8.1599940700531, val_g_loss=721.1536376953125
Epoch 45:g_trn_l=4854.5229,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:34<00:00, 2.3
val_d_loss=-16327.855934143066, val_g_loss=1427248.0895996094: 100%|█| 250/250 [
Epoch 46: val_d_loss=-8.163927967071533, val_g_loss=713.6240447998047
Epoch 46:g_trn_l=1594.9669,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:34<00:00, 2.3
val_d_loss=-16331.874336242676, val_g_loss=1439591.0356445312: 100%|█| 250/250 [
Epoch 47: val_d_loss=-8.165937168121339, val_g_loss=719.7955178222657
Epoch 47:g_trn_l=1727.7327,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:34<00:00, 2.3
val_d_loss=-16301.70266342163, val_g_loss=1430494.6499023438: 100%|█| 250/250 [0
Epoch 48: val_d_loss=-8.150851331710815, val_g_loss=715.2473249511719
Epoch 48:g_trn_l=662.2933,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:29<00:00, 2.38
val_d_loss=-16289.168937683105, val_g_loss=1445139.4096679688: 100%|█| 250/250 [
Epoch 49: val_d_loss=-8.144584468841552, val_g_loss=722.5697048339844
Epoch 49:g_trn_l=638.9958,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:34<00:00, 2.37
val_d_loss=-16676.23886871338, val_g_loss=1443492.8112792969: 100%|█| 250/250 [0
Epoch 50: val_d_loss=-8.33811943435669, val_g_loss=721.7464056396484
Epoch 50:g_trn_l=449.2137,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:34<00:00, 2.37
val_d_loss=-16709.131187438965, val_g_loss=1430881.7004394531: 100%|█| 250/250 [
Epoch 51: val_d_loss=-8.354565593719482, val_g_loss=715.4408502197266
Epoch 51:g_trn_l=1438.2651,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:35<00:00, 2.3
val_d_loss=-16641.16509246826, val_g_loss=1435487.1213378906: 100%|█| 250/250 [0
Epoch 52: val_d_loss=-8.320582546234132, val_g_loss=717.7435606689453
Epoch 52:g_trn_l=597.417,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:32<00:00, 2.38i
val_d_loss=-16499.351203918457, val_g_loss=1440687.0793457031: 100%|█| 250/250 [
Epoch 53: val_d_loss=-8.249675601959229, val_g_loss=720.3435396728515
Epoch 53:g_trn_l=569.0695,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:31<00:00, 2.38
val_d_loss=-16576.316123962402, val_g_loss=1441669.537109375: 100%|█| 250/250 [0
Epoch 54: val_d_loss=-8.288158061981202, val_g_loss=720.8347685546875
Epoch 54:g_trn_l=1885.5017,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:36<00:00, 2.3
val_d_loss=-16543.578399658203, val_g_loss=1450123.9528808594: 100%|█| 250/250 [
Epoch 55: val_d_loss=-8.2717891998291, val_g_loss=725.0619764404297
Epoch 55:g_trn_l=976.4413,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:33<00:00, 2.37
val_d_loss=-16528.236602783203, val_g_loss=1476003.5375976562: 100%|█| 250/250 [
Epoch 56: val_d_loss=-8.264118301391601, val_g_loss=738.0017687988282
Epoch 56:g_trn_l=872.3619,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:35<00:00, 2.37
val_d_loss=-16424.807899475098, val_g_loss=1428827.9528808594: 100%|█| 250/250 [
Epoch 57: val_d_loss=-8.21240394973755, val_g_loss=714.4139764404297
Epoch 57:g_trn_l=58598960922624.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:34<00:0
val_d_loss=3051.3138489723206, val_g_loss=6.354772355868262e+16: 100%|█| 250/250
Epoch 58: val_d_loss=1.5256569244861602, val_g_loss=31773861779341.312
Epoch 58:g_trn_l=3881488875520.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:34<00:00
val_d_loss=7193.342471122742, val_g_loss=5.553211802701005e+16: 100%|█| 250/250
Epoch 59: val_d_loss=3.596671235561371, val_g_loss=27766059013505.023
Epoch 59:g_trn_l=67181735837696.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:34<00:0
val_d_loss=8914.092770576477, val_g_loss=1.2930694204778086e+17: 100%|█| 250/250
Epoch 60: val_d_loss=4.457046385288239, val_g_loss=64653471023890.43
Epoch 60:g_trn_l=54223576236032.0,d_trn_l=-8.6331: 100%|█| 2500/2500 [17:27<00:0
val_d_loss=16679.991931915283, val_g_loss=1.2193291783530086e+17: 100%|█| 250/25
Epoch 61: val_d_loss=8.339995965957641, val_g_loss=60966458917650.43
Epoch 61:g_trn_l=38286747762688.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:26<00:0
val_d_loss=7301.153503417969, val_g_loss=7.05499680955433e+16: 100%|█| 250/250 [
Epoch 62: val_d_loss=3.6505767517089844, val_g_loss=35274984047771.65
Epoch 62:g_trn_l=1624666144768.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:24<00:00
val_d_loss=10444.144174575806, val_g_loss=4079118023655424.0: 100%|█| 250/250 [0
Epoch 63: val_d_loss=5.222072087287903, val_g_loss=2039559011827.712
Epoch 63:g_trn_l=3864946802688.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:24<00:00
val_d_loss=10191.777889251709, val_g_loss=5970656558579712.0: 100%|█| 250/250 [0
Epoch 64: val_d_loss=5.095888944625854, val_g_loss=2985328279289.856
Epoch 64:g_trn_l=19007925125120.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:21<00:0
val_d_loss=14041.098752975464, val_g_loss=6.728756782484685e+16: 100%|█| 250/250
Epoch 65: val_d_loss=7.020549376487732, val_g_loss=33643783912423.426
Epoch 65:g_trn_l=41974014935040.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:24<00:0
val_d_loss=15734.480403900146, val_g_loss=9.42083598796718e+16: 100%|█| 250/250
Epoch 66: val_d_loss=7.867240201950073, val_g_loss=47104179939835.91
Epoch 66:g_trn_l=4564226932736.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:25<00:00
val_d_loss=-8555.47631263733, val_g_loss=1.115331718283264e+16: 100%|█| 250/250
Epoch 67: val_d_loss=-4.277738156318665, val_g_loss=5576658591416.32
Epoch 67:g_trn_l=2083470442496.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:32<00:00
val_d_loss=-10110.21891784668, val_g_loss=5013995595497472.0: 100%|█| 250/250 [0
Epoch 68: val_d_loss=-5.05510945892334, val_g_loss=2506997797748.736
Epoch 68:g_trn_l=2037396799488.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:21<00:00
val_d_loss=-6897.837250709534, val_g_loss=6761572339810304.0: 100%|█| 250/250 [0
Epoch 69: val_d_loss=-3.448918625354767, val_g_loss=3380786169905.152
Epoch 69:g_trn_l=19517893771264.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:31<00:0
val_d_loss=-4325.360991001129, val_g_loss=2.0603421086449664e+16: 100%|█| 250/25
Epoch 70: val_d_loss=-2.1626804955005645, val_g_loss=10301710543224.832
Epoch 70:g_trn_l=2508975767552.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:28<00:00
val_d_loss=-9451.344500541687, val_g_loss=3735847221329920.0: 100%|█| 250/250 [0
Epoch 71: val_d_loss=-4.7256722502708435, val_g_loss=1867923610664.96
Epoch 71:g_trn_l=1847262969856.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:31<00:00
val_d_loss=-5519.979823589325, val_g_loss=3760646322651136.0: 100%|█| 250/250 [0
Epoch 72: val_d_loss=-2.7599899117946625, val_g_loss=1880323161325.568
Epoch 72:g_trn_l=520190787584.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:30<00:00,
val_d_loss=-6887.619359970093, val_g_loss=1009447890845696.0: 100%|█| 250/250 [0
Epoch 73: val_d_loss=-3.4438096799850464, val_g_loss=504723945422.848
Epoch 73:g_trn_l=226598469632.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:27<00:00,
val_d_loss=-15223.686599731445, val_g_loss=439739038236672.0: 100%|█| 250/250 [0
Epoch 74: val_d_loss=-7.611843299865723, val_g_loss=219869519118.336
Epoch 74:g_trn_l=176480763904.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:29<00:00,
val_d_loss=-4537.72459936142, val_g_loss=388233302310912.0: 100%|█| 250/250 [00:
Epoch 75: val_d_loss=-2.26886229968071, val_g_loss=194116651155.456
Epoch 75:g_trn_l=111401664512.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:27<00:00,
val_d_loss=-2710.772897720337, val_g_loss=248896543784960.0: 100%|█| 250/250 [00
Epoch 76: val_d_loss=-1.3553864488601683, val_g_loss=124448271892.48
Epoch 76:g_trn_l=223202803712.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:25<00:00,
val_d_loss=275.7913808822632, val_g_loss=495202342076416.0: 100%|█| 250/250 [00:
Epoch 77: val_d_loss=0.1378956904411316, val_g_loss=247601171038.208
Epoch 77:g_trn_l=145705172992.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:32<00:00,
val_d_loss=8066.014381408691, val_g_loss=424581223809024.0: 100%|█| 250/250 [00:
Epoch 78: val_d_loss=4.033007190704346, val_g_loss=212290611904.512
Epoch 78:g_trn_l=1403458289664.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:28<00:00
val_d_loss=3504.658802986145, val_g_loss=2283288738398208.0: 100%|█| 250/250 [00
Epoch 79: val_d_loss=1.7523294014930726, val_g_loss=1141644369199.104
Epoch 79:g_trn_l=233238986752.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:34<00:00,
val_d_loss=6305.426145553589, val_g_loss=489256021917696.0: 100%|█| 250/250 [00:
Epoch 80: val_d_loss=3.1527130727767942, val_g_loss=244628010958.848
Epoch 80:g_trn_l=8642871427072.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:33<00:00
val_d_loss=-7856.273780822754, val_g_loss=1.2819610196770816e+16: 100%|█| 250/25
Epoch 81: val_d_loss=-3.928136890411377, val_g_loss=6409805098385.408
Epoch 81:g_trn_l=91949588480.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:36<00:00,
val_d_loss=-411.3753271102905, val_g_loss=437178517553152.0: 100%|█| 250/250 [00
Epoch 82: val_d_loss=-0.20568766355514526, val_g_loss=218589258776.576
Epoch 82:g_trn_l=125667479912448.0,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:35<00:
val_d_loss=-2535.5058431625366, val_g_loss=2.571578478784676e+17: 100%|█| 250/25
Epoch 83: val_d_loss=-1.2677529215812684, val_g_loss=128578923939233.8
Epoch 83:g_trn_l=2.7062537286275625e+23,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:3
val_d_loss=-11571.54788017273, val_g_loss=1.6632423230309563e+27: 100%|█| 250/25
Epoch 84: val_d_loss=-5.785773940086365, val_g_loss=8.316211615154782e+23
Epoch 84:g_trn_l=3.415223712623123e+23,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:37
val_d_loss=-10394.510999679565, val_g_loss=1.6684358548098048e+27: 100%|█| 250/2
Epoch 85: val_d_loss=-5.197255499839783, val_g_loss=8.342179274049023e+23
Epoch 85:g_trn_l=2.7823559154188393e+23,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:2
val_d_loss=-11055.715322494507, val_g_loss=1.622653395616832e+27: 100%|█| 250/25
Epoch 86: val_d_loss=-5.527857661247253, val_g_loss=8.113266978084161e+23
Epoch 86:g_trn_l=4.9750231057087425e+23,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:2
val_d_loss=-9373.261775970459, val_g_loss=1.617555301238983e+27: 100%|█| 250/250
Epoch 87: val_d_loss=-4.686630887985229, val_g_loss=8.087776506194916e+23
Epoch 87:g_trn_l=2.3236702020938654e+24,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:3
val_d_loss=-5703.948229789734, val_g_loss=1.6296847968279857e+27: 100%|█| 250/25
Epoch 88: val_d_loss=-2.8519741148948667, val_g_loss=8.148423984139929e+23
Epoch 88:g_trn_l=3.810222546436773e+23,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:31
val_d_loss=-10268.105506896973, val_g_loss=1.6421163303251944e+27: 100%|█| 250/2
Epoch 89: val_d_loss=-5.134052753448486, val_g_loss=8.210581651625973e+23
Epoch 89:g_trn_l=1.5643003265820106e+24,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:2
val_d_loss=-7419.597512245178, val_g_loss=1.6674274361631297e+27: 100%|█| 250/25
Epoch 90: val_d_loss=-3.709798756122589, val_g_loss=8.337137180815649e+23
Epoch 90:g_trn_l=7.42748494410314e+23,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:32<
val_d_loss=-10104.679119110107, val_g_loss=1.62342235217877e+27: 100%|█| 250/250
Epoch 91: val_d_loss=-5.052339559555054, val_g_loss=8.117111760893849e+23
Epoch 91:g_trn_l=1.0870287259692399e+24,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:3
val_d_loss=-8483.54418373108, val_g_loss=1.5998357661512992e+27: 100%|█| 250/250
Epoch 92: val_d_loss=-4.241772091865539, val_g_loss=7.999178830756496e+23
Epoch 92:g_trn_l=1.1925134775933924e+24,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:3
val_d_loss=-8301.296371459961, val_g_loss=1.6230275357947843e+27: 100%|█| 250/25
Epoch 93: val_d_loss=-4.15064818572998, val_g_loss=8.115137678973922e+23
Epoch 93:g_trn_l=2.9071117499924953e+23,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:3
val_d_loss=-10424.478651046753, val_g_loss=1.6232417481779947e+27: 100%|█| 250/2
Epoch 94: val_d_loss=-5.2122393255233765, val_g_loss=8.116208740889973e+23
Epoch 94:g_trn_l=4.567849420806444e+23,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:38
val_d_loss=-8979.812364578247, val_g_loss=1.6686692055458765e+27: 100%|█| 250/25
Epoch 95: val_d_loss=-4.489906182289124, val_g_loss=8.343346027729382e+23
Epoch 95:g_trn_l=4.156632224014708e+23,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:36
val_d_loss=-10392.455976486206, val_g_loss=1.631881676505223e+27: 100%|█| 250/25
Epoch 96: val_d_loss=-5.196227988243103, val_g_loss=8.159408382526115e+23
Epoch 96:g_trn_l=5.133901453651089e+23,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:34
val_d_loss=-7938.717601776123, val_g_loss=1.6561539731179674e+27: 100%|█| 250/25
Epoch 97: val_d_loss=-3.9693588008880614, val_g_loss=8.280769865589838e+23
Epoch 97:g_trn_l=5.745301492151623e+23,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:33
val_d_loss=-7501.854158401489, val_g_loss=1.63275133052841e+27: 100%|█| 250/250
Epoch 98: val_d_loss=-3.7509270792007445, val_g_loss=8.16375665264205e+23
Epoch 98:g_trn_l=8.744793599716533e+23,d_trn_l=-8.6347: 100%|█| 2500/2500 [17:34
val_d_loss=-10151.39899635315, val_g_loss=1.6066991155063287e+27: 100%|█| 250/25
Epoch 99: val_d_loss=-5.075699498176575, val_g_loss=8.033495577531644e+23

Questioning the finetuning experiments of MPRNET/MIRNET

Dear author,
According to the conclusion of your experiment based on the data proportion, fine-tuning only with the original SIDD data actually performs the best on the SIDD validation set. That is to say, if you only use the original SIDD data for finetuning in your quantity increment experiment, MIRNET can even achieve higher scores than 40.07, which means an improvement of more than 0.35DB. However, this is obviously not reasonable. After conducting numerous experiments, we found that with only the original SIDD data for finetuning, there is only a slight increase of 0.01~0.02DB, which is far below such a high level of improvement. So we want to ask how did you achieve this?I look forward to your response and would greatly appreciate it.

Details about provided weights

Hi, @caiyuanhao1998 !
Thank you for your awesome work!
Can you please clarify the difference between MIRNet_sidd and MIRNet_polyu? I suppose these two models were trained on exact data but with different hyperparameters; am I correct?

A question about experiment setup

Hi,

Thanks for your amazing work.

I would like to ask a step concerning the "3.1 Experiment step" in the paper.

In the paper, you stated that "G uses clean images from DIV2K [59], Flickr2K [60], BSD68 [61], Kodak24 [62], and Urban100 [63] to generate realistic noisy-clean image pairs". I assume this step is in the (b) training phrase and the generated (fake) noisy images and corresponding real noisy images will be inputted into two discriminator to train them.

However, some of the datasets (Flickr2K, Kodak24, Urban100) you used in this phase don't contain noisy images. I don't know how could the training of the two discriminator be done in the phase. Or you just add gaussian noise to these three dataset and regard them as real noisy images.

Thanks in advance for the clarification.

Could you show some detail about the training loss implementation

Hi there,
I'm wondering how you implement the loss function for Generator and Discriminator respectively. I notice that your overall loss is quite original, but I still hope to know which part will be involved in the Generator's Loss, and which part will be involved in the Discriminator's Loss.
Thanks for any response!

Noising Generator

Hi, thanks for the great work, is there any plans to share the Generator weights? I would like to use the pre-trained generated on create noisy images from my clean images data.

Example to train PNGAN with new data

Hi,

Do you have any example to train PNGAN with new data? Right now i didn't see any related code in the repo and in case i overlooked it, could you point me to the correct section?

Thanks.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.