HINet: Half Instance Normalization Network for Image Restoration

This is nice work and thanks for your share. however, when I train the model there have some issues, in the val dataset. I found you utilize the VAlidationNoisyBlocksSrgb.mat to validation the model. but there have some mistakes when I run the program. can you help me to solve this problem? thank you very much.

我按照您的方式组织了数据集的结构，但是报错： FileNotFoundError: [Errno 2] No such file or directory: './datasets/GoPro/train/blur_crops.lmdb/meta_info.txt' 请问’blur_crops.lmdb/meta_info.txt‘是什么呢 我该如何解决

Thanks for writing a good paper. I have data[input,target] that I have. Now

./datasets/
./datasets/GoPro/
./datasets/GoPro/train/
./datasets/GoPro/train/input/
./datasets/GoPro/train/target/
./datasets/GoPro/test/
./datasets/GoPro/test/input/
./datasets/GoPro/test/target/

Create a folder in the same way as

python scripts/data_preparation/gopro.py
Through preprocessing, blur_crops, blur_crops.imdb / sharp_crops, sharp_crops.imdb datasets were created.

Finally

python -m torch.distributed.launch --nproc_per_node=8 --master_port=4321 basicsr/train.py -opt options/train/GoPro/HINet.yml --launcher pytorch

I tried to learn through the above command. But I got the following error:

ValueError: Keys in lq_folder and gt_folder are different.
...
...
subprocess.CalledProcessError: Command '['/home/ubuntu/anaconda3/envs/HINet/bin/python', '-u', 'basicsr/train.py', '--local_rank=7', '-opt', 'options/train/GoPro/HINet.yml', '--launcher', 'pytorch']' returned non-zero exit status 1.

How can I fix the error?

Thanks.

Hi,

Thank you for this wonderful project. I am able to successfully run the demo with the GOPRO weights on multiple images and it works flawlessly. However, if I just tweak the demo.yml file and change the pre-trained weights to HINet-SIDD-1x.pth, I get an error with the state_dit. I am new to PyTorch, so this may be a trivial issue. Your help and guidance are much appreciated. I am attaching the demo.yml file to this post and the output:

# ------------------------------------------------------------------------
# Copyright (c) 2021 megvii-model. All Rights Reserved.
# ------------------------------------------------------------------------
# Modified from BasicSR (https://github.com/xinntao/BasicSR)
# Copyright 2018-2020 BasicSR Authors
# ------------------------------------------------------------------------
# general settings
name: demo
model_type: ImageRestorationModel
scale: 1
num_gpu: 1 # set num_gpu: 0 for cpu mode
manual_seed: 10

# single image inference and save image
img_path:
  input_img: ./demo/GOPR0384_11_00-000001.png
  output_img: ./demo/demo1.png

# network structures
network_g:
  type: HINet
  wf: 64
  hin_position_left: 3
  hin_position_right: 4

# path
path:
  pretrain_network_g: ./experiments/pretrained_models/HINet-SIDD-1x.pth
  strict_load_g: true
  resume_state: ~

# validation settings
val:
  grids: true
  crop_size: 256

# dist training settings
dist_params:
  backend: nccl
  port: 29500

The error log is as follows:

(HINet) C:\Users\Yusuf\Desktop\HMC Research\HINet>python basicsr/demo.py -opt options/demo/demo.yml
Disable distributed.
 load net keys <built-in method keys of collections.OrderedDict object at 0x000002284651FEA0>
Current net - loaded net:
Loaded net - current net:
  down_path_1.0.norm.bias
  down_path_1.0.norm.weight
  down_path_1.1.norm.bias
  down_path_1.1.norm.weight
  down_path_1.2.norm.bias
  down_path_1.2.norm.weight
  down_path_2.0.norm.bias
  down_path_2.0.norm.weight
  down_path_2.1.norm.bias
  down_path_2.1.norm.weight
  down_path_2.2.norm.bias
  down_path_2.2.norm.weight
Traceback (most recent call last):
  File "basicsr/demo.py", line 46, in <module>
    main()
  File "basicsr/demo.py", line 40, in main
    model = create_model(opt)
  File "c:\users\yusuf\desktop\hmc research\hinet\basicsr\models\__init__.py", line 44, in create_model
    model = model_cls(opt)
  File "c:\users\yusuf\desktop\hmc research\hinet\basicsr\models\image_restoration_model.py", line 37, in __init__
    self.opt['path'].get('strict_load_g', True), param_key=self.opt['path'].get('param_key', 'params'))
  File "c:\users\yusuf\desktop\hmc research\hinet\basicsr\models\base_model.py", line 287, in load_network
    net.load_state_dict(load_net, strict=strict)
  File "C:\Users\Yusuf\anaconda3\envs\HINet\lib\site-packages\torch\nn\modules\module.py", line 847, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for HINet:
        Unexpected key(s) in state_dict: "down_path_1.0.norm.weight", "down_path_1.0.norm.bias", "down_path_1.1.norm.weight", "down_path_1.1.norm.bias", "down_path_1.2.norm.weight", "down_path_1.2.norm.bias", "down_path_2.0.norm.weight", "down_path_2.0.norm.bias", "down_path_2.1.norm.weight", "down_path_2.1.norm.bias", "down_path_2.2.norm.weight", "down_path_2.2.norm.bias".

Hi,

I was able to reproduce your results on PSNR/SSIM. However, I observed that the output images have border artifacts even though I turned off the grids attribute:

I guessed during inference the image was paritioned, inferenced, and then stitched back. I am not very familiar with basicsr lib, so I simply tried turning off girds attribute, but it had no effect. Any ideas on what happened? Thank you!

Hi, thanks for sharing this great works! It seems MPRNet uses different parameters for each task, (like channels=40, 64, 80, etc). Could you please add an extra column in Table 4 showing the exact parameters for each model? I really appreciate if you can do so!

Hello! Thank you for your work. I measured the number of parameters of the HINet model for deblurring. Is it correct that the number of parameters of HINet used for deblurring is 88.67M (Million)?

When I used your model and wrote my own training code, I found a bug.

def main(opt):

loader_train = getDataLoader(opt)
model = getModel(opt)
model_saver = utils.modelSaver(opt['save_path'], opt['save_freq'])
criterion = nn.L1Loss()
model, criterion = updateGPU(model, criterion, opt)
optimizer, scheduler = getOptimizer(model, opt)

last_avg_psnr = 0
last_avg_loss = 0
for epoch in range(0, opt['epochs']):
    
    scheduler.step(epoch)
    avg_psnr = 0
    avg_loss = 0

    num_iterations = len(loader_train)
    num_iter_counts = 0
    l_total = 0
    loss = 0.
    for i, (input_train, target_train) in enumerate(loader_train):

        if(random.random() > opt['sampling_rate']):
            continue

        num_iter_counts += 1

        model.train()
        model.zero_grad()
        optimizer.zero_grad()

        input_train = Variable(input_train)
        target_train = Variable(target_train)
        
        if opt['use_gpu']:
            input_train = input_train.cuda()
            target_train = target_train.cuda()
        
        out_train = model(input_train)
        if not isinstance(out_train, list):
            out_train = [out_train]
        for pred in out_train:

            loss += criterion(target_train, pred)
        torch.autograd.set_detect_anomaly(True)
        loss.backward(retain_graph=True)
        optimizer.step()

        model.eval()
        out_train= model(input_train)
        for pp in out_train:
            pp += torch.clamp(pp, 0., 1.)
        psnr_train = utils.batch_PSNR(pp, target_train, 1.)
        avg_psnr += psnr_train 
        avg_loss += loss.item() 
            
        print("[epoch %d/%d][%d/%d], last avg loss: %.4f, avg PSNR: %.4f" % (epoch+1, opt['epochs'], i+1, num_iterations, last_avg_loss, last_avg_psnr), end='\r')

    # macs, params = get_model_complexity_info(model, (3, 80, 80), as_strings=True,
    #                                          print_per_layer_stat=True, verbose=True)
    # print('{:<30}  {:<8}'.format('Computational complexity: ', macs))
    # print('{:<30}  {:<8}'.format('Number of parameters: ', params))
    last_avg_psnr = avg_psnr / (num_iter_counts * 1.0)
    last_avg_loss = avg_loss / (num_iter_counts * 1.0)
    model_saver.saveModel(model, epoch, last_avg_psnr)

File "/home/szp/src/train.py", line 158, in main(opt) File "/home/szp/src/train.py", line 105, in main loss.backward(retain_graph=True) File "/home/szp/.local/lib/python3.6/site-packages/torch/tensor.py", line 245, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/szp/.local/lib/python3.6/site-packages/torch/autograd/init.py", line 147, in backward allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3, 64, 3, 3]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Process finished with exit code 1

hello, thanks for your excellent work, but i have a question about the validation set, the validation set of MPRNet is Rain100H, but your validation sets is all the five test sets, is it fair?

Great job creators! Very impressive work! I wonder what book influenced your python skills the most.

Is this project adopting the MPRNets' license?

Any plans for realising inference code for a single image. E.g demo code. If not maybe I can submit a pull request, any preferences for a way it should be formalized?

Keep it up!

Hi thank you for the great work I have a question about denoising task (training SIDD)

In the options, it seems like training SIDD only use L1 loss not PSNR loss Is this option same with your paper experiment setting? Then use only L1 loss is sufficient for denoising task?

Thank you!

Thank you for your awesome code!

I am hoping you might open-source the log files you have from training. Maybe the training and validation loss as a function of epoch (and/or batch) with an estimate of the runtime?

Hey @mayorx ,

Thanks for your impressive work! Since we want to do some similar work based on your project, we are in short of computation resources.

Hence, may I know the training cost of your work?

What kind of GPUs were you using? and How many GPUs? May I know how long you cost to train a model for a draining task?

Best Regards and many thanks,

maybe you have forgotten: "python setup.py develop --no_cuda_ext"

Originally posted by @mexthecat in https://github.com/megvii-model/HINet/issues/38#issuecomment-979738992

What is the operation of "python setup.py develop --no_cuda_ext" used for?

感谢和赞叹您的成就！ 关于PSNRLOSS，它看起来是类似MSELOSS的。在文章中的PSNRLOSS公式中，我有些疑问：请问为什么是（Ri+Xi），Y 呢，我个人以为是仅使用每阶段的重建图像（Xi）和ground truth（Y）计算即可。另外，Ri作为输入的模糊图像，我以为两阶段的Ri是相同的，但是实际上是不同的吗？（因为您用了下标i）。这两个问题我比较费解，能否请您帮忙讲解。 期待您的回复！再次感谢您的耐心！