Thank you for providing the wonderful codes. However, I am confused about the function forward_chop in model/__init.py. It seems that this function unfolds the input image into several patches and then feeds those patches into the IPT model, but I didn't find any detailed explanation in the paper or as comments in the code. For example, what does shave mean here? If I want to unfold the input image into non-overlapped patches, how should I do it?

Thank you.

Dear noah ： Thank you very much for your work！I looked at your paper and was struck by the beauty of it！ I want to know When you will released the code! THANK YOU!

I input : python main.py --dir_data C:/Users/Lenovo/Desktop/Pretrained-IPT-main/data_test/benchmark/ --pretrain C:/Users/Lenovo/Desktop/Pretrained-IPT-main/pretrained_model/IPT_sr2.pt --data_test Set5+Set14+B100+Urban100 --scale 2

出现了 Making model... Preparing loss function: 1.000 * L1

Evaluation: 0it [00:00, ?it/s]

[Set5 x2] PSNR: nan (Best: nan @epoch 1) 0it [00:00, ?it/s] 0it [00:12, ?it/s]

[Set14 x2] PSNR: nan (Best: nan @epoch 1) 0it [00:00, ?it/s]

[B100 x2] PSNR: nan (Best: nan @epoch 1) 0it [00:00, ?it/s]

[Urban100 x2] PSNR: nan (Best: nan @epoch 1) Forward: 49.39s

Thank you for the nice work and for sharing the pre-trained model!

I found an error in the implementation of PSNR calculation.

"For all tasks, PSNR is calculated only for y, rather than RGB values."

The below function has "y" as the default argument for cal_type, and for all tasks, no argument for cal_type is given.

https://github.com/huawei-noah/Pretrained-IPT/blob/a8f63ddda41498b38c5322be48308dc8d005e526/utility.py#L168-L184

For example, this is the way to calculate PSNR for denoising task.

https://github.com/huawei-noah/Pretrained-IPT/blob/a8f63ddda41498b38c5322be48308dc8d005e526/trainer.py#L63-L75

If we use cal_type="y", it gives 32.415 dB for gaussian denoising (simga=30) with CBSDS68 dataset.

But, if we give the other argument, it gives 30.748 dB which is much lower than the reported value in the paper.

Furthermore, if we save the result and calculate PSNR using skimage library, it gives around 30.75 dB, too.

Could you check the PSNR values?

Thank you for sharing this great repo!

I found that some links about datasets are not valid now. And some links are directed to some project pages, instead of downloadable links?

It would be great if you can update the links and let users directly download the datasets from those links. Thank you.

Hello,

Thanks for your great work of pretraining in low-level vision.

Could you please tell me which subset of ImageNet is used for pretraining, ImagNet-2012?

当使用save_result命令会出现错误 Evaluation: Traceback (most recent call last): File "main.py", line 37, in main() File "main.py", line 33, in main t.test() File "C:\Users\Lenovo\Desktop\Pretrained-IPT-main\trainer.py", line 34, in test if self.args.save_results: self.ckp.begin_background() File "C:\Users\Lenovo\Desktop\Pretrained-IPT-main\utility.py", line 144, in begin_background for p in self.process: p.start() File "E:\python3.7\lib\multiprocessing\process.py", line 112, in start self._popen = self._Popen(self) File "E:\python3.7\lib\multiprocessing\context.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "E:\python3.7\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "E:\python3.7\lib\multiprocessing\popen_spawn_win32.py", line 65, in init reduction.dump(process_obj, to_child) File "E:\python3.7\lib\multiprocessing\reduction.py", line 59, in dump ForkingPickler(file, protocol).dump(obj)

AttributeError: Can't pickle local object 'checkpoint.begin_background..bg_target'

C:\Users\Lenovo\Desktop\Pretrained-IPT-main>WARNING: Ignoring invalid distribution -ip (e:\python3.7\lib\site-packages) Traceback (most recent call last): File "", line 1, in File "E:\python3.7\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "E:\python3.7\lib\multiprocessing\spawn.py", line 115, in _main self = reduction.pickle.load(from_parent)

EOFError: Ran out of input

输入python main.py --dir_data test_data --pretrain model/IPT_sr2.pt --data_test Set5 --scale 1 出现了 Making model... Preparing loss function: 1.000 * L1 Evaluation: 0%| | 0/5 [00:00<?, ?it/s] RuntimeError: CUDA out of memory. Tried to allocate 118.00 MiB (GPU 0; 2.00 GiB total capacity; 1.19 GiB already allocated; 0 bytes free; 1.27 GiB reserved in total by PyTorch) 但是只有一个显卡，该怎么处理呢

Hi: Your work is really amazing，and I have some question after reading the paper: You mentioned in the article, "The whole IPT has 114M parameters and 33G FLOPs". What size of input image are the 33G FLOPs calculated here?

在测试sr x2任务时我将scale设置为2，PSNR结果是正确的 但我测试sr x3时，将scale 设置为3的时候出现了 RuntimeError: Error(s) in loading state_dict for ipt: size mismatch for tail.0.0.0.weight: copying a param with shape torch.Size([256, 64, 3, 3]) from checkpoint, the shape in current model is torch.Size([576, 64, 3, 3]). size mismatch for tail.0.0.0.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([576]). 测试sr x4 时，将scale 设置为4，但是PSNR仅为12.448

这是为什么呢

在测试sr x2任务时我将scale设置为2，PSNR结果是正确的 但我测试sr x3时，将scale 设置为3的时候出现了 RuntimeError: Error(s) in loading state_dict for ipt: size mismatch for tail.0.0.0.weight: copying a param with shape torch.Size([256, 64, 3, 3]) from checkpoint, the shape in current model is torch.Size([576, 64, 3, 3]). size mismatch for tail.0.0.0.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([576]). 测试sr x4 时，将scale 设置为4，但是PSNR仅为12.448

这是为什么呢

Hi can you please guide me, how i can get the LR and HR images for testing purpose? or there is some operation before testing the dataset? i mean we have to apply the operation to get x2,x3,x4 images?

I am using set5 for testing .can you please share the picture of your arranged dataset?

python main.py --dir_data IPT_DATA --pretrain tmodel/IPT_sr2.pt --data_test Set5 --scale 2 --test_only True --save_results True

Note: path of images should be like: $MODEL_PATH/benchmark/Set5/HR/XXX.png and $MODEL_PATH/benchmark/Set5/LR_bicubic/XXX.png

In the paper, the description of the encoder in Chapter 3.1. IPT architecture is:

$y_0 = [ E_{p1} + f_{p1} , E_{p2} + f_{p2} , \dots, E_{pN} + f_{pN} ],$ $q_i = k_i = v_i = LN(y_{i-1}),$ $y_i^{\prime} = MSA(q_i, k_i, v_i) + y_{i-1},$ $\cdots$

however, in the class TransformerEncoderLayer of the python file model/ipt.py, the code writes:

class TransformerEncoderLayer(nn.Module):

    def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1, no_norm = False,
                 activation="relu"):
        ...

    def with_pos_embed(self, tensor, pos):
        return tensor if pos is None else tensor + pos
    
    def forward(self, src, pos = None):
        src2 = self.norm1(src)                                         # here
        q = k = self.with_pos_embed(src2, pos)             # here
        src2 = self.self_attn(q, k, src2)                            # here
        src = src + self.dropout1(src2[0])
        src2 = self.norm2(src)
        src2 = self.linear2(self.dropout(self.activation(self.linear1(src2))))
        src = src + self.dropout2(src2)
        return src

It's more likely the formula should be: $v_i=LN([ f_{p1} , f_{p2} , \dots, f_{pN} ]),$ $y_0 = [ E_{p1} + f_{p1} , E_{p2} + f_{p2} , \dots, E_{pN} + f_{pN} ],$ $q_i = k_i = LN(y_{i-1}),$

, OR may they are equivalent ? I'm confused. Thanks.

第一阶段： Pre-training 第二阶段： Finetuning on the specific task

但在第一阶段时是要训练multi-heads，multi-tails; 训练时一个batch 只是随机选一种task 的pair 数据送入到model中，利用反向传播来更新相应的head，tail，和body；其中是不是需要设置，在训练A task时，其他 task 所对应的heads，tails是保持不变的（不会被更新的）

第二阶段：只保留相应的task的head 和tail，其他的heads和tails是直接丢弃的

这个过程想确认一下