Thanks for your great work, but it seems to lack a related file (crop_sub_images.py) in this project for training. Could you upload this file? I would appreciate it.

Hi, thanks for the excellent work! But could you please release the dataset on Google Drive, too? I can't download the dropbox link in China... Thanks very much!

when using

mim install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cpu/torch1.10.0/index.html

I am looking to setup a space on huggingface spaces(https://huggingface.co/spaces) for this model at https://huggingface.co/spaces/akhaliq/RealBasicVSR

I was able to get the model working in colab but the space does not support cuda, is there a way around this thanks?

space code: https://huggingface.co/spaces/akhaliq/RealBasicVSR/blob/main/app.py#L4

I used the official NIQE code to evaluate the demo_000 and the result, got a unexpected result, as the niqe value of the raw video is 3.9829 while the sr video is 4.3407. I just input every frame and calculate the average value. I don't know where is wrong, as this result is totally opposite towards that in paper.

Hi. First of all, thank you very much for your project. The quality is impressive. I'm trying to train a neural network, but after every save checkpoint it starts some long process for 300 iterations. It looks like evaluating, but I couldn't find a value of 300 in the config file. I train on the REDS dataset (24k images) and that process takes longer than the training itself for 10k iterations. What is it? Is there any way to reduce this value (300)? Is it possible to disable it and what is the risk?

Example:

Hi there,thanks for the work.But getting some errors....

packages in environment at conda\envs\realbasicvsr:

Name Version Build Channel

absl-py 1.0.0 pypi_0 pypi addict 2.4.0 pypi_0 pypi blas 1.0 mkl ca-certificates 2021.10.26 haa95532_2 cachetools 4.2.4 pypi_0 pypi certifi 2021.10.8 py37haa95532_0 charset-normalizer 2.0.10 pypi_0 pypi click 7.1.2 pypi_0 pypi colorama 0.4.4 pypi_0 pypi cudatoolkit 10.1.243 h74a9793_0 freetype 2.10.4 hd328e21_0 google-auth 2.3.3 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi grpcio 1.43.0 pypi_0 pypi idna 3.3 pypi_0 pypi imageio 2.13.5 pypi_0 pypi importlib-metadata 4.10.0 pypi_0 pypi intel-openmp 2021.4.0 haa95532_3556 jpeg 9b hb83a4c4_2 libpng 1.6.37 h2a8f88b_0 libtiff 4.2.0 hd0e1b90_0 libuv 1.40.0 he774522_0 libwebp 1.2.0 h2bbff1b_0 lmdb 1.3.0 pypi_0 pypi lz4-c 1.9.3 h2bbff1b_1 markdown 3.3.6 pypi_0 pypi mkl 2021.4.0 haa95532_640 mkl-service 2.4.0 py37h2bbff1b_0 mkl_fft 1.3.1 py37h277e83a_0 mkl_random 1.2.2 py37hf11a4ad_0 mmcv-full 1.4.2 pypi_0 pypi mmedit 0.12.0 pypi_0 pypi model-index 0.1.11 pypi_0 pypi networkx 2.6.3 pypi_0 pypi ninja 1.10.2 py37h559b2a2_3 numpy 1.21.2 py37hfca59bb_0 numpy-base 1.21.2 py37h0829f74_0 oauthlib 3.1.1 pypi_0 pypi olefile 0.46 py37_0 opencv-python-headless 4.5.4.60 pypi_0 pypi openmim 0.1.5 pypi_0 pypi openssl 1.1.1l h2bbff1b_0 ordered-set 4.0.2 pypi_0 pypi packaging 21.3 pypi_0 pypi pandas 1.3.5 pypi_0 pypi pillow 8.4.0 py37hd45dc43_0 pip 21.2.4 py37haa95532_0 protobuf 3.19.1 pypi_0 pypi pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pyparsing 3.0.6 pypi_0 pypi python 3.7.11 h6244533_0 python-dateutil 2.8.2 pypi_0 pypi pytorch 1.7.1 py3.7_cuda101_cudnn7_0 pytorch pytz 2021.3 pypi_0 pypi pywavelets 1.2.0 pypi_0 pypi pyyaml 6.0 pypi_0 pypi regex 2021.11.10 pypi_0 pypi requests 2.27.1 pypi_0 pypi requests-oauthlib 1.3.0 pypi_0 pypi rsa 4.8 pypi_0 pypi scikit-image 0.19.1 pypi_0 pypi scipy 1.7.3 pypi_0 pypi setuptools 58.0.4 py37haa95532_0 six 1.16.0 pyhd3eb1b0_0 sqlite 3.37.0 h2bbff1b_0 tabulate 0.8.9 pypi_0 pypi tensorboard 2.7.0 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.1 pypi_0 pypi tifffile 2021.11.2 pypi_0 pypi tk 8.6.11 h2bbff1b_0 torchaudio 0.7.2 py37 pytorch torchvision 0.8.2 py37_cu101 pytorch typing_extensions 3.10.0.2 pyh06a4308_0 urllib3 1.26.7 pypi_0 pypi vc 14.2 h21ff451_1 vs2015_runtime 14.27.29016 h5e58377_2 werkzeug 2.0.2 pypi_0 pypi wheel 0.37.0 pyhd3eb1b0_1 wincertstore 0.2 py37haa95532_2 xz 5.2.5 h62dcd97_0 yapf 0.32.0 pypi_0 pypi zipp 3.7.0 pypi_0 pypi zlib 1.2.11 h8cc25b3_4 zstd 1.4.9 h19a0ad4_0

For pictures I ran the test code :

(realbasicvsr) C:\RealBasicVSR>python inference_realbasicvsr.py configs/realbasicvsr_x4.py checkpoints/RealBasicVSR_x4.pth data/demo_000 results/demo_000 2022-01-07 06:36:34,070 - mmedit - INFO - load checkpoint from torchvision path: torchvision://vgg19 load checkpoint from local path: checkpoints/RealBasicVSR_x4.pth

it did nothing.

For video I ran the test code setting the --max_seq_len=2 :

(realbasicvsr) C:\RealBasicVSR>python inference_realbasicvsr.py configs/realbasicvsr_x4.py checkpoints/RealBasicVSR_x4.pth data/demo_001.mp4 results/demo_001.mp4 --max_seq_len=2 --fps=12.5 2022-01-07 06:38:02,236 - mmedit - INFO - load checkpoint from torchvision path: torchvision://vgg19 load checkpoint from local path: checkpoints/RealBasicVSR_x4.pth Traceback (most recent call last): File "inference_realbasicvsr.py", line 144, in main() File "inference_realbasicvsr.py", line 130, in main cv2.destroyAllWindows() cv2.error: OpenCV(4.5.4) D:\a\opencv-python\opencv-python\opencv\modules\highgui\src\window.cpp:1268: error: (-2:Unspecified error) The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Cocoa support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script in function 'cvDestroyAllWindows'

it gave this error.

For video I ran the test code default :

(realbasicvsr) C:\RealBasicVSR>python inference_realbasicvsr.py configs/realbasicvsr_x4.py checkpoints/RealBasicVSR_x4.pth data/demo_001.mp4 results/demo_001.mp4 --fps=12.5 2022-01-07 06:40:14,850 - mmedit - INFO - load checkpoint from torchvision path: torchvision://vgg19 load checkpoint from local path: checkpoints/RealBasicVSR_x4.pth Traceback (most recent call last): File "inference_realbasicvsr.py", line 144, in main() File "inference_realbasicvsr.py", line 117, in main outputs = model(inputs, test_mode=True)['output'].cpu() File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\mmcv\runner\fp16_utils.py", line 98, in new_func return old_func(*args, **kwargs) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\mmedit\models\restorers\srgan.py", line 95, in forward return self.forward_test(lq, gt, **kwargs) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\mmedit\models\restorers\real_esrgan.py", line 211, in forward_test output = _model(lq) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\mmedit\models\backbones\sr_backbones\real_basicvsr_net.py", line 87, in forward outputs = self.basicvsr(lqs) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\mmedit\models\backbones\sr_backbones\basicvsr_net.py", line 126, in forward flows_forward, flows_backward = self.compute_flow(lrs) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\mmedit\models\backbones\sr_backbones\basicvsr_net.py", line 98, in compute_flow flows_backward = self.spynet(lrs_1, lrs_2).view(n, t - 1, 2, h, w) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\mmedit\models\backbones\sr_backbones\basicvsr_net.py", line 346, in forward input=self.compute_flow(ref, supp), File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\mmedit\models\backbones\sr_backbones\basicvsr_net.py", line 315, in compute_flow ], 1)) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\mmedit\models\backbones\sr_backbones\basicvsr_net.py", line 420, in forward return self.basic_module(tensor_input) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\torch\nn\modules\container.py", line 117, in forward input = module(input) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\mmcv\cnn\bricks\conv_module.py", line 201, in forward x = self.conv(x) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\torch\nn\modules\conv.py", line 423, in forward return self._conv_forward(input, self.weight) File "C:\Users\breakmycurse.conda\envs\realbasicvsr\lib\site-packages\torch\nn\modules\conv.py", line 420, in _conv_forward self.padding, self.dilation, self.groups) RuntimeError: CUDA out of memory. Tried to allocate 4.35 GiB (GPU 0; 11.00 GiB total capacity; 4.37 GiB already allocated; 2.46 GiB free; 7.07 GiB reserved in total by PyTorch)

it gave an oom error.

System is Win 10 64 bit, 1080 ti =11 gb. model is in the right folder and the environment is done with conda with the given commands in order.

Exception has occurred: RuntimeError RealBasicVSR: PerceptualLoss: storage has wrong size: expected 0 got 1728

During handling of the above exception, another exception occurred:

File "D:\Users......\RealBasicVSR-master\realbasicvsr\models\restorers\real_basicvsr.py", line 65, in init super().init(generator, discriminator, gan_loss, pixel_loss,

During handling of the above exception, another exception occurred:

File "D:\Users......\RealBasicVSR-master\realbasicvsr\models\builder.py", line 20, in build return build_from_cfg(cfg, registry, default_args) File "D:\Users......\RealBasicVSR-master\realbasicvsr\models\builder.py", line 58, in build_model return build(cfg, MODELS, dict(train_cfg=train_cfg, test_cfg=test_cfg)) File "D:\Users......\RealBasicVSR-master\inference_realbasicvsr.py", line 67, in init_model model = build_model(config.model, test_cfg=config.test_cfg) File "D:\Users......\RealBasicVSR-master\inference_realbasicvsr.py", line 81, in main model = init_model(args.config, args.checkpoint) File "D:\Users......\RealBasicVSR-master\inference_realbasicvsr.py", line 149, in main()

Hi, thanks for your wonderful work. In the paper, you use cb loss, but in the codes, you use L1 loss in the config file. Which one is the right. Have you ever tried to modify the model for x1? Looking forward for your return. Thank you.

Hi. I am installed that:

conda create -n vsr3 python=3.7 -y
conda activate vsr3
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.1 -c pytorch -y
conda install -c omnia openmm -y
#conda install -c esri mmcv-full -y
pip install mmcv-full==1.3.17 -f https://download.openmmlab.com/mmcv/dist/11.1/torch1.10.0/index.html
python3 -m pip install mmedit

Then i run:

python inference_realbasicvsr.py configs/realbasicvsr_x4.py checkpoints/RealBasicVSR_x4.pth data/demo_001.mp4 results/demo_001.mp4 --fps=12.5

Then get an error:

Traceback (most recent call last):
  File "inference_realbasicvsr.py", line 148, in <module>
    main()
  File "inference_realbasicvsr.py", line 80, in main
    model = init_model(args.config, args.checkpoint)
  File "inference_realbasicvsr.py", line 64, in init_model
    config.model.pretrained = None
  File "/home/alex/anaconda3/envs/vsr3/lib/python3.7/site-packages/mmcv/utils/config.py", line 507, in __getattr__
    return getattr(self._cfg_dict, name)
  File "/home/alex/anaconda3/envs/vsr3/lib/python3.7/site-packages/mmcv/utils/config.py", line 48, in __getattr__
    raise ex
AttributeError: 'ConfigDict' object has no attribute 'model'

Also i check in jupyter notebook the object:

config = mmcv.Config.fromfile(config)

And config contains:

Config (path: configs/realbasicvsr_x4.py): {'argparse': <module 'argparse' from '/home/alex/anaconda3/envs/vsr3/lib/python3.7/argparse.py'>, 'os': <module 'os' from '/home/alex/anaconda3/envs/vsr3/lib/python3.7/os.py'>, 'osp': <module 'posixpath' from '/home/alex/anaconda3/envs/vsr3/lib/python3.7/posixpath.py'>, 'sys': <module 'sys' (built-in)>, 'Pool': <bound method BaseContext.Pool of <multiprocessing.context.DefaultContext object at 0x7f8f1c640d10>>, 'cv2': <module 'cv2' from '/home/alex/anaconda3/envs/vsr3/lib/python3.7/site-packages/cv2/__init__.py'>, 'mmcv': <module 'mmcv' from '/home/alex/anaconda3/envs/vsr3/lib/python3.7/site-packages/mmcv/__init__.py'>, 'np': <module 'numpy' from '/home/alex/anaconda3/envs/vsr3/lib/python3.7/site-packages/numpy/__init__.py'>, 'worker': <function worker at 0x7f8e83b0a170>, 'extract_subimages': <function extract_subimages at 0x7f8e83b0a200>, 'main_extract_subimages': <function main_extract_subimages at 0x7f8e83b0a440>, 'parse_args': <function parse_args at 0x7f8e83b0a050>}

When training video material, the memory and video memory resources are always insufficient. Is there any parameter to solve this problem? almost 1min .mp4 25fps, 3MB, running on the 12GBRAM 12GvRam, will meet resources lack or how should I deal with the input video before to more easily run the code

I want an information about Colab Demo environment specification. I just know RAM and DISK capacity. Can you notice the specification about Colab Goolge Compute Engine GPU ?(Name and any other information).

Hello @ckkelvinchan , I want to retraining this model, I have a problem for about the config file UDM10 folder structure. I don't know that how to construct this folder. is it put every folder blurx4 file to merge a folder like this is correct?

I wonder that what kind of dataset you use to train the checkpoint (RealBasicVSRx4.pth) that you share through the gg drive? Is it REDS_bicubic or any else?

Does anybody try to convert this model to ScriptModule by torch.jit.trace, we finally get it, but given the same input, the generated script module get the wrong result. During converting there are some messages like below:

/home/hermanhe/.local/lib/python3.10/site-packages/torch/functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2894.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/home/hermanhe/.local/lib/python3.10/site-packages/mmedit/models/restorers/srgan.py:94: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if test_mode:
/home/hermanhe/.local/lib/python3.10/site-packages/mmedit/models/backbones/sr_backbones/real_basicvsr_net.py:83: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if torch.mean(torch.abs(residues)) < self.dynamic_refine_thres:
/home/hermanhe/.local/lib/python3.10/site-packages/mmedit/models/backbones/sr_backbones/basicvsr_net.py:118: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert h >= 64 and w >= 64, (
/home/hermanhe/.local/lib/python3.10/site-packages/mmedit/models/backbones/sr_backbones/basicvsr_net.py:73: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if lrs.size(1) % 2 == 0:
/home/hermanhe/.local/lib/python3.10/site-packages/mmedit/models/backbones/sr_backbones/basicvsr_net.py:334: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  w_up = w if (w % 32) == 0 else 32 * (w // 32 + 1)
/home/hermanhe/.local/lib/python3.10/site-packages/mmedit/models/backbones/sr_backbones/basicvsr_net.py:335: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  h_up = h if (h % 32) == 0 else 32 * (h // 32 + 1)
/home/hermanhe/.local/lib/python3.10/site-packages/mmedit/models/backbones/sr_backbones/basicvsr_net.py:335: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  h_up = h if (h % 32) == 0 else 32 * (h // 32 + 1)
/home/hermanhe/.local/lib/python3.10/site-packages/mmedit/models/backbones/sr_backbones/basicvsr_net.py:296: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  flow = ref[0].new_zeros(n, 2, h // 32, w // 32)
/home/hermanhe/.local/lib/python3.10/site-packages/mmedit/models/common/flow_warp.py:27: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if x.size()[-2:] != flow.size()[1:3]:
/home/hermanhe/.local/lib/python3.10/site-packages/mmedit/models/common/flow_warp.py:41: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  grid_flow_x = 2.0 * grid_flow[:, :, :, 0] / max(w - 1, 1) - 1.0
/home/hermanhe/.local/lib/python3.10/site-packages/mmedit/models/common/flow_warp.py:42: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  grid_flow_y = 2.0 * grid_flow[:, :, :, 1] / max(h - 1, 1) - 1.0
/home/hermanhe/.local/lib/python3.10/site-packages/mmedit/models/backbones/sr_backbones/basicvsr_net.py:352: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  flow[:, 0, :, :] *= float(w) / float(w_up)
/home/hermanhe/.local/lib/python3.10/site-packages/mmedit/models/backbones/sr_backbones/basicvsr_net.py:353: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  flow[:, 1, :, :] *= float(h) / float(h_up)
/home/hermanhe/.local/lib/python3.10/site-packages/mmedit/models/backbones/sr_backbones/basicvsr_net.py:132: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if i < t - 1:  # no warping required for the last timestep

Any advice about getting the right script module? Thanks!