Edge-oriented Convolution Block for Real-time Super Resolution on Mobile Devices, ACM Multimedia 2021

Overview

Codes for ECBSR

Edge-oriented Convolution Block for Real-time Super Resolution on Mobile Devices
Xindong Zhang, Hui Zeng, Lei Zhang
ACM Multimedia 2021

Codes

An older version implemented based on EDSR is place on /legacy folder. For more details, please refer to /legacy/README.md. The following is the lighten version implemented by us.

Dependencies & Installation

Please refer to the following simple steps for installation.

git clone https://github.com/xindongzhang/ECBSR.git
cd ECBSR
pip install -r requirements.txt

Training and benchmarking data can be downloaded from DIV2K and benchmark, respectively. Thanks for excellent work by EDSR.

Training & Testing

You could also try less/larger batch-size, if there are limited/enough hardware resources in your GPU-server. ECBSR is trained and tested with colors=1, e.g Y channel out of Ycbcr.

cd ECBSR

## ecbsr-m4c8-x2-prelu(you can revise the parameters of the yaml-config file accordding to your environments)
python train.py --config ./configs/ecbsr_x2_m4c8_prelu.yml

## ecbsr-m4c8-x4-prelu
python train.py --config ./configs/ecbsr_x4_m4c8_prelu.yml

## ecbsr-m4c16-x2-prelu
python train.py --config ./configs/ecbsr_x2_m4c16_prelu.yml

## ecbsr-m4c16-x4-prelu
python train.py --config ./configs/ecbsr_x4_m4c16_prelu.yml

Hardware deployment

Frontend conversion

We provide convertor for model conversion to different frontend, e.g. onnx/pb/tflite. We currently developed and tested the model with only one-channel(Y out of Ycbcr). Since the internal data-layout are quite different between tf(NHWC) and pytorch(NCHW), espetially for the pixelshuffle operation. Care must be taken to handle the data-layout, if you want to extend the pytorch-based training framework to RGB input data and deploy it on tensorflow. Follow are the demo scripts for model conversion to specific frontend:

## convert the trained pytorch model to onnx with plain-topology.
python convert.py --config xxx.yml --target_frontend onnx --output_folder XXX --inp_n 1 --inp_c 1 --inp_h 270 --inp_w 480

## convert the trained pytorch model to pb-1.x with plain-topology.
python convert.py --config xxx.yml --target_frontend pb-1.x --output_folder XXX --inp_n 1 --inp_c 1 --inp_h 270 --inp_w 480

## convert the trained pytorch model to pb-ckpt with plain-topology
python convert.py --config xxx.yml --target_frontend pb-ckpt --output_folder XXX --inp_n 1 --inp_c 1 --inp_h 270 --inp_w 480

AI-Benchmark

You can download the newest version of evaluation tool from AI-Benchmark. Then you can install the app via ADB tools,

adb install -r [name-of-ai-benchmar].apk

MNN (Come soon!)

For universal CPU & GPU of mobile hardware implementation.

RKNN (Come soon!)

For NPU inplementation of Rockchip hardware, e.g. RK3399Pro/RK1808.

MiniNet (Come soon!)

A super light-weight CNN inference framework implemented by us, with only conv-3x3, element-wise op, ReLU(PReLU) activations, and pixel-shuffle for common super resolution task. For more details, please refer to /ECBSR/deploy/mininet

Quantization tools (Come soon!)

For fixed-arithmetic quantization of image super resolution.

Citation


@article{zhang2021edge,
  title={Edge-oriented Convolution Block for Real-time Super Resolution on Mobile Devices},
  author={Zhang, Xindong and Zeng, Hui and Zhang, Lei},
  booktitle={Proceedings of the 29th ACM International Conference on Multimedia (ACM MM)},
  year={2021}
}

Acknowledgement

Thanks EDSR for the pioneering work and excellent codebase! The implementation integrated with EDSR is placed on /legacy

Comments
  • 训练时loss波动异常

    训练时loss波动异常

    感谢作者的贡献!我正尝试训练3通道图片,目前遇到几个问题: 1、当训练ECBSR-M4C16时,训练基本正常,但是psnr比论文中低不少,我用了IMDN代码中的测试方法,将图片转成Ycbcr之后,针对Y通道计算psnr和ssim,但是测试结果在set14上psnr只有31.2x左右,在set5上psnr只有34.96左右。如果直接计算3通道图片的psnr和ssim,结果会更低。 2、当训练ECBSR-M16C64时,训练loss波动异常,loss会不定期的变得非常大:

    ##===========Epoch: 15=============##
    Epoch:015, 03200/51200, loss: 4.5040, time: 35.135
    Epoch:015, 06400/51200, loss: 4.3543, time: 30.960
    Epoch:015, 09600/51200, loss: 4.3308, time: 30.984
    Epoch:015, 12800/51200, loss: 4.3714, time: 30.946
    Epoch:015, 16000/51200, loss: 4.3591, time: 31.149
    Epoch:015, 19200/51200, loss: 498924.5582, time: 31.234
    Epoch:015, 22400/51200, loss: 529279.5449, time: 31.021
    Epoch:015, 25600/51200, loss: 463260.1697, time: 31.044
    Epoch:015, 28800/51200, loss: 411850.2629, time: 30.985
    Epoch:015, 32000/51200, loss: 370703.8992, time: 31.002
    Epoch:015, 35200/51200, loss: 337032.6480, time: 31.208
    Epoch:015, 38400/51200, loss: 308969.3531, time: 31.480
    Epoch:015, 41600/51200, loss: 285220.5471, time: 31.479
    Epoch:015, 44800/51200, loss: 264861.9574, time: 30.958
    Epoch:015, 48000/51200, loss: 247216.6885, time: 30.925
    Epoch:015, 51200/51200, loss: 231775.5556, time: 31.046
    [set5-X2], PSNR/SSIM: 7.86/0.0751 (Best: 32.21/0.9010, Epoch: 14/14)
    [set14-X2], PSNR/SSIM: 9.28/0.1133 (Best: 28.74/0.8311, Epoch: 14/14)
    
    

    隔几个epoch后loss就会变得异常大,但是ECBSR-M4C16不会。其他训练参数都是默认的,训练数据是DIV2K,只修改了训练时的colors为3,然后forward里y = self.backbone(x) + x进行相应的修改。目前没看出来哪里的问题。

    opened by buptlj 8
  • DLC模型转换时pixel shuffle GPU/DSP runtime不支持

    DLC模型转换时pixel shuffle GPU/DSP runtime不支持

    你好,最近我将SR模型转换为SNPE DLC模型的时候出现GPU/DSP runtime不支持pixelshuffle的问题,请教一下如何解决? 我的具体环境如下: 芯片:865 snpe版本:1.55.0 系统:aarch64_ubuntu1804_rb5

    具体问题表述:将pytorch模型转换为onnx,然后将onnx模型转成dlc模型的时候,snpe会自动将pixelshuffle操作分解为reshape、permute等操作,在模型输入尺寸超过320*320的时候转换模型就会出现dsp/gpu runtime不支持的情况。

    在您的论文中有提到在DSP上实现了1080p的实时超分,请问一下您有没有遇到这个问题?然后您是怎么解决的呢? 非常感谢!!!

    opened by JimmyLauren 7
  • About color training

    About color training

    Thank you for your work. default colors set as 1, when I tried to train 3(RGB) image, error occurred... maybe x need concat more times?

    train the model from scratch! ##===========Epoch: 0=============## Traceback (most recent call last): File "/code/ECBSR/train.py", line 143, in sr = model(lr) File "/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/code/ECBSR/models/ecbsr.py", line 28, in forward y = self.backbone(x) + x RuntimeError: The size of tensor a (12) must match the size of tensor b (3) at non-singleton dimension 1

    opened by mychina75 5
  • Inference issue

    Inference issue

    I used the pt file that came out through the learning process you provided. But, the results converted to onnx and executed using pt files are different. The results of the same pre-processing and the same post-processing are different.

    Here is Picture [pytorch inference] nisan

    [onnx inference] output_3 Looking at it, the onnx model shows better results. If it's the opposite situation, I'll understand, but I don't understand that onnx is a better result.

    opened by puppy9207 4
  • About convert.py

    About convert.py

    What is this file used for? Convolution parameter fusion in the inference stage is not directly read from the training model, but after the training model is converted by convert.py, then inference is performed?

    opened by typhoonlee 3
  • Where is the inference moduel?

    Where is the inference moduel?

    Thank you very much for your work, it is very helpful to my study! I did not find the code for the inference where shows how to implement the re-parameterizable building during testing?

    opened by typhoonlee 3
  • merge sr Ychannel & CbCr to rgb

    merge sr Ychannel & CbCr to rgb

    i wirte the code to merge Ychannel by sr with CbCr upsampled by bicubic, but the final rgb mat contains many blue or green or red pixels.

    将超分后的Y通道和直接只用bicubic插值上采样的CbCr转换为rgb格式图片保存,发现有很多红绿蓝的像素点,怎么解决?能分享一下代码吗

    opened by fangichao 2
  • About the ycbcr and quantization

    About the ycbcr and quantization

    Hi, it is a good job to incorporate the Re-parameterization into SR model. I still have some questions about the code and paper.

    • In Section 2, Computation Reduction, the paper says

    However, quantization of SR models often can hardly maintain the SR quality because of the precision requirement of pixel-wise prediction

    The original RepVGG model does not perform well in quantization. How about the error of quantization in low-level Re-parameterization quantization? It seems that there is no further explanation or results for the quantization error in the paper.

    • Common SR models are trained on RGB channel and tested on Matlab version Y of YCbCr. However, in the provided code https://github.com/xindongzhang/ECBSR/blob/eac2ff12b4fa24a24c99b6e32deff59d0ad9e3d4/datas/div2k.py#L10 https://github.com/xindongzhang/ECBSR/blob/eac2ff12b4fa24a24c99b6e32deff59d0ad9e3d4/datas/div2k.py#L54 How about the error between different versions of Ycbcr? Thanks!
    opened by JuZiSYJ 2
  • some questions about the ECB block structure diagram in the paper

    some questions about the ECB block structure diagram in the paper

    Thanks for your great work. I have some questions about the ECB block structure diagram in the paper. The ECB block in the paper has conv-33 and sobel-Dy branch but conv-11 and sobel-Dy int the code. Is there someting wrong in the structure diagram (3.1 b) of the paper?

    opened by zhangliukun 1
  • About evaluation result

    About evaluation result

    Hi~ one more question.. As you know, the y channel super resolution result has to merge back to all channels' image, which are simple resized up. so, when I compute the PSNR using the color image, result is not good as the evaluation result during training. the x3 model test using DIV2K val DB. the PSNR result drop from 31.0x to 29.x...

    anything wrong? thank you.

    opened by mychina75 1
  • ecb统计计算量为0

    ecb统计计算量为0

    请问为什么我用 flops, params = profile(model, inputs=(input,)) print('计算量:{}'.format(flops)) print('参数量:{}'.format(params))

    统计ecb 参数量和计算量显示是0呢? 换了其他统计方法也是.

    请问你们是用什么工具统计参数量和计算量的

    opened by yumulinfeng1 0
  • color=3 or color=1 &&recover pictures

    color=3 or color=1 &&recover pictures

    If I don't need deploy the model to mobile phone, only want to remcover SR pictures from LR ,which are RGB pictures, what should Ido? How to train and recover? Thanks!

    opened by wxslby 0
Owner
xindong zhang
xindong zhang
This repo is official PyTorch implementation of MobileHumanPose: Toward real-time 3D human pose estimation in mobile devices(CVPRW 2021).

Github Code of "MobileHumanPose: Toward real-time 3D human pose estimation in mobile devices" Introduction This repo is official PyTorch implementatio

Choi Sang Bum 203 Jan 5, 2023
[ACM MM 2021] Joint Implicit Image Function for Guided Depth Super-Resolution

Joint Implicit Image Function for Guided Depth Super-Resolution This repository contains the code for: Joint Implicit Image Function for Guided Depth

hawkey 78 Dec 27, 2022
BED: A Real-Time Object Detection System for Edge Devices

BED: A Real-Time Object Detection System for Edge Devices About this project Thi

Data Analytics Lab at Texas A&M University 44 Nov 18, 2022
Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data

Real-ESRGAN Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data Ported from https://github.com/xinntao/Real-ESRGAN Depend

Holy Wu 44 Dec 27, 2022
Real-CUGAN - Real Cascade U-Nets for Anime Image Super Resolution

Real Cascade U-Nets for Anime Image Super Resolution 中文 | English ?? Real-CUGAN

tarsin 111 Dec 28, 2022
MoCoPnet - Deformable 3D Convolution for Video Super-Resolution

Deformable 3D Convolution for Video Super-Resolution Pytorch implementation of l

Xinyi Ying 28 Dec 15, 2022
Diverse Branch Block: Building a Convolution as an Inception-like Unit

Diverse Branch Block: Building a Convolution as an Inception-like Unit (PyTorch) (CVPR-2021) DBB is a powerful ConvNet building block to replace regul

null 253 Dec 24, 2022
Lyapunov-guided Deep Reinforcement Learning for Stable Online Computation Offloading in Mobile-Edge Computing Networks

PyTorch code to reproduce LyDROO algorithm [1], which is an online computation offloading algorithm to maximize the network data processing capability subject to the long-term data queue stability and average power constraints. It applies Lyapunov optimization to decouple the multi-stage stochastic MINLP into deterministic per-frame MINLP subproblems and solves each subproblem via DROO algorithm. It includes:

Liang HUANG 87 Dec 28, 2022
Fast and Context-Aware Framework for Space-Time Video Super-Resolution (VCIP 2021)

Fast and Context-Aware Framework for Space-Time Video Super-Resolution Preparation Dependencies PyTorch 1.2.0 CUDA 10.0 DCNv2 cd model/DCNv2 bash make

Xueheng Zhang 1 Mar 29, 2022
YolactEdge: Real-time Instance Segmentation on the Edge

YolactEdge, the first competitive instance segmentation approach that runs on small edge devices at real-time speeds. Specifically, YolactEdge runs at up to 30.8 FPS on a Jetson AGX Xavier (and 172.7 FPS on an RTX 2080 Ti) with a ResNet-101 backbone on 550x550 resolution images.

Haotian Liu 1.1k Jan 6, 2023
Implementation for the paper 'YOLO-ReT: Towards High Accuracy Real-time Object Detection on Edge GPUs'

YOLO-ReT This is the original implementation of the paper: YOLO-ReT: Towards High Accuracy Real-time Object Detection on Edge GPUs. Prakhar Ganesh, Ya

null 69 Oct 19, 2022
A DNN inference latency prediction toolkit for accurately modeling and predicting the latency on diverse edge devices.

Note: This is an alpha (preview) version which is still under refining. nn-Meter is a novel and efficient system to accurately predict the inference l

Microsoft 244 Jan 6, 2023
CenterFace(size of 7.3MB) is a practical anchor-free face detection and alignment method for edge devices.

CenterFace Introduce CenterFace(size of 7.3MB) is a practical anchor-free face detection and alignment method for edge devices. Recent Update 2019.09.

StarClouds 1.2k Dec 21, 2022
A light and fast one class detection framework for edge devices. We provide face detector, head detector, pedestrian detector, vehicle detector......

A Light and Fast Face Detector for Edge Devices Big News: LFD, which is a big update of LFFD, now is released (2021.03.09). It is strongly recommended

YonghaoHe 1.3k Dec 25, 2022
A framework for joint super-resolution and image synthesis, without requiring real training data

SynthSR This repository contains code to train a Convolutional Neural Network (CNN) for Super-resolution (SR), or joint SR and data synthesis. The met

null 83 Jan 1, 2023
Official implementation of the paper 'Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolution'

DASR Paper Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolution Jie Liang, Hui Zeng, and Lei Zhang. In arxiv preprint. Abs

null 81 Dec 28, 2022
[2021 MultiMedia] CONQUER: Contextual Query-aware Ranking for Video Corpus Moment Retrieval

CONQUER: Contexutal Query-aware Ranking for Video Corpus Moment Retreival PyTorch implementation of CONQUER: Contexutal Query-aware Ranking for Video

Hou zhijian 23 Dec 26, 2022
CondenseNet: Light weighted CNN for mobile devices

CondenseNets This repository contains the code (in PyTorch) for "CondenseNet: An Efficient DenseNet using Learned Group Convolutions" paper by Gao Hua

Shichen Liu 690 Nov 30, 2022
Monitor your ML jobs on mobile devices📱, especially for Google Colab / Kaggle

TF Watcher TF Watcher is a simple to use Python package and web app which allows you to monitor ?? your Machine Learning training or testing process o

Rishit Dagli 54 Nov 1, 2022