[ACM MM 2021] Joint Implicit Image Function for Guided Depth Super-Resolution

Related tags

Deep Learning jiif
Overview

Joint Implicit Image Function for Guided Depth Super-Resolution

This repository contains the code for:

Joint Implicit Image Function for Guided Depth Super-Resolution
Jiaxiang Tang, Xiaokang Chen, Gang Zeng
ACM MM 2021

model

Installation

Environments:

  • Python >= 3.6
  • PyTorch >= 1.6.0
  • tensorboardX
  • tqdm, opencv-python, Pillow
  • NVIDIA apex (python-only build is ok.)

Data preparation

Please see data/prepare_data.md for the details.

Training

You can use the provided scripts (scripts/train*) to train models.

For example:

# train JIIF with scale = 8 on the NYU dataset.
OMP_NUM_THREADS=8 CUDA_VISIBLE_DEVICES=2 python main.py \
    --name jiif_8 --model JIIF --scale 8 \
    --sample_q 30720 --input_size 256 --train_batch 1 \
    --epoch 200 --eval_interval 10 \
    --lr 0.0001 --lr_step 60 --lr_gamma 0.2

Testing

To test the performance of the models on difference datasets, you can use the provided scripts (scripts/test*).

For example:

# test the best checkpoint on MiddleBury dataest with scale = 8
OMP_NUM_THREADS=8 CUDA_VISIBLE_DEVICES=1 python main.py \
    --test --checkpoint best \
    --name jiif_8 --model JIIF \
    --dataset Middlebury --scale 8 --data_root ./data/depth_enhance/01_Middlebury_Dataset

Pretrained models and Reproducing

We provide the pretrained models here.

To test the performance of the pretrained models, please download the corresponding models and put them under pretrained folder. Then you can use scripts/test_jiif_pretrained.sh and scripts/test_denoise_jiif_pretrained.sh to reproduce the results reported in our paper.

Citation

If you find the code useful for your research, please use the following BibTeX entry:

@article{tang2021joint,
    title        = {Joint Implicit Image Function for Guided Depth Super-Resolution},
    author       = {Jiaxiang Tang, Xiaokang Chen, Gang Zeng},
    year         = 2021,
    journal      = {arXiv preprint arXiv:2107.08717}
}

Acknowledgment

The model implementation is based on liif.

You might also like...
Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)
Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)

Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021) This repository is the official P

Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)
Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)

Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021) This repository is the official P

Repository for
Repository for "Exploring Sparsity in Image Super-Resolution for Efficient Inference", CVPR 2021

SMSR Reposity for "Exploring Sparsity in Image Super-Resolution for Efficient Inference" [arXiv] Highlights Locate and skip redundant computation in S

PyTorch implementation of Graph Convolutional Networks in Feature Space for Image Deblurring and Super-resolution, IJCNN 2021.

GCResNet PyTorch implementation of Graph Convolutional Networks in Feature Space for Image Deblurring and Super-resolution, IJCNN 2021. The code will

Monocular Depth Estimation - Weighted-average prediction from multiple pre-trained depth estimation models
Monocular Depth Estimation - Weighted-average prediction from multiple pre-trained depth estimation models

merged_depth runs (1) AdaBins, (2) DiverseDepth, (3) MiDaS, (4) SGDepth, and (5) Monodepth2, and calculates a weighted-average per-pixel absolute dept

The implemention of Video Depth Estimation by Fusing Flow-to-Depth Proposals

Flow-to-depth (FDNet) video-depth-estimation This is the implementation of paper Video Depth Estimation by Fusing Flow-to-Depth Proposals Jiaxin Xie,

Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals
Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals

LapDepth-release This repository is a Pytorch implementation of the paper "Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals" M

Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in ONNX
Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in ONNX

ONNX msg_chn_wacv20 depth completion Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20 model in

Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in Tensorflow Lite.
Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in Tensorflow Lite.

TFLite-msg_chn_wacv20-depth-completion Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model

Comments
  • the input of mlp

    the input of mlp

    Great work!

    But in model/jiif.py: 79~81

    q_guide = torch.cat([q_guide_hr, q_guide_hr - q_guide_lr], dim=-1)
    
    inp = torch.cat([q_feat, q_guide, rel_coord], dim=-1)
    

    It seems inconsistent with the equation (8) of your paper. In eq.(8), the second input feature is g_i, which is a feature sampled from the low-resolution feature map. But in the code, q_guide_hr is used, which is sampled from the high-resolution features, corresponding to g_q but not g_i.

    opened by Philipflyg 2
Owner
hawkey
nameless kiui.
hawkey
A framework for joint super-resolution and image synthesis, without requiring real training data

SynthSR This repository contains code to train a Convolutional Neural Network (CNN) for Super-resolution (SR), or joint SR and data synthesis. The met

null 76 Sep 9, 2022
RGB-D Local Implicit Function for Depth Completion of Transparent Objects

RGB-D Local Implicit Function for Depth Completion of Transparent Objects [Project Page] [Paper] Overview This repository maintains the official imple

NVIDIA Research Projects 39 Jul 20, 2022
This repository contains the code for the paper "PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization"

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization News: [2020/05/04] Added EGL rendering option for training data g

Shunsuke Saito 1.4k Sep 25, 2022
Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging

Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging This repository contains an implementation

Computational Photography Lab @ SFU 947 Sep 29, 2022
Beyond Image to Depth: Improving Depth Prediction using Echoes (CVPR 2021)

Beyond Image to Depth: Improving Depth Prediction using Echoes (CVPR 2021) Kranti Kumar Parida, Siddharth Srivastava, Gaurav Sharma. We address the pr

Kranti Kumar Parida 33 Jun 27, 2022
ICRA 2021 "Towards Precise and Efficient Image Guided Depth Completion"

PENet: Precise and Efficient Depth Completion This repo is the PyTorch implementation of our paper to appear in ICRA2021 on "Towards Precise and Effic

null 207 Sep 20, 2022
Learning Continuous Image Representation with Local Implicit Image Function

LIIF This repository contains the official implementation for LIIF introduced in the following paper: Learning Continuous Image Representation with Lo

Yinbo Chen 954 Sep 20, 2022
An unofficial personal implementation of UM-Adapt, specifically to tackle joint estimation of panoptic segmentation and depth prediction for autonomous driving datasets.

Semisupervised Multitask Learning This repository is an unofficial and slightly modified implementation of UM-Adapt[1] using PyTorch. This code primar

Abhinav Atrishi 10 Jun 1, 2022
Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation (RA-L/ICRA 2020)

Aerial Depth Completion This work is described in the letter "Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation", by Lucas

ETHZ V4RL 67 Sep 21, 2022
Code for 'Self-Guided and Cross-Guided Learning for Few-shot segmentation. (CVPR' 2021)'

SCL Introduction Code for 'Self-Guided and Cross-Guided Learning for Few-shot segmentation. (CVPR' 2021)' We evaluated our approach using two baseline

null 33 Jul 15, 2022