The official PyTorch implementation of the paper: *Xili Dai, Xiaojun Yuan, Haigang Gong, Yi Ma. "Fully Convolutional Line Parsing." *.

Related tags

F-Clip
Overview

F-Clip — Fully Convolutional Line Parsing

This repository contains the official PyTorch implementation of the paper: *Xili Dai, Xiaojun Yuan, Haigang Gong, Yi Ma. "Fully Convolutional Line Parsing." *.

Introduction

Our method (F-Clip) is a simple and effective neural network for detecting the line from a given image and video. It outperforms the previous state-of-the-art wireframe and line detectors by a large margin on both accuracy and speed. We hope that this repository serves as a new reproducible baseline for future researches in this area.

Main results

The accuracy and speed trade-off among most recent wireframe detection methods on ShanghaiTech dataset

Qualitative Measures

More random sampled results can be found in the paper.

Quantitative Measures

The following table reports the performance metrics of several wireframes and line detectors on the ShanghaiTech dataset.

Reproducing Results

Installation

For the ease of reproducibility, you are suggested to install miniconda (or anaconda if you prefer) before following executing the following commands.

git clone https://github.com/Delay-Xili/F-Clip
cd F-Clip
conda create -y -n fclip
source activate fclip
# Replace cudatoolkit=10.1 with your CUDA version: https://pytorch.org/
conda install -y pytorch cudatoolkit=10.1 -c pytorch
conda install -y pyyaml docopt matplotlib scikit-image opencv
mkdir data logs post

Testing Pre-trained Models

You can download our reference 6 pre-trained models HG1_D2, HG1_D3, HG1, HG2, HG2_LB, and HR from Google Drive. Those models were trained with their corresponding settings config/fclip_xxx.yaml.
To generate wireframes on the validation dataset with the pretrained model, execute

python test.py -d 0 -i <directory-to-storage-results> config/fclip_xxx.yaml <path-to-xxx-ckpt-file> shanghaiTech/york <path-to-validation-set>

Detect Wireframes for Your Own Images or Videos

To test F-Clip on your own images or videos, you need to download the pre-trained models and execute

CUDA_VISIBLE_DEVICES=0 python demo.py <path-to-image-or-video> --model HR --output_dir logs/demo_result --ckpt <path-to-pretrained-pth> --display True

Here, --output_dir is specifying the directory where the results will store, and you can specify --display to see the results on time.

Downloading the Processed Dataset

You can download the processed dataset wireframe.zip and york.zip manually from Google Drive (link1, link2).

Processing the Dataset

Optionally, you can pre-process (e.g., generate heat maps, do data augmentation) the dataset from scratch rather than downloading the processed one.

dataset/wireframe.py data/wireframe_raw data/wireframe
dataset/wireframe_line.py data/wireframe_raw data/wireframe

Evaluation

To evaluate the sAP (recommended) of all your checkpoints under logs/, execute

python eval-sAP.py logs/*/npz/*

MATLAB is required for APH evaluation and matlab should be under your $PATH. The parallel computing toolbox is highly suggested due to the usage of parfor. After post processing, execute

python eval-APH.py pth/to/input/npz pth/to/output/dir

Due to the usage of pixel-wise matching, the evaluation of APH may take up to an hour depending on your CPUs. See the source code of eval-sAP.py, eval-APH.py, and FClip/postprocess.py for more details on evaluation.

Training

To train the neural network on GPU 0 (specified by -d 0) with the different 6 parameters, execute

python train.py -d 0 -i HG1_D2 config/fclip_HG1_D2.yaml
python train.py -d 0 -i HG1_D3 config/fclip_HG1_D3.yaml
python train.py -d 0 -i HG1 config/fclip_HG1.yaml
python train.py -d 0 -i HG2 config/fclip_HG2.yaml
python train.py -d 0 -i HG2_LB config/fclip_HG2_LB.yaml
python train.py -d 0 -i HR config/fclip_HR.yaml

Citation

If you find F-Clip useful in your research, please consider citing:

@inproceedings{dai2021fully,
 author={Xili Dai, Xiaojun Yuan, Haigang Gong, and Yi Ma},
 title={Fully Convolutional Line Parsing},
 journal={CoRR},
 year={2021}
}
Issues
  • unable to build models due to key error in M.dict

    unable to build models due to key error in M.dict

    Hi - wanted to test out the models but in trying to run the demo, it's unable to build any of the models (HR, HG2) due to this key error: fclip_multihead_key_error

    I'm working on debugging but if you have any insight, please let me know. Thanks!

    opened by lessw2020 5
  • The inference takes 30 seconds per image on CPU and the precision is a little bad.

    The inference takes 30 seconds per image on CPU and the precision is a little bad.

    Env:

    • Ubuntu 20.04
    • Python 3.8
    • PyTorch 1.8.1

    CPU:

    Architecture:                    x86_64
    CPU op-mode(s):                  32-bit, 64-bit
    Byte Order:                      Little Endian
    Address sizes:                   39 bits physical, 48 bits virtual
    CPU(s):                          4
    On-line CPU(s) list:             0-3
    Thread(s) per core:              1
    Core(s) per socket:              4
    Socket(s):                       1
    NUMA node(s):                    1
    Vendor ID:                       GenuineIntel
    CPU family:                      6
    Model:                           94
    Model name:                      Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
    Stepping:                        3
    CPU MHz:                         3503.549
    CPU max MHz:                     3600.0000
    CPU min MHz:                     800.0000
    

    Inference log:

    $ CUDA_VISIBLE_DEVICES= python demo.py /data/ --model HG2 --output_dir logs/demo_result --ckpt /fclip_ckpt/ckpt/HG2/checkpoint.pth.tar --display False
    Namespace(camid=0, ckpt='/fclip_ckpt/ckpt/HG2/checkpoint.pth.tar', display=True, img_glob='*.png', input='/data/', model='HG2', output_dir='logs/demo_result', skip=1, waitkey=1)
    logs/demo_result
    ==> Loading video.
    [ERROR:0] global /tmp/pip-req-build-rcdoave0/opencv/modules/videoio/src/cap.cpp (160) open VIDEOIO(CV_IMAGES): raised OpenCV exception:
    
    OpenCV(4.5.2) /tmp/pip-req-build-rcdoave0/opencv/modules/videoio/src/cap_images.cpp:253: error: (-5:Bad argument) CAP_IMAGES: can't find starting number (in the name of file): /data/ in function 'icvExtractPattern'
    
    
    ==> Processing Image Directory Input.
    ==> Successfully loaded video.
    Using device:  cpu
    0-th head, head type raw, head output 2
    1-th head, head type raw, head output 2
    2-th head, head type raw, head output 1
    3-th head, head type raw, head output 1
    0-th head, head type raw, head output 2
    1-th head, head type raw, head output 2
    2-th head, head type raw, head output 1
    3-th head, head type raw, head output 1
    => loading model from /fclip_ckpt/ckpt/HG2/checkpoint.pth.tar
    Finished constructing model!
    ==> Running Demo.
    Processing: 1 Total time spent:30.598598
    Average frame rate:0.032681
    ==> Finshed Demo.
    

    The origin image: origin image

    The output image: output image

    opened by Honghe 4
  • what's the application scenarios of this model?

    what's the application scenarios of this model?

    what's the application scenarios of this model?

    opened by jinfagang 1
  • License

    License

    Hi! Excellent work and paper! I was wondering if you could add a LICENSE to your codebase? Thanks!

    opened by BrianPugh 1
  • A little error of yaml path

    A little error of yaml path

    Firstly, thanks for the marvelous work and well written codes.

    A little remind : when I was trying to run the code, I found that in the file FClip/models/pose_hrnet.py, at line 525, the path of yaml should be changed to my own path to run correctly. 图像 1

    bug 
    opened by Yifei6672 1
  • about training time

    about training time

    Hello, I want to know how long it took you to train FClip-HR? It took me more than seven days. Is there any way to reduce the training time?

    opened by chenyuyuyu 1
  • Does the training support multi-GPU?

    Does the training support multi-GPU?

    Does the training support multi-GPU?

    opened by starsdeep 1
Owner
Xili Dai
UC Berkeley, California, USA. [email protected]
Xili Dai
The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

This is a curated list of tutorials, projects, libraries, videos, papers, books and anything related to the incredible PyTorch. Feel free to make a pu

Ritchie Ng 8.4k Sep 22, 2021
Collection of generative models in Pytorch version.

pytorch-generative-model-collections Original : [Tensorflow version] Pytorch implementation of various GANs. This repository was re-implemented with r

Hyeonwoo Kang 2.3k Sep 16, 2021
🍀 Pytorch implementation of various Attention Mechanisms, MLP, Re-parameter, Convolution, which is helpful to further understand papers.⭐⭐⭐

?? Pytorch implementation of various Attention Mechanisms, MLP, Re-parameter, Convolution, which is helpful to further understand papers.⭐⭐⭐

xmu-xiaoma66 1.8k Sep 19, 2021
PaddleViT: State-of-the-art Visual Transformer and MLP Models for PaddlePaddle 2.0+

PaddlePaddle Vision Transformers State-of-the-art Visual Transformer and MLP Models for PaddlePaddle ?? PaddlePaddle Visual Transformers (PaddleViT or

null 87 Sep 23, 2021
Code for the Interspeech 2021 paper "AST: Audio Spectrogram Transformer".

AST: Audio Spectrogram Transformer Introduction Citing Getting Started ESC-50 Recipe Speechcommands Recipe AudioSet Recipe Pretrained Models Contact I

Yuan Gong 155 Sep 19, 2021
Tensors and Dynamic neural networks in Python with strong GPU acceleration

PyTorch is a Python package that provides two high-level features: Tensor computation (like NumPy) with strong GPU acceleration Deep neural networks b

null 51k Sep 24, 2021
Tensors and Dynamic neural networks in Python with strong GPU acceleration

PyTorch is a Python package that provides two high-level features: Tensor computation (like NumPy) with strong GPU acceleration Deep neural networks b

null 46.1k Feb 13, 2021
🤗 Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX.

English | 简体中文 | 繁體中文 State-of-the-art Natural Language Processing for Jax, PyTorch and TensorFlow ?? Transformers provides thousands of pretrained mo

Hugging Face 51.7k Sep 23, 2021
StudioGAN is a Pytorch library providing implementations of representative Generative Adversarial Networks (GANs) for conditional/unconditional image generation.

StudioGAN is a Pytorch library providing implementations of representative Generative Adversarial Networks (GANs) for conditional/unconditional image generation.

null 1.8k Sep 26, 2021
PyTorch implementations of Generative Adversarial Networks.

This repository has gone stale as I unfortunately do not have the time to maintain it anymore. If you would like to continue the development of it as

Erik Linder-Norén 10.2k Sep 22, 2021
Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

Phil Wang 5.7k Sep 17, 2021
PyTorch implementation of "Learning to Discover Cross-Domain Relations with Generative Adversarial Networks"

DiscoGAN in PyTorch PyTorch implementation of Learning to Discover Cross-Domain Relations with Generative Adversarial Networks. * All samples in READM

Taehoon Kim 983 Sep 21, 2021
High-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently.

TL;DR Ignite is a high-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently. Click on the image to

null 3.7k Sep 23, 2021
The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate.

The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate. Website • Key Features • How To Use • Docs •

Pytorch Lightning 15.4k Sep 22, 2021
Geometric Deep Learning Extension Library for PyTorch

Documentation | Paper | Colab Notebooks | External Resources | OGB Examples PyTorch Geometric (PyG) is a geometric deep learning extension library for

Matthias Fey 12.5k Sep 22, 2021
The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate.

The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate. Website • Key Features • How To Use • Docs •

Pytorch Lightning 11.9k Feb 13, 2021
A library for finding knowledge neurons in pretrained transformer models.

knowledge-neurons An open source repository replicating the 2021 paper Knowledge Neurons in Pretrained Transformers by Dai et al., and extending the t

EleutherAI 50 Sep 19, 2021
PyTorch implementation of Progressive Growing of GANs for Improved Quality, Stability, and Variation.

PyTorch implementation of Progressive Growing of GANs for Improved Quality, Stability, and Variation. Warning: the master branch might collapse. To ob

null 533 Sep 5, 2021
A collection of SOTA Image Classification Models in PyTorch

A collection of SOTA Image Classification Models in PyTorch

sithu3 20 Sep 18, 2021