Generate indoor scenes with Transformers

Overview

SceneFormer: Indoor Scene Generation with Transformers

Examples of scenes generated by Sceneformer

Initial code release for the Sceneformer paper, contains models, train and test scripts for the shape conditioned model. Text conditioned model and detailed README coming soon.

Please also check the project website here

Setup

Install the requirements in requirements.txt and environment.yaml in a conda environment. Packages that are common can be installed either through pip or conda.

Prepare Data

The SUNCG dataset is currently not available, hence all related files have been removed. The dataset can be prepared with the scripts which were taken from deepsynth.

Train

Configure the experiment in configs/scene_shift_X_config.yaml where X is one of cat, dim, loc, ori

Then run

python scene_scripts/train_shift_X_lt.py configs/scene_shift_X_config.yaml

to train the model X.

Test

Configure the model paths in scene_scripts/test.py and then run

python scene_scripts/test.py

If you find our work useful, please consider citing us:

@article{wang2020sceneformer,
  title={SceneFormer: Indoor Scene Generation with Transformers},
  author={Wang, Xinpeng and Yeshwanth, Chandan and Nie{\ss}ner, Matthias},
  journal={arXiv preprint arXiv:2012.09793},
  year={2020}
}
Comments
  • About loc model

    About loc model

    Thanks for your great paper and code.

    I used the configuration provided in configs/scene_shift_X_config.yaml to train 4 models. The minimum losses of the cat, dim, ori models are about 1.5, and the corresponding epochs are about 300. But the loss of loc model can't be minimized to less than 2.5, and the training stopped early at about 100 epochs.

    Does the training of the cat, dim, ori models meet expectations? And what configuration should I modify to fix the training of model loc? (e.g. for configs/scene_shift_loc_config.yaml, dose warmup: 2 need to modified?)

    opened by mingjiayang 5
  • About model_dims.pkl

    About model_dims.pkl

    thanks for your great paper i use the scripts creat_data.py to process suncg dataset, but it did not generate the model_dims.pkl file.

    is there any scripts i can use to generate model_dims.pkl on suncg dataset?

    Best regards, Li

    opened by lilinyuanvr 4
  • the model_dims.pkl

    the model_dims.pkl

    Hi,after I run the script.py which the deep synth have,I get the filtered room data,but the train code need "model_dims.pkl",I just have "model_prior.pkl" ,"model_frequency.pkl","fine_categories_frequency","final_categories_frequency" and "coarse_categories_frequency",which pkl files is the "model_dims.pkl"?

    opened by JackW987 2
  • dimensions of objects

    dimensions of objects

    Thanks for the great paper and code,

    I'm trying to use your model and I've a question regarding the implementation, about the contents of the model_dims.pkl, in the paper you say Similarly the dimensions of each object; length, width and height or (l, w, h) are scaled and quantized given how you read model_dims.pkl, it seems the scaling part at least was not implemented, right?

    Best regards, Mohamad.

    opened by mayman99 2
  • error when pickl.load('35.pkl')

    error when pickl.load('35.pkl')

    room_pkl = pickle.load(f) AttributeError: Can't get attribute 'Wall' on <module 'data.house' from '/mnt/cfs/sceneformer/data/house.py'>

    how to load the pkl file?

    opened by Song-xx 1
  • Location model curr_cat_emb question

    Location model curr_cat_emb question

    Thanks for your greatest work I have some question when I browse the scene_shift_loc_col.py "joint_emb = cat_emb+pos_emb+loc_emb+ori_emb+coor_type_emb+curr_cat_emb" what does the curr_cat_emb means ? looking forward for your reply

    opened by JackW987 0
  • About the (floor,wall,windows,door,nodes)

    About the (floor,wall,windows,door,nodes)

    Thanks for your great paper When I run the deep-synth create_data.py to prepare the dataset ,I just get the (floor,wall,nodes) tuple,but the suncg_shift_seperate_dataset_deepsynth.py want the (floor,wall,windows,door,nodes) tuple.Does some wrong I make when I prepaer the dataset? Wish for your reply.

    opened by JackW987 0
  • conda env create --file environment.yaml error

    conda env create --file environment.yaml error

    conda env create --name sceneformer3 --file environment.yaml

    >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/root/anaconda3/lib/python3.8/site-packages/conda/exceptions.py", line 1079, in __call__
        return func(*args, **kwargs)
      File "/root/anaconda3/lib/python3.8/site-packages/conda_env/cli/main.py", line 80, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "/root/anaconda3/lib/python3.8/site-packages/conda_env/cli/main_create.py", line 88, in execute
        spec = specs.detect(name=name, filename=get_filename(args.file), directory=os.getcwd())
      File "/root/anaconda3/lib/python3.8/site-packages/conda_env/specs/__init__.py", line 43, in detect
        if spec.can_handle():
      File "/root/anaconda3/lib/python3.8/site-packages/conda_env/specs/yaml_file.py", line 18, in can_handle
        self._environment = env.from_file(self.filename)
      File "/root/anaconda3/lib/python3.8/site-packages/conda_env/env.py", line 166, in from_file
        return from_yaml(yamlstr, filename=filename)
      File "/root/anaconda3/lib/python3.8/site-packages/conda_env/env.py", line 144, in from_yaml
        data = validate_keys(data, kwargs)
      File "/root/anaconda3/lib/python3.8/site-packages/conda_env/env.py", line 37, in validate_keys
        new_data = data.copy() if data else {}
    AttributeError: 'str' object has no attribute 'copy'
    
    
    who knows what is wrong with my operation,conda version or something else ?
    
    opened by Song-xx 0
  • Object categories

    Object categories

    Thanks for your great paper and code. As described in your paper, you used 50 object categories for bedrooms and 39 object categories for living room. But Kai Wang used 31 categories and 21 categories respectively in DeepSynth. Could you please give me more details about the object categories you selected? Thank you very much.

    opened by mingjiayang 2
Owner
Chandan Yeshwanth
Studying Computer Science, interested in Deep Learning
Chandan Yeshwanth
Winning solution of the Indoor Location & Navigation Kaggle competition

This repository contains the code to generate the winning solution of the Kaggle competition on indoor location and navigation organized by Microsoft

Tom Van de Wiele 62 Dec 28, 2022
PyTorch implementation of ShapeConv: Shape-aware Convolutional Layer for RGB-D Indoor Semantic Segmentation.

Shape-aware Convolutional Layer (ShapeConv) PyTorch implementation of ShapeConv: Shape-aware Convolutional Layer for RGB-D Indoor Semantic Segmentatio

Hanchao Leng 82 Dec 29, 2022
PyTorch implementation of our ICCV2021 paper: StructDepth: Leveraging the structural regularities for self-supervised indoor depth estimation

StructDepth PyTorch implementation of our ICCV2021 paper: StructDepth: Leveraging the structural regularities for self-supervised indoor depth estimat

SJTU-ViSYS 112 Nov 28, 2022
[ICCV 2021 Oral] NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo

NerfingMVS Project Page | Paper | Video | Data NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo Yi Wei, Shaohui

Yi Wei 369 Dec 24, 2022
Pathdreamer: A World Model for Indoor Navigation

Pathdreamer: A World Model for Indoor Navigation This repository hosts the open source code for Pathdreamer, to be presented at ICCV 2021. Paper | Pro

Google Research 122 Jan 4, 2023
Indoor Panorama Planar 3D Reconstruction via Divide and Conquer

HV-plane reconstruction from a single 360 image Code for our paper in CVPR 2021: Indoor Panorama Planar 3D Reconstruction via Divide and Conquer (pape

sunset 36 Jan 3, 2023
Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

The Hypersim Dataset For many fundamental scene understanding tasks, it is difficult or impossible to obtain per-pixel ground truth labels from real i

Apple 1.3k Jan 4, 2023
The implementation of the paper "A Deep Feature Aggregation Network for Accurate Indoor Camera Localization".

A Deep Feature Aggregation Network for Accurate Indoor Camera Localization This is the PyTorch implementation of our paper "A Deep Feature Aggregation

null 9 Dec 9, 2022
The Habitat-Matterport 3D Research Dataset - the largest-ever dataset of 3D indoor spaces.

Habitat-Matterport 3D Dataset (HM3D) The Habitat-Matterport 3D Research Dataset is the largest-ever dataset of 3D indoor spaces. It consists of 1,000

Meta Research 62 Dec 27, 2022
Tensorflow 2.x implementation of Panoramic BlitzNet for object detection and semantic segmentation on indoor panoramic images.

Deep neural network for object detection and semantic segmentation on indoor panoramic images. The implementation is based on the papers:

Alejandro de Nova Guerrero 9 Nov 24, 2022
Repository for "Toward Practical Monocular Indoor Depth Estimation" (CVPR 2022)

Toward Practical Monocular Indoor Depth Estimation Cho-Ying Wu, Jialiang Wang, Michael Hall, Ulrich Neumann, Shuochen Su [arXiv] [project site] DistDe

Meta Research 122 Dec 13, 2022
HSC4D: Human-centered 4D Scene Capture in Large-scale Indoor-outdoor Space Using Wearable IMUs and LiDAR. CVPR 2022

HSC4D: Human-centered 4D Scene Capture in Large-scale Indoor-outdoor Space Using Wearable IMUs and LiDAR. CVPR 2022 [Project page | Video] Getting sta

null 51 Nov 29, 2022
PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 2021

Neural Scene Flow Fields PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 20

Zhengqi Li 585 Jan 4, 2023
Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes (CVPR2021)

RSCD (BS-RSCD & JCD) Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes (CVPR2021) by Zhihang Zhong, Yinqiang Zheng, Imari Sato We co

null 81 Dec 15, 2022
Populating 3D Scenes by Learning Human-Scene Interaction https://posa.is.tue.mpg.de/

Populating 3D Scenes by Learning Human-Scene Interaction [Project Page] [Paper] License Software Copyright License for non-commercial scientific resea

Mohamed Hassan 81 Nov 8, 2022
Efficient 6-DoF Grasp Generation in Cluttered Scenes

Contact-GraspNet Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, Dieter

NVIDIA Research Projects 148 Dec 28, 2022
Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors, CVPR 2021

Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors Human POSEitioning System (H

Aymen Mir 66 Dec 21, 2022
Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes

Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes

null 111 Dec 29, 2022
The first dataset on shadow generation for the foreground object in real-world scenes.

Object-Shadow-Generation-Dataset-DESOBA Object Shadow Generation is to deal with the shadow inconsistency between the foreground object and the backgr

BCMI 105 Dec 30, 2022