Codes of CVPR2022 paper: Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction

Yining Hong

Last update: Dec 29, 2022

Related tags

Code Refactoring FixIt

Overview

Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction

Figure 1. Teaser.

Introduction

This paper studies the problem of fixing malfunctional 3D objects. While previous works focus on building passive perception models to learn the functionality from static 3D objects, we argue that functionality is reckoned with respect to the physical interactions between the object and the user. Given a malfunctional object, humans can perform mental simulations to reason about its functionality and figure out how to fix it. Inspired by this, we propose FixIt, a dataset that contains about 5k poorly-designed 3D physical objects paired with choices to fix them. To mimic humans' mental simulation process, we present FixNet, a novel framework that seamlessly incorporates perception and physical dynamics. Specifically, FixNet consists of a perception module to extract the structured representation from the 3D point cloud, a physical dynamics prediction module to simulate the results of interactions on 3D objects, and a functionality prediction module to evaluate the functionality and choose the correct fix. Experimental results show that our framework outperforms baseline models by a large margin, and can generalize well to objects with similar interaction types.

This paper is accepted by CVPR2022.

Project Page: http://fixing-malfunctional.csail.mit.edu/

Data and Checkpoints

Download Link

Put the data under ./data folder

We are still updating the dataset and Checkpoints

SetUp

conda create -n fixing python=3.7 pytorch=1.4.0 torchvision -c pytorch
conda activate fixing

pip install -r requirements.txt

cd flownet3d/lib
python setup.py install

Flow Prediction Network

cd flownet3d
python main.py --exp_name fridge #for training
python main.py --exp_name fridge --eval #for testing
cd utils
python hungarian.py 
python hungarian.py --split test

Instance Segmentation Network

cd pointnet++ 
python train_insseg.py --model pointnet2_part_seg_msg --log_dir fridge
python test_insseg.py --log_dir fridge

Dynamics Prediction Module

# Training
cd utils
python rearrange_data_for_dpi.py
python rearrange_data_for_dpi.py --split test
python create_train_valid.py
cd ../
cd dynamics
bash scripts/train_dpi.sh
# Inference
cd utils
python rearrange_test_data_for_dpi.py
python rearrange_test_data_for_dpi.py --split test
cd dynamics
bash scripts/eval_dpi.sh

Functional Prediction Module

cd pointnet++
python train_classification.py  --model pointnet2_cls_ssg --log_dir fridge
python test_classification.py fridge
cd ../
cd utils
python calculate_final_result.py

Citation

@inproceedings{hong2021fixing,
author = {Hong, Yining and Mo, Kaichun and Yi, Li and Guibas, Leonidas J and Torralba, Antonio and Tenenbaum, Joshua B and Gan, Chuang},
title = {Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction},
booktitle = {CVPR},
year = {2022}
}

You might also like...

coala provides a unified command-line interface for linting and fixing all your code, regardless of the programming languages you use.

"Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live." ― John F. Woods coala provides a

3.4k Dec 29, 2022

wxPython app for converting encodings, modifying and fixing SRT files

Subtitle Converter Program za obradu srt i txt fajlova. Requirements: Python version 3.8 wxPython version 4.1.0 or newer Libraries: srt, PyDispatcher

4 Nov 25, 2022

A tool for fixing inconsistent timestamp metadata (atime, ctime, and mtime).

Mtime Fixer Mtime Fixer is a tool for fixing inconsistent timestamp metadata (atime, ctime, and mtime). Sometimes timestamp metadata of folders are in

2 Jan 11, 2022

Automates the fixing of problems reported by yamllint by parsing its output

yamlfixer yamlfixer automates the fixing of problems reported by yamllint by parsing its output. Usage This software automatically fixes some errors a

26 Dec 26, 2022

Codes for realizing theories learned from Data Mining, Machine Learning, Deep Learning without using the present Python packages.

Codes-for-Algorithms Codes for realizing theories learned from Data Mining, Machine Learning, Deep Learning without using the present Python packages.

1 Apr 12, 2022

This is an incredible led matrix simulation using the ultimate mosaik co-simulation framework.

This project uses the mosaik co-simulation framework, developed by the brilliant developers at the high-ranked Offis institue for computer science, Oldenburg, Germany, to simulate multidimensional LED matrices.

1 Jan 28, 2022

Codes and models for the paper "Learning Unknown from Correlations: Graph Neural Network for Inter-novel-protein Interaction Prediction".

GNN_PPI Codes and models for the paper "Learning Unknown from Correlations: Graph Neural Network for Inter-novel-protein Interaction Prediction". Lear

2 Dec 14, 2022

Official Pytorch implementation of ICLR 2018 paper Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge.

Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge: Official Pytorch implementation of ICLR 2018 paper Deep Learning for Phy

47 Nov 6, 2022

A PyTorch implementation of Radio Transformer Networks from the paper "An Introduction to Deep Learning for the Physical Layer".

An Introduction to Deep Learning for the Physical Layer An usable PyTorch implementation of the noisy autoencoder infrastructure in the paper "An Intr

120 Nov 21, 2022

The code for CVPR2022 paper "Likert Scoring with Grade Decoupling for Long-term Action Assessment".

Likert Scoring with Grade Decoupling for Long-term Action Assessment This is the code for CVPR2022 paper "Likert Scoring with Grade Decoupling for Lon

10 Oct 21, 2022

This is an official implementation of the CVPR2022 paper "Blind2Unblind: Self-Supervised Image Denoising with Visible Blind Spots".

Blind2Unblind: Self-Supervised Image Denoising with Visible Blind Spots Blind2Unblind Citing Blind2Unblind @inproceedings{wang2022blind2unblind, tit

58 Dec 6, 2022

Code for the CVPR2022 paper "Frequency-driven Imperceptible Adversarial Attack on Semantic Similarity"

Introduction This is an official release of the paper "Frequency-driven Imperceptible Adversarial Attack on Semantic Similarity" (arxiv link). Abstrac

21 Nov 23, 2022

CVPR2022 paper "Dense Learning based Semi-Supervised Object Detection"

[CVPR2022] DSL: Dense Learning based Semi-Supervised Object Detection DSL is the first work on Anchor-Free detector for Semi-Supervised Object Detecti

69 Dec 8, 2022

Source code for CVPR2022 paper "Abandoning the Bayer-Filter to See in the Dark"

Abandoning the Bayer-Filter to See in the Dark (CVPR 2022) Paper: https://arxiv.org/abs/2203.04042 (Arxiv version) This code includes the training and

74 Dec 15, 2022

[CVPR2022] This repository contains code for the paper "Nested Collaborative Learning for Long-Tailed Visual Recognition", published at CVPR 2022

Nested Collaborative Learning for Long-Tailed Visual Recognition This repository is the official PyTorch implementation of the paper in CVPR 2022: Nes

65 Dec 9, 2022

Official code for CVPR2022 paper: Depth-Aware Generative Adversarial Network for Talking Head Video Generation

📖 Depth-Aware Generative Adversarial Network for Talking Head Video Generation (CVPR 2022) 🔥 If DaGAN is helpful in your photos/projects, please hel

503 Jan 4, 2023

WATTS provides a set of Python classes that can manage simulation workflows for multiple codes where information is exchanged at a coarse level

WATTS (Workflow and Template Toolkit for Simulation) provides a set of Python classes that can manage simulation workflows for multiple codes where information is exchanged at a coarse level.

13 Dec 23, 2022

A Python application that helps users determine their calorie intake, and automatically generates customized weekly meal and workout plans based on metrics computed using their physical parameters

1 Jan 13, 2022

SceneCollisionNet This repo contains the code for "Object Rearrangement Using Learned Implicit Collision Functions", an ICRA 2021 paper. For more info

31 Nov 22, 2022

Comments

Missing dataset file for instance segmentation

Hi, Thank you for the great work! For the segmentation part, the implementation of the dataset class seems missing in current repo. Would you please provide this code or some hints to to reproduce the results? Thank you so much!

https://github.com/evelinehong/FixIt/blob/8a19202b8731a649eb27936c57bd6df0831bbd80/pointnet%2B%2B/train_insseg.py#L16

opened by daoyuan98 0

Codes of CVPR2022 paper: Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction

Related tags

Overview

Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction

Introduction

Data and Checkpoints

SetUp

Flow Prediction Network

Instance Segmentation Network

Dynamics Prediction Module

Functional Prediction Module

Citation

You might also like...

coala provides a unified command-line interface for linting and fixing all your code, regardless of the programming languages you use.

wxPython app for converting encodings, modifying and fixing SRT files

A tool for fixing inconsistent timestamp metadata (atime, ctime, and mtime).

Automates the fixing of problems reported by yamllint by parsing its output

Codes for realizing theories learned from Data Mining, Machine Learning, Deep Learning without using the present Python packages.

This is an incredible led matrix simulation using the ultimate mosaik co-simulation framework.

Codes and models for the paper "Learning Unknown from Correlations: Graph Neural Network for Inter-novel-protein Interaction Prediction".

Official Pytorch implementation of ICLR 2018 paper Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge.

A PyTorch implementation of Radio Transformer Networks from the paper "An Introduction to Deep Learning for the Physical Layer".

The code for CVPR2022 paper "Likert Scoring with Grade Decoupling for Long-term Action Assessment".

This is an official implementation of the CVPR2022 paper "Blind2Unblind: Self-Supervised Image Denoising with Visible Blind Spots".

Code for the CVPR2022 paper "Frequency-driven Imperceptible Adversarial Attack on Semantic Similarity"

CVPR2022 paper "Dense Learning based Semi-Supervised Object Detection"

Source code for CVPR2022 paper "Abandoning the Bayer-Filter to See in the Dark"

[CVPR2022] This repository contains code for the paper "Nested Collaborative Learning for Long-Tailed Visual Recognition", published at CVPR 2022

Official code for CVPR2022 paper: Depth-Aware Generative Adversarial Network for Talking Head Video Generation

WATTS provides a set of Python classes that can manage simulation workflows for multiple codes where information is exchanged at a coarse level

A Python application that helps users determine their calorie intake, and automatically generates customized weekly meal and workout plans based on metrics computed using their physical parameters

SceneCollisionNet This repo contains the code for "Object Rearrangement Using Learned Implicit Collision Functions", an ICRA 2021 paper. For more info

Comments

Missing dataset file for instance segmentation

Owner

Yining Hong

Code generation and code search for Python and Javascript.

Awesome autocompletion, static analysis and refactoring library for python

Removes unused imports and unused variables as reported by pyflakes

A tool (and pre-commit hook) to automatically upgrade syntax for newer versions of the language.

Turn your C++/Java code into a Python-like format for extra style points and to make everyone hates you

Fully Automated YouTube Channel ▶️with Added Extra Features.

gjf: A tool for fixing invalid GeoJSON objects

POPPY (Physical Optics Propagation in Python) is a Python package that simulates physical optical propagation including diffraction

The official codes of our CVPR2022 paper: A Differentiable Two-stage Alignment Scheme for Burst Image Reconstruction with Large Shift

coala provides a unified command-line interface for linting and fixing all your code, regardless of the programming languages you use.