ComPhy: Compositional Physical Reasoning ofObjects and Events from Videos

Last update: Dec 29, 2022

Overview

ComPhy

This repository holds the code for the paper.

ComPhy: Compositional Physical Reasoning ofObjects and Events from Videos, (Under review)

PDF

Project Website

Framework

Code Preparation

git clone https://github.com/comphyreasoning/compositional_physics_learner.git

Installation

pip install -r requirements

Data Preparation

Download videos, video annotation, questions from the project website.

Fast Evaluation

Download the regional proposals with attribute and physical property prediction from the anonymous Google drive
Download the dynamic predictions from the anonymous Google drive
Run executor for factual questions.

sh scripts/test_oe_release.sh

Run executor for multiple-choice questions.

sh scripts/test_mc_release.sh

Supporting sub-modules

Physical Property Learner and Dynamic predictor

Please refer to this repo for property learning and dynamics prediction.

Perception

This module uses the public NS-VQA's perception module object detection and visual attribute extraction.

Program parser

This module uses the public NS-VQA's program parser module to tranform language into executable programs.

You might also like...

Code repo for "RBSRICNN: Raw Burst Super-Resolution through Iterative Convolutional Neural Network" (Machine Learning and the Physical Sciences workshop in NeurIPS 2021).

RBSRICNN: Raw Burst Super-Resolution through Iterative Convolutional Neural Network An official PyTorch implementation of the RBSRICNN network as desc

6 Nov 14, 2022

Physics-Aware Training (PAT) is a method to train real physical systems with backpropagation.

Physics-Aware Training (PAT) is a method to train real physical systems with backpropagation. It was introduced in Wright, Logan G. & Onodera, Tatsuhiro et al. (2021)1 to train Physical Neural Networks (PNNs) - neural networks whose building blocks are physical systems.

230 Jan 5, 2023

Official Pytorch implementation of ICLR 2018 paper Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge.

Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge: Official Pytorch implementation of ICLR 2018 paper Deep Learning for Phy

47 Nov 6, 2022

A PyTorch implementation of Radio Transformer Networks from the paper "An Introduction to Deep Learning for the Physical Layer".

An Introduction to Deep Learning for the Physical Layer An usable PyTorch implementation of the noisy autoencoder infrastructure in the paper "An Intr

120 Nov 21, 2022

FCA: Learning a 3D Full-coverage Vehicle Camouflage for Multi-view Physical Adversarial Attack

FCA: Learning a 3D Full-coverage Vehicle Camouflage for Multi-view Physical Adversarial Attack Case study of the FCA. The code can be find in FCA. Cas

21 Dec 15, 2022

Code and datasets for the paper "Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction" (RA-L, 2021)

Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction This is the code for the paper Combining E

69 Dec 26, 2022

Face and Pose detector that emits MQTT events when a face or human body is detected and not detected.

Face Detect MQTT Face or Pose detector that emits MQTT events when a face or human body is detected and not detected. I built this as an alternative t

38 Oct 21, 2022

Deep Learning and Logical Reasoning from Data and Knowledge

Logic Tensor Networks (LTN) Logic Tensor Network (LTN) is a neurosymbolic framework that supports querying, learning and reasoning with both rich data

171 Dec 29, 2022

Data and Code for ACL 2021 Paper "Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning"

Introduction Code and data for ACL 2021 Paper "Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning". We cons

81 Dec 27, 2022

Comments

Fast Evaluation on Validation set

Hello,

I notice that there are no “regional proposals with attribute and physical property prediction” or “dynamic predictions” included for the validation (and training) videos. Can this be made available to do fast evaluation on the validation set? Though these are included for the test videos, the ground-truth label is not included in test.json, so I cannot evaluate the accuracy there either. (FYI, for the test set I notice that 25 multiple choice test questions have no choices, which end up counting as a correct question because of this).

Thank you!

opened by AdamIshay 2
Missing test videos

On the Comphy website it looks like the link to the testing videos actually points to the testing annotations: https://drive.google.com/file/d/1qhaRGebUnwt9LdPIWvuxaBNQ-h3ahiXt/view. Would it be possible to add the testing videos to the website? Thanks!

opened by mbchang 1

ComPhy: Compositional Physical Reasoning ofObjects and Events from Videos

Related tags

Overview

ComPhy

Framework

Code Preparation

Installation

Data Preparation

Fast Evaluation

Supporting sub-modules

Physical Property Learner and Dynamic predictor

Perception

Program parser

You might also like...

Code repo for "RBSRICNN: Raw Burst Super-Resolution through Iterative Convolutional Neural Network" (Machine Learning and the Physical Sciences workshop in NeurIPS 2021).

Physics-Aware Training (PAT) is a method to train real physical systems with backpropagation.

Official Pytorch implementation of ICLR 2018 paper Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge.

A PyTorch implementation of Radio Transformer Networks from the paper "An Introduction to Deep Learning for the Physical Layer".

FCA: Learning a 3D Full-coverage Vehicle Camouflage for Multi-view Physical Adversarial Attack

Code and datasets for the paper "Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction" (RA-L, 2021)

Face and Pose detector that emits MQTT events when a face or human body is detected and not detected.

Deep Learning and Logical Reasoning from Data and Knowledge

Data and Code for ACL 2021 Paper "Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning"

Comments

Fast Evaluation on Validation set

Missing test videos

Owner

Phy-Q: A Benchmark for Physical Reasoning

Code for the AAAI-2022 paper: Imagine by Reasoning: A Reasoning-Based Implicit Semantic Data Augmentation for Long-Tailed Classification

Compositional and Parameter-Efficient Representations for Large Knowledge Graphs

Code to reproduce the results for Compositional Attention: Disentangling Search and Retrieval.

Code for SentiBERT: A Transferable Transformer-Based Architecture for Compositional Sentiment Semantics (ACL'2020).

Compositional Sketch Search

Code for EMNLP 2021 main conference paper "Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification"

This repository contains the code for the CVPR 2021 paper "GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields"

[NeurIPS 2021] Code for Unsupervised Learning of Compositional Energy Concepts

[CVPR'22] COAP: Learning Compositional Occupancy of People