Hi, thanks for opening source this great work! I'm currently working on a CSWM-like object-centric model and want to test its performance on the Physion dataset. It would be very great if you can release the code of training/testing CSWM on this dataset, so that I can easily integrate my model to the train/test protocols here. So I'm just wondering is there any plan for releasing any of the baseline models benchmarked in your paper? Thanks!

EDIT: okay so I found the code for particle models #45. But they're quite different from e.g. object-centric models because particle models are directly evaluated by calculating the min distance between patient/agent points. They don't involve the readout phase, while my question is mainly regarding the readout phase...

Spotted this error in feedback in second familiarization trial of this dominoes demo (https://cogtoolslab.org:8882/index.html): Problem: Says that I got it right when I predicted that "red" would not hit "yellow" but then said that "The red object did indeed hit the yellow area."

@felixbinder , could you augment the README for the human experiments to explain the necessary steps to create the MongoDB and ensure the JS code can connect to it?

• [x] Create requirements.txt?

Check

• [x] Can we download all necessary data to reproduce analysis?
• [x] Given the data, can we reproduce the analysis using the notebooks?

Link to details: https://neurips.cc/Conferences/2021/CallForDatasetsBenchmarks

Main submission

• [x] Verify that submission is <= 9 content pages in NeurIPS format, including all figures and tables; additional pages containing the paper checklist and references are allowed.

Please carefully follow the Latex template for this track when preparing proposals. We follow the NeurIPS format, but with the appropriate headings, and without hiding the names of the authors.

neurips_data_2021.tex -- LaTeX Template

neurips_data_2021.sty -- style file for LaTeX 2e

neurips_data_2021.pdf -- example PDF output

• [x] Papers should be submitted via OpenReview (click to start your submission). Reviewing is single-blind, hence the paper should not be anonymized.
• [x] Include public link to the dataset or benchmark data. Set S3 ACL settings to public-read now. @judithfan

Supplementary materials

• [x] Dataset documentation and intended uses. Recommended documentation frameworks include datasheets for datasets, dataset nutrition labels, data statements for NLP, and accountability frameworks.@yamins81 / @judithfan
• [x] Create URL to website/platform where the dataset/benchmark can be viewed and downloaded by the reviewers. If the dataset can only be released later, you must include instructions for reviewers on how to access the dataset. This can only be done after the first submission: after submission, there will be an 'add dataset or benchmark' button where you can leave information for reviewers. We highly recommend making the dataset publicly available immediately or before the start of the NeurIPS conference. In select cases, requiring solid motivation, the release date can be stretched up to a year after the submission deadline. @judithfan
  • [ ] Link to model URLs: appendix.tex see section A.2: Computational details
  • [x] Training data generation
  • [x] Test data hdf5s - uploading the hdf5s
  • [x] Test data mp4 - write script that fetches the "new data" (videos)
  • [x] Human psychophysics - pointer to repo & analysis notebooks/scripts
• [x] Add Author statement that they bear all responsibility in case of violation of rights, etc., and confirmation of the data license. @yamins81
• [x] Hosting, licensing, and maintenance plan. The choice of hosting platform is yours, as long as you ensure access to the data (possibly through a curated interface) and will provide the necessary maintenance. @judithfan To ensure accessibility, we largely follow the NeurIPS guidelines for data submission, but also allowing more freedom for non static datasets. The supplementary materials for datasets must include the following:
• [x] Links to access the dataset and its metadata. This can be hidden upon submission if the dataset is not yet publicly available but must be added in the camera-ready version. In select cases, e.g when the data can only be released at a later date, this can be added afterward. Simulation environments should link to (open source) code repositories. @judithfan
• [x] The dataset itself should ideally use an open and widely used data format. Provide a detailed explanation on how the dataset can be read. For simulation environments, use existing frameworks or explain how they can be used.
• [x] Long-term preservation: It must be clear that the dataset will be available for a long time, either by uploading to a data repository or by explaining how the authors themselves will ensure this
• [x] Explicit license: Authors must choose a license, ideally a CC license for datasets, or an open source license for code (e.g. RL environments). An overview of licenses can be found here: https://paperswithcode.com/datasets/license @yamins81
• [x] Add structured metadata to a dataset's meta-data page using Web standards (like schema.org and DCAT): This allows it to be discovered and organized by anyone. A guide can be found here: https://developers.google.com/search/docs/data-types/dataset. If you use an existing data repository, this is often done automatically. @yamins81
• [x] Highly recommended: a persistent dereferenceable identifier (e.g. a DOI minted by a data repository or a prefix on identifiers.org) for datasets, or a code repository (e.g. GitHub, GitLab,...) for code. If this is not possible or useful, please explain why. @yamins81
• [x] For benchmarks, the supplementary materials must ensure that all results are easily reproducible. Where possible, use a reproducibility framework such as the ML reproducibility checklist, or otherwise guarantee that all results can be easily reproduced, i.e. all necessary datasets, code, and evaluation procedures must be accessible and documented. @judithfan

Similar to: https://github.com/dicarlolab/dldata/blob/2c6cfd038861bbcff94be6c8897768179aff74fc/dldata/stimulus_sets/rust_datasets.py

TODO: replace with pointers to public repos

Draw occluders and distractors from a common subset of the model library and in a common sampling distribution across controllers.

IMPORTANT occluders, distractors, and "quirky" objects should all be models with use_flex=True. This is because only these models have easily accessed meshes that can be used for modeling.

For FLEX-based controllers, such as clothiness (@arty-p), all objects should have use_flex=True.

A way to get all the models that are FLEX-enabled is:

# lib = ModelLibrarian("models_full.json")
# for record in lib.records:
#     if record.flex:
#         print(record.name)

Write a test that automatically retrieves a single record from all collections where behavioral data are being saved & cross-checks that the variables being saved are consistent

• [x] write markdown with steps
• [x] Link osf ~~add documentation for server-side code~~ ~~add documentation for metadata generation notebooks & S3 upload~~
• [x] update readme on main page, point to more detailed readme inside ./experiments

Check

• [x] Can we follow these instructions to replicate our human experimental setup using exactly the same infrastructure @sradwan15 ~~Can we generate the input files needed for analysis, given experimental data inside a mongoDB?~~
• [x] Can we use this codebase to extend to variants of our human experiments? **

• [x] Naming consistency: Only 80-90 matching trials between human and modeling experiments? Need to debug the stimulus matching issue, there are known discrepancies between naming conventions for stimuli for model & human evaluations
• [x] Replicate RMSE analysis for all trials (not just adversarial subset) to verify that we can recover the same pattern in model-human consistency across models (with particle models doing the best, convnets worse, etc.)
• [x] Visualize raw correlation between human & model predictions on adversarial trials, contextualized among all trials.
• [x] Computing noise ceiling on the adversarial trials only (same as in paper). Then normalizing these metrics by these noise ceiling estimates.

• [x] Maybe the wording of "red object" is ambiguous, since naturally occurring objects in the scene are also red.
• [x] We'll need updated familiarization trial instructions

Hi

I was trying to run the RPIN model and was bumped into this import error

ModuleNotFoundError: No module named 'neuralphys'

It seems like neuralphys is not an open-source package that I can download.

Also would you please explain how I can run the DPI model?

It seems like the model takes as input some preprocessed dataset that is different from other H5 files.

Any help is appreciated.

- Tell users which set of controllers to use (the ones under tdw_physics/tdw_physics/target_controller or physics-benchmarking-neurips2021/stimuli/generation/controllers/)

• If we want them to use tdw_physics ones, we need
  • clear instructions for where to find physion controllers (i.e maybe rename target_controller as physion_controller, or bolden target_controller to emphasize; the controller folder can easily be mistaken for the right one)
  • better pointer to command_line_args for regeneration (say it's under "physics-benchmarking-neurips2021/stimuli/generation/configs/" but not in tdw)
• If we want them to use tdw_physics ones, we need to fix
  • a clear pointer to stimuli controllers (currently the right controller is in tdw_physics/tdw_physics/target_controller and there is also a ./controllers folder)

Hi,

This is very impressive work! I believe it will be like ImageNet for Physicals prediction. However, I face many difficulties when I try to follow it. The problems are mostly about model evaluations of DPI-NET/GNS etc. The questions are followings:

1. Did this dataset offers particle representations for evaluation and training of DPI-NET/GNS/HRN? Could we download it directly like the mp4 data as well? (That would be very very helpful!) How many particles are used? For each particle, is it contains information like mass, velocity, stiffness, etc? How could I obtain the particles data information?

2. For DPI-Net, it only takes one G_0 as the initial state to produce the future simulations. For Physion, how many time steps are used as inputs? How many time steps are used as predictions? (In the human accuracy evaluation as well) Is the last frame is used as initial state for DPI-NET?

3. This code offers the Physpot as tools to train the DPI-NET model. But I somehow find out it is very complex. Could you present the script to train DPI-NET and other particle-based models? It would be very important to provide more detailed explanations for this part for other researchers to follow this work.

Thank you very much if you would like to help!

• [ ] move utils from physion to physopt.model.utils, clean up physion
• [ ] Update dataprovider with new file structure
• [ ] Document paths, naming of artifacts, make sure aligns with analysis code
• [ ] Add particle-based models to physopt
• [ ] In physopt, make it easy to only parts of the full pipeline. Document in README
• [ ] Update README for running each model
• [ ] In "Modeling Experiments" section of README: describe how to reproduce physion modeling results, provide final model checkpoints?
• [ ] create private+public repo setup

Check

• [ ] Can we train each model?
• [ ] Can we train the readout and evaluate on human test set?
• [ ] Can we generate the input files needed for analysis?
• [ ] Can we add new models easily?