Hello,

I was trying to train the network and the first step is the precomputation of the AtlasNet features.

I followed the README (and fixed a few lines).

self.rel_json_file = os.path.join(self.root, '3DSSG_processed_files', '{}.json'.format(splits_fname))
self.box_json_file = os.path.join(self.root, 'GT 3D boxes 3DSSG', 'obj_boxes_train_refined.json')

But I often get empty objects with zero points and an error in this line

choice2 = np.random.choice(len(obj_pointset), self.npoints - choice.shape[0], replace=True)

How do you resolve this?

Hi, when I want to download the checkpoint for the training model and the Atlasnet weight, I perform "chmod + x ./download_checkpoints.sh & ./download_checkpoints.sh and the same "404 not found" also appears. How did you solve it?

Hello, I hope to combine 3DSSG and oriented bounding box you annotated on the paper.

But the number of objects for same scan doesn't match.

For example, 3DSSG has 30 objects but yours has 25 objects. And I don't understand why the id is not in order(like 35->37->50)

Could you explain about it?

Execute chmod +x ./download_metadata_3rscan.sh && ./download_metadata_3rscan.sh, or chmod +x ./download_3dssg.sh && ./download_3dssg.sh, and 404 not fuound appears.

I have one a small problem with your provided checkpoints.

I downloaded the 3DSSG dataset from here: https://campar.in.tum.de/public_datasets/3DSSG/3DSSG.zip including the classes.txt which has 528 classes. I downloaded the checkpoints from here: https://drive.google.com/drive/folders/1RbNUpHLQiY1zskd0buAgvl-yFwnD9N8M?usp=sharing

When I try to evaluate the model using:

python scripts/evaluate_vaegan.py --dataset_3RScan ../3RScan_v2/data/ --exp ./experiments/final_checkpoints/shared --with_points False --with_feats True --epoch 100 --path2atlas ./experiments/atlasnet/model_70.pth --evaluate_diversity False

I get the following error:

RuntimeError: Error(s) in loading state_dict for Sg2ScVAEModel:
        size mismatch for obj_embeddings_ec_box.weight: copying a param with shape torch.Size([162, 128]) from checkpoint, the shape in current model is torch.Size([530, 128]).
        size mismatch for obj_embeddings_ec_shape.weight: copying a param with shape torch.Size([162, 128]) from checkpoint, the shape in current model is torch.Size([530, 128]).
        size mismatch for pred_embeddings_ec_box.weight: copying a param with shape torch.Size([27, 256]) from checkpoint, the shape in current model is torch.Size([41, 256]).
        size mismatch for pred_embeddings_ec_shape.weight: copying a param with shape torch.Size([27, 256]) from checkpoint, the shape in current model is torch.Size([41, 256]).
        size mismatch for obj_embeddings_dc_box.weight: copying a param with shape torch.Size([162, 256]) from checkpoint, the shape in current model is torch.Size([530, 256]).
        size mismatch for obj_embeddings_dc_man.weight: copying a param with shape torch.Size([162, 256]) from checkpoint, the shape in current model is torch.Size([530, 256]).
        size mismatch for obj_embeddings_dc_shape.weight: copying a param with shape torch.Size([162, 256]) from checkpoint, the shape in current model is torch.Size([530, 256]).
        size mismatch for pred_embeddings_dc_box.weight: copying a param with shape torch.Size([27, 512]) from checkpoint, the shape in current model is torch.Size([41, 512]).
        size mismatch for pred_embeddings_dc_shape.weight: copying a param with shape torch.Size([27, 512]) from checkpoint, the shape in current model is torch.Size([41, 512]).
        size mismatch for pred_embeddings_dc.weight: copying a param with shape torch.Size([27, 256]) from checkpoint, the shape in current model is torch.Size([41, 256]).
        size mismatch for pred_embeddings_man_dc.weight: copying a param with shape torch.Size([27, 768]) from checkpoint, the shape in current model is torch.Size([41, 768]).

I think this error is generated because self.obj_embeddings_ec_box = nn.Embedding(num_objs + 1, obj_embedding_dim) in Sg2ScVAEModel in VAEGAN_SHARED.py takes the number of classes as the number of input features. And the number of classes is defined by the length of vocab['object_idx_to_name'] which is loaded from the classes.txt which is 528 classes + the first row scene. This similar for the relationships and self.pred_embeddings_ec_box = nn.Embedding(num_preds, 2 * embedding_dim).

Seams like you trained only with 161 classes and 27 relationships. Could you provide me with the correct classes.txt file and relationships.json to replicate your results?

Hi, thank you for your amazing work. I trained your main model with shared shape and layout embedding, the accuracy is the same as the paper, but when I visualize the point cloud object, the quality of point cloud generation is not good. I followed the steps in the readme, am I missing any details?

With your project you provide the relationships_train_clean.json and relationships_validation_clean.json files which contain the scene graphs. These scene graphs are different from the original scene graphs from 3DSSG as they are split up into splits which do not cover the entire scene.The splits make sense to me to make the scene graphs less complex for scene generation. But the the construction of the splits is not really obvious to me.

In this screenshot I have visualized the scene and scene graph from e61b0e04-bada-2f31-82d6-72831a602ba7 split 6. Corresponding objects are highlighted with the same color.

This scene graph is especially weird for me because I would have assumed that all the chairs around the dining table were part of this split of the scene graph, but instead objects very far away from the dining table were selected like the door or wall. I couldn't find any documentation how you created these splits and as far as I know they are not part of the original 3DSSG dataset. So it would be awesome if you could provide some information about this. Maybe even provide a script to show how you created the splits which could also be changed to create more local splits.

Hello,

When downloading the checkpoints using the bash file download_checkpoints.sh, I get the following error.

Error:

ERROR: cannot verify campar.in.tum.de's certificate, issued by ‘CN=DFN-Verein Global Issuing CA,OU=DFN-PKI,O=Verein zur Foerderung eines Deutschen Forschungsnetzes e. V.,C=DE’:
  Unable to locally verify the issuer's authority.
To connect to campar.in.tum.de insecurely, use `--no-check-certificate'.
unzip:  cannot find or open final_checkpoints.zip, final_checkpoints.zip.zip or final_checkpoints.zip.ZIP.
mkdir: atlasnet: File exists
mv: rename model_70.pth to ./atlasnet/model_70.pth: No such file or directory

Steps to reproduce:

• conda create --name g2s_env python=3.7 --file requirements.txt
• conda activate g2s_env # activate virtual environment
• conda install pytorch==1.2.0 cudatoolkit=10.0 -c pytorch
• pip install tensorboardx graphviz plyfile open3d==0.9.0.0 open3d-python==0.7.0.0
• export PYTHONPATH="$PWD"

Then download the checkpoints using the following command to get the error:

chmod +x ./download_checkpoints.sh && ./download_checkpoints.sh

Environment:

• macOS 12.0.1
• Python 3.7.13

The title of the paper says end-to-end.

Is it true that you train the DeepSDF model together with the GCNs?

If so where can I find the architecture of the shape Encoder ? What is the input modality for the shape - is it a point cloud, SDF, voxel grid? Is that encoder pretrained? Otherwise training with loss makes no sense.

I can't figure out where in the paper this is discussed.

Another thing that is unclear to me is the labelling of the latent graph - is the generated per class and per node? I am guessing only the object nodes get any latent while the edge/relationship nodes are processed by the GCN to learn their meaning and do not have any KL term. But instead just propagate as are? Eq 1,2,3 are a bit unclear to me.

Thanks!

Hello, while analyzing your code and paper, I have a question about them (especially in evaluation).

You mentioned the evaluation metrics for boxes and shapes in the paper.

I understood that in your 'evaluate_vaegan.py' code, your metrics for boxes are printed as 'acc & ... & ...', but I can't find any metrics for shapes that you mentioned in your paper as 'cycle-consistency experiment'. (There isn't any recall values printed in the result of evaluation)

Is it experimented with another code?

Thank you!

Some possible errors in the edge annotations have been detected.

For example, in the scene graph ID e61b0e04-bada-2f31-82d6-72831a602ba7 the objects 7 and 42 have the following relationship : [7, 42, 8, 'bigger than'].

However, based on the volume of their bounding boxes, the relationship seems to be incorrect.

"7": {"param7": [0.7991325537319209, 0.17943594268441287, 0.5799999926239252, ...]
"42": {"param7": [0.4229016144056281, 0.43046053349812485, 0.7599999904632568, ...],

vol_7  = 0.08316799874
vol_42 = 0.13835226372

For a screenshot of the scene and the full list of possible conflicts please refer to this link : https://drive.google.com/drive/folders/1B_1hb8wH3TleII2iKEy_FOv5t0eXIHvG?usp=sharing

Pietro.

Hi,

The download links in download_metadata_3rscan.sh are not working.

If you can let us know what the different files are, we can also create the text files from the json files.

Hello,I tried to run the evaluate_vaegan.py to visualize 3D scene in "meshes", and there is some bug ,I want to know what do I need to modify in the args or the code? Looking forward to your reply！

Hey, was getting the following error on Colab with Python 3.7.12

ERROR: Invalid requirement: 'cloudpickle=0.5.3' (from line 3 of requirements.txt)
Hint: = is not a valid operator. Did you mean == ?

Changing the = to == for each requirement might fix it.