Hi, When we only change the config to pretrain_r2r_e2e.json, and still run the main_r2r.py, it seems that the image files are not used. Besides, as described in the paper, ViT e2e training is done in the second stage. Thus I train a model checkpoint that finishes first stage pretrain, and I directly use pretrain_r2r_e2e.json (also modify the ckpt path) and main_r2r_image.py to run the ViT e2e training. However, there's some dim mismatch error occurs in image_pretrain.py line 80 (which points to vilmodel.py line 550). Could you help check if there's anything I need to change? Thanks so much!

Hi,

Thanks for the great work!

I noticed that if we use preprocess/precompute_img_features_vit.py to extract visual features, the outputted features are slightly different from features in provided HDF5 files. For pretrained ViT-B/16 from TIMM, there's about 0.01 average difference between outputted features and features in datasets/R2R/features/pth_vit_base_patch16_224_imagenet.hdf5. For e2e pretrained ViT-B/16 from datasets/R2R/trained_models/vitbase-6tasks-pretrain-e2e/model_step_22000.pt, there's about 0.007 average difference between outputted features and features in datasets/R2R/features/pth_vit_base_patch16_224_imagenet_r2r.e2e.ft.22k.hdf5.

I also ensured this isn't caused by potentially different batchsize in forward step. Feature differences caused by different batchsize shouldn't be around 1e-6.

Could you provide any suggestions on the reason causing these differences?

Hi, thanks for your great work!

I noted that you used CLIP to extract feature to train on RxR dataset. May I ask which version did you use to extract the features? ViT/B-16 or ViT/B-32? The version from openAI or huggingface??

Thanks for your attention on this matter! Best regards,

Hi Shizhe,

Thank you for releasing the code! I am wondering if you can specify or release the models for us to reproduce the reported results. I would appreciate it greatly if you can provide additional instructions!

Hi, Thanks for the great work! I'm trying to directly load the pretrained model you provide "../datasets/R2R/trained_models/vitbase-6tasks-pretrain/model_step_130000.pt" and then fine-tune on r2r task. However, I encounter the problem:

 RuntimeError: Error(s) in loading state_dict for NavCMT:
size mismatch for bert.hist_embeddings.position_embeddings.weight: copying a param with shape torch.Size([50, 768]) from checkpoint, the shape in current model is torch.Size([100, 768]).

I also followed your provided scripts to pre-train the model (w/o e2e vit), and then fine-tune on r2r, which seems to work fine.
Just hope to check whether the provided pre-trained model (w/o e2e vit) is the correct one, and if I need to modify any code/hyperparameter to load the pre-trained model?

Thanks so much!

Dear developers,

Your project VLN-HAMT requires "networkx==2.5.1" in its dependency. After analyzing the source code, we found that the following versions of networkx can also be suitable without affecting your project, i.e., networkx 2.5. Therefore, we suggest to loosen the dependency on networkx from "networkx==2.5.1" to "networkx>=2.5,<=2.5.1" to avoid any possible conflict for importing more packages or for downstream projects that may use ddos_script.

May I pull a request to further loosen the dependency on networkx?

By the way, could you please tell us whether such dependency analysis may be potentially helpful for maintaining dependencies easier during your development?

Details:

Your project (commit id: 08918ddcea7b7822831a5b535038732f8dfeab23) directly uses 5 APIs from package networkx.

networkx.algorithms.shortest_paths.weighted.all_pairs_dijkstra_path, networkx.classes.function.set_node_attributes, networkx.classes.graph.Graph.__init__, networkx.algorithms.shortest_paths.weighted.all_pairs_dijkstra_path_length, networkx.classes.graph.Graph.add_edge

Beginning fromwhich, 1 functions are then indirectly called, including 0 networkx's internal APIs and 1 outsider APIs as follows:

[/cshizhe/VLN-HAMT]
+--networkx.algorithms.shortest_paths.weighted.all_pairs_dijkstra_path
+--networkx.classes.function.set_node_attributes
+--networkx.classes.graph.Graph.__init__
|      +--networkx.convert.to_networkx_graph
|      |      +--networkx.convert.from_dict_of_dicts
|      |      +--networkx.convert.from_dict_of_lists
|      |      +--warnings.warn
|      |      +--networkx.convert.from_edgelist
+--networkx.algorithms.shortest_paths.weighted.all_pairs_dijkstra_path_length
+--networkx.classes.graph.Graph.add_edge

Since all these functions have not been changed between any version for package "networkx" from [2.5] and 2.5.1. Therefore, we believe it is safe to loosen the corresponding dependency.

Hi, Shizhe. Thanks very much for the great HAMT work! I was recently using your code trying to run the VLN experiments myself. I noticed that you provided the running scripts for pretraining and fine-tuning, but not for training the model on R2R from scratch. And I guess the model configuration for training-from-scratch should be different from finetuning-after-pretraining, e.g., the --fix_lang_embedding and --fix_hist_embedding should not be set as the two embeddings are totally random right? So I hope you could share the scripts for training HAMT from scratch.