Am I right in understanding that currently, you only implement the STLT model, without any of the appearance features?

Is there a timeline for when the use of appearance features will be incorporated? I am particularly interested in either the PBF (hoping to use my own appearance features) or CACNF models (if it can be pretrained end-to-end).

Thanks

As I was tweaking the code, I found some design choices in the position embeddings which I cannot understand.

1. For the STLT, it seems that the position embedding vector does not take into consideration the number of sampled frames. Specifically, although it does use the config parameter config.layout_num_frames, this is never set in train.py (lines 87-96) or inference.py (lines 47-54). This means that the position embedding vector is always of size 256 (configs.py line 109). Is there a reason for this? I have changed this in my code, and ensuring that the position embedding is always config.layout_num_frames + 1 - does this make sense?

2. For CACNF, the position embedding seems to be dependent on the Batch-Size and Image size itself for the summation with the features to work (models.py line 267). If I change the config.spatial_size or the batch size to less than 16, then there is a dimension mismatch. I hacked this by computing some intermediary values in my code but it is somewhat hard-coded all the same based on my data. Is there a reason for this dependency/architecture?

Hi Gorjan

I am running into an issue with setting the number of frames to sample from each video. To put it into context, I need to classify 1s clips at a time, which amount to 25 frames, and hence, cannot sample more than those. The current setup is 32 frames, and I changed appearance_num_frames to 25. However, this may be interfering with the forward_features() method of TransformerResnet. It seems that the ResNet outputs by default 32 sequence length, and not sure if this can be modified.

The error happens in models.py line 267, when it tries to join it with the position embedding. Any idea how I can rectify? am I interpreting the appearance_num_frames correctly to begin with?

hello, I tried to train the stlt model but got some errors. I first run create_dataset.py to prepare dataset, after that, I canot find the videoid2size.json file, which however is necessary for running the code. How can I get this json file? Thanks a lot ~

Hi Gorjan

Once trained, how can I save a CACNF model? does setting the save_model_path store also the backbones (including the fine-tuned Resnet) or do I need to store these (and load them) separately using the resnet_model_path and save_backbone_path?

Just checking

The labels for the classifier, are they 1- or 0-based indexed? i.e. is the smallest label 0 or 1? the reason I am asking is that the associated label file seems to start from 1, but when I did the same for my classes, it gave me indexing errors and I had to do everything starting from 0.

@gorjanradevski Dear author, I downloaded the STLT model on Action Genome Oracle and tried to run the following code, but the map is 58.75, slightly lower than 60.6 in the paper. Can you tell me why? `(dp-env) wn@node01:~/temp/STLT$ python inference.py --test_dataset_path "data/ActionGenome/val_dataset.json" --batch_size 1 --dataset_type "layout" --model_name "stlt" --checkpoint_path "checkpoints/action_genome_gt_stlt.pt" --dataset_name "action_genome" --labels_path "data/ActionGenome/labels.json" --videoid2size_path "data/ActionGenome/videoid2size.json" INFO:root:Preparing dataset... INFO:root:Inference on 1814 INFO:root:Preparing model... INFO:root:================================== INFO:root:The model's configuration is:

• Unique categories: 38
• Number of classes: 157
• Hidden size: 768
• Hidden dropout probability: 0.1
• Layer normalization epsilon: 1e-12
• Number of attention heads: 12
• Number of spatial layers: 4
• Number of temporal layers: 8
• Max number of layout frames: 256
• The backbone path is: None
• Freezing the backbone: False INFO:root:================================== INFO:root:Starting inference... 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1814/1814 [00:37<00:00, 48.10it/s] INFO:root:================================= INFO:root:The metrics are: INFO:root:map: 58.75 INFO:root:================================= `