Hi again,

In certain scenarios, I'm sure there are cases when the moment an action occurs, it's not visible, but then after let's say 2s, the action is still in the process of occuring, and then it becomes visible. In these cases, would you label the action as "visible" or "not shown"? An idea I had was to label it "not shown" first, then after 2s when the action is visible, label the same action again but this time as "visible".

Does the visibility label impact the training at all? Or just the metric calculation?

Looking forward to your response. Thank you for reading!

Hi SoccerNet Dev Team,

I've managed to plug my own dataset into NetVLAD++, but am unable to train due to overloading my 32 GB of RAM.

I have ~80 matches of ~50 minutes of badminton matches with ResNet-152 features sampled at 5fps. After loading my dataset, I have ~18/32 GB of RAM used. The program gets killed while loading the model. I'm confused why, as it's only ~5.5 GB as shown in the TorchInfo summary below. I believe I should still have ~8 GB to spare. Is this a feature of NetVLAD++ specifically? I noticed that in #28 experiments were done with 60-90 GB of RAM.

Thank you for reading, and looking forwawrd to your insights!

TorchInfo Summary:

==========================================================================================
Layer (type:depth-idx)                   Output Shape              Param #
==========================================================================================
NetVLAD_plus_plus                        --                        --
├─Linear: 1-1                            [5236, 512]               1,049,088
├─NetVLAD: 1-2                           [44, 14336]               28,672
├─NetVLAD: 1-3                           [44, 14336]               28,672
├─Dropout: 1-4                           [44, 28672]               --
├─Linear: 1-5                            [44, 3]                   86,019
├─Sigmoid: 1-6                           [44, 3]                   --
==========================================================================================
Total params: 1,192,451
Trainable params: 1,192,451
Non-trainable params: 0
Total mult-adds (G): 5.50
==========================================================================================
Input size (MB): 42.89
Forward/backward pass size (MB): 31.54
Params size (MB): 4.77
Estimated Total Size (MB): 79.20
==========================================================================================

Hi again,

I'm trying to remove the "halves" (1 or 2) in Labels-v2.json when using the Annotation Tool - I believe it has something to do with Annotation/actions/interface/main_window.py and self.half = 1

I'm also trying to remove the need for a 3rd class, specifically I don't need Annotation/actions/config/third_class.txt. It seems that something must be changed in Annotation/actions/interface/event_selection.py to allow this.

After opening my video with the tool, it seems the Events List (right side) has shrunk and can't be adjusted (see image below). Is there a way to change the code to make this adjustment possible?

As always, thank you so much for taking the time to read my issue!

Hello SoccerNet Dev Team,

I'm currently in the process of reading your paper on CALF-Calibration, and the entire pipeline along with the results are very impressive. I have a (many) few questions on parts of the paper/code I'm confused with, and would really appreciate it if any of you could help clear up my confusion. I know I have many questions written below, and completely understand if you're unable to answer them due to the volume. Still, I would love to dig deeper into your work, and it would be amazing if you could help me do so. As always, thank you so much for your time, and I'm of course looking forward to your new discoveries!

Note: As mentioned in my previous issues, I'm trying to use action-spotting in the context of badminton.

1. Section 3: Calibration Algorithm Here it says "We base our calibration on the Camera Calibration for Broadcast Videos (CCBV) of Sha et al. [38], but we write our own implementation, given the absence of usable public code". I thought the public implement is here based on mentions from #19 and #32.

2. Section 3: Our training process Since there's not a large enough public dataset of ground-truth calibrations, it seems you needed to use a student-teacher distillation approach. Why does this require this approach? It's also mentioned that you use the "Xeebra" from Evs to obtain the pseudo-GT calibrations. And in the CCBV repo in ./calibration_data/model.png I assume I have to swap this out for a badminton court for my application. And in ./calibration_data/dictionary.json, the data format is:

{
  {
        "posX": 0.03973018142882254,
        "posY": 68.63033722968056,
        "posZ": -15.718964999679423,
        "focalLength": 4576.520967734781,
        "pan": 7.544952667759858,
        "tilt": 77.55662442882397,
        "template_id": 0,
        "calibration": [
            4576.520967734781,
            0.0,
            960.0,
            0.0,
            4576.520967734781,
            540.0,
            0.0,
            0.0,
            1.0
        ],
        "homography": [
            4659.98895334099,
            -328.4171986412632,
            25605.797859625953,
            -60.243484492166004,
            454.8368668686411,
            40864.09122521112,
            0.1282196063258542,
            -0.9680549606981291,
            69.81988273325322
        ],
        "homography_resize": [
            621.3318481445312,
            -43.788963317871094,
            3414.106689453125,
            -8.032465934753418,
            60.64492416381836,
            5448.544921875,
            0.1282196044921875,
            -0.9680549502372742,
            69.81988525390625
        ],
        "image": "/home/fmg/sources/mmeu/data/meanshift2/gmm/dictionary-four/dict-0000.png"
    },
    ...
}

I assume this is the format that "Xeebra Evs" product will write the data as? And I believe the model predictions are in the 1_field_calib_ccbv.json files?

As a follow up to Q1, if you didn't use the CCBV code, have you open-sourced your student calibration algorithm?

3. Section 3: Player localization It says for each frame, you use Mask R-CNN to obtain the bounding box, segmentation mask, and average RGB color of each detected/segmented person. When checking the 1_player_boundingbox_maskrcnn.json files, I see bbox, color and onfield predictions. Is onfield the image segmentation mask?

4. Section 3: Player localization It then says "Then, we compute a field mask following [10] to filter out the bounding boxes that do not intersect the field, thus removing e.g. staff and detections in the public". Is this lines 233-237?

5. Section 3: Player localization Following up on Q4, it then says "We use the homography computed by CCBV-SN to estimate the player localization on the field in real-world coordinates from the middle point of the bottom of their bounding box". Is this lines 260-273?

6. Code Blob From #6 I gather that lines 240-257 is to transform the current frame into a top view representation? What's a "calibration cone"? Do you have an image of what it looks like?

7. Section 4: Top view image representations From issue #32 I've gathered that lines 68-82 save the top view images, and I can just edit the save paths to keep them instead of overwriting them. You also read in images src/config/radar.png and src/config/model-radar-mini.png. What are these "radar" images?

8. Section 4: Feature vector representations It seems that lines 89-130 load the model required to get the feature vector representations from the top view representations (depending on which backbone we want to use) and are compute & saved in lines 303-324?

9. Section 4: Player graph representation Is this somewhere in the repo? I can't find the code where this at all.

10. Other What do lines 276-286 and lines 289-297 do?

Hi,

I am new to using SoccerNet. Do you know what code I can run to get the top down views as PNGs? I have all the mkv files downloaded, and here is a picture of the files I have downloaded.

Arth Bohra

Hi Silvio Giancola & SoccerNetv2-DevKit team, this is a great repo you have put together here, thank you so much for making it public!

I came across this SoccerNet-v2 challenge - Tutorial #2 (live session) video from your YouTube channel. I'm following along in "02:18​ Demo 1: Annotation and visualization tool", and upon running main.py, I've come across an error:

Traceback (most recent call last):
  File "main.py", line 4, in <module>
    from interface.main_window import MainWindow
ModuleNotFoundError: No module named 'interface'

My Steps:

1. git clone-ed the repo
2. Created environment based on 4 lines in Annotation's "Getting Started"
3. Ran ./Annotation/actions/src/main.py

If there's anyone that could let me know if this is an error on my end, or how to fix this, I'd be greatly appreciated!

(Note: Regarding the issue I opened a few minutes ago, it was a miss-click but I can't delete it - apologies for the inconvenience)

I would like to know that using this kit, from where to start? Can I use the already trained model for the fine tuning? or I have to train from scratch using your features?

https://github.com/SilvioGiancola/SoccerNetv2-DevKit/issues/5

I'm using model CALF_Calibration_GCN. I want to use U to get the action of an external video.

Hi, I read your Camera Calibration and Player Localization paper, and it was awesome work.

So, I'd like to try to add "ball detection" process, since you detected players only.

Unfortunately, I couldn't find a code which makes "player_boundingbox_maskrcnn.json" and "field_calibration_ccbv.json" files as output.

Could you let me know where can I find the code?

Best Regards.

Hi Silvio, I have some questions about the baidu emdeddings, I wonder if you have any information about them. I couldn't find any information regarding these questions from either of their github repo or the published paper:

1. Is the baidu embeddings already gone through PCA, or was it still a "raw" features? I noticed it's in Tx8576 dimension, which could probably mean they are still "raw", am I right about this?
2. In your opinion, if I were to reduce the dimension, would it be better to have PCA reduce them, or have them go through a FCL like in the implementation of TemporallyAwarePooling? https://github.com/SilvioGiancola/SoccerNetv2-DevKit/blob/20f2f74007c82b68a73c519dff852188df4a8b5a/Task1-ActionSpotting/TemporallyAwarePooling/src/model.py#L32
3. If they are still raw, do you have any idea what was the initial dimension before they are flatten to 8576? They used 398x224 video as mentioned in the paper, but it's not possible to reshape them to it. I was thinking maybe I could used them in video transformer based architecture (MViT etc) if I'm able to reshape them to the original dimension.
4. Do we have any of their fine-tuned feature extraction code publicly available? I think not, but I'm just going to ask anyway in case you know any since their embeddings have very few public information available.

Thanks!

How can i fine tune the "model.pth.tar" model for a dataset that has different output class? lets say the pretrained model("model.pth.tar") has 17 output class but i want to fine-tune it for a dataset which has 4 class.

Originally posted by @skfaysal in https://github.com/SilvioGiancola/SoccerNetv2-DevKit/issues/37#issuecomment-1186591705

Hello, I apologise if this isn't the correct place to raise this issue. But seeing as there isn't a public repository available for the Baidu paper transformer method, I figured there would be more eyes (and willing helpers) here.

I have been trying to replicate the results of the transformer method published above using the baidu soccer embeddings. Unfortunately, it seems that the network is unable to learn meaningful class distinctions and I suspect it is just always predicting the background class. In my experiments using a single dense layer only with the baidu embeddings performs better than when including encoders.

The model summary is given below, I use BCE loss.

Below is the loss curve that I obtained with the LR as stated in the paper (5e-4), batch size 256 -- although I have tried several batch sizes. While the (training) loss does decrease and precision and recall increase briefly, average (training) AUC remains ~0.5. Validation loss also decreases, but all other validation metrics remain constant. When testing, a-mAP is essentially 0.

While I have mainly used keras, I also modified the NetVLAD++ source code to make use of the transformer model (using this encoder and the positional encoder implemented here). Therefore, I could be sure that my results were not just a data loading issue. I used the following parameters for main.py after implementing the transformer method as a pooling method in model.py called "Baidu". Using these same parameters with pool=NetVLAD++ yields proper results.

--features=baidu_soccer_embeddings.npy --max_epochs=50 --framerate=1 --window_size=7 --NMS_window=20 --LR=5e-4 --patience=10 --model_name=Baidu --pool=Baidu

In particular, I added these lines to model.py:

In init:

if not self.input_size == 512:
    if self.input_size == 8576:
        self.feature_extractor = PositionalEncoding(8576)

and at the end of the if/elif chain:

elif self.pool == "Baidu":
    self.enc = torch.nn.TransformerEncoderLayer(8576, 4, 64)
    self.enc = torch.nn.TransformerEncoder(self.enc, num_layers=3)
    self.fc = nn.Linear(8576, self.num_classes + 1)

In the 'forward' function:

if not IC == 512:
    if IC == 8576:
        inputs = inputs.reshape(FR, BS, IC)
        inputs = self.feature_extractor(inputs)
        inputs = inputs.reshape(BS, FR, -1)

and at the end of the if/elif chain:

elif self.pool == "Baidu":
    inputs = self.enc(inputs)
    maxed = torch.max(inputs, 1).values  # also tried flattening
    inputs_pooled = maxed

But the results are similar to those that I obtained with my own code, so I am not sure what the problem could be. While I am not applying mix-up augmentation as stated in the paper, and the training data is less than what they used, I expected the performance to be degraded as a result, but not that I would essentially get 0 a-mAP.

Has anyone else tried to replicate the transformer method? I would appreciate any advice/insights in the name of reproducible research. The intention is to use the transformer method as a baseline in further research.

Accuracy and recall plots show similar behaviour to precision below.

I found that several game matches have a mismatch of the players' bounding boxes and possibly their calibration as well. The games with this problem were the following:

italy_serie-a/2016-2017/2017-03-04 - 17-00 AS Roma 1 - 2 Napoli
italy_serie-a/2015-2016/2015-09-22 - 21-45 Udinese 2 - 3 AC Milan   
italy_serie-a/2015-2016/2015-11-22 - 22-45 Inter 4 - 0 Frosinone
italy_serie-a/2016-2017/2016-08-27 - 21-45 Napoli 4 - 2 AC Milan
italy_serie-a/2016-2017/2016-08-28 - 21-45 Cagliari 2 - 2 AS Roma
italy_serie-a/2016-2017/2016-09-11 - 16-00 AC Milan 0 - 1 Udinese
italy_serie-a/2016-2017/2016-09-16 - 21-45 Sampdoria 0 - 1 AC Milan
italy_serie-a/2016-2017/2016-09-18 - 21-45 Fiorentina 1 - 0 AS Roma
italy_serie-a/2016-2017/2016-09-20 - 21-45 AC Milan 2 - 0 Lazio
italy_serie-a/2016-2017/2016-09-21 - 21-45 AS Roma 4 - 0 Crotone
italy_serie-a/2016-2017/2016-09-25 - 13-30 Torino 3 - 1 AS Roma
italy_serie-a/2016-2017/2016-10-02 - 21-45 AS Roma 2 - 1 Inter
italy_serie-a/2016-2017/2017-02-10 - 22-45 Napoli 2 - 0 Genoa
italy_serie-a/2016-2017/2017-02-25 - 20-00 Napoli 0 - 2 Atalanta
italy_serie-a/2016-2017/2017-04-15 - 21-45 Napoli 3 - 0 Udinese
spain_laliga/2015-2016/2015-09-12 - 17-00 Espanyol 0 - 6 Real Madrid
spain_laliga/2015-2016/2015-09-12 - 21-30 Atl. Madrid 1 - 2 Barcelona   
spain_laliga/2016-2017/2016-11-19 - 18-15 Barcelona 0 - 0 Malaga
spain_laliga/2016-2017/2017-01-08 - 22-45 Villarreal 1 - 1 Barcelona
spain_laliga/2016-2017/2017-04-26 - 22-30 Dep. La Coruna 2 - 6 Real Madrid
spain_laliga/2019-2020/2020-02-09 - 18-00 Osasuna 1 - 4 Real Madrid
spain_laliga/2019-2020/2020-02-16 - 18-00 Real Madrid 2 - 2 Celta Vigo
england_epl/2015-2016/2015-11-07 - 20-30 Stoke City 1 - 0 Chelsea
england_epl/2015-2016/2016-04-09 - 17-00 Swansea 1 - 0 Chelsea
england_epl/2016-2017/2017-01-03 - 18-00 Bournemouth 3 - 3 Arsenal                   
europe_uefa-champions-league/2014-2015/2014-11-04 - 22-45 Arsenal 3 - 3 Anderlecht
europe_uefa-champions-league/2014-2015/2014-11-05 - 22-45 Ajax 0 - 2 Barcelona
europe_uefa-champions-league/2015-2016/2015-11-03 - 22-45 PSV 2 - 0 Wolfsburg   
france_ligue-1/2016-2017/2017-04-18 - 19-30 Metz 2 - 3 Paris SG
germany_bundesliga/2014-2015/2015-05-09 - 16-30 Dortmund 2 - 0 Hertha Berlin 

For the rest of the games the bounding boxes match perfectly. I extract the frames by sampling 2 fps from a half game match by doing the following:

ffmpeg -ss "$start_time" \
       -t "$duration_time" \
       -i "$half_match_HQ_video_path"
       -vf fps=fps=2.:round=down \
       -vsync 1 \
       -q:v 1 \
       "$frames_dest_dir"/%05d.jpg

I initially thought that I was incorrectly sampling frames from the videos, but I also tried with the official frame loader class FrameCV from the DataLoader.py with no success.

I am not quite sure what might be the problem, is there an appropiate way to sample the frames from the videos or are the bounding boxes or is the data wrongly labeled?