TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?
A PyTorch implementation of TokenLearner: What Can 8 Learned Tokens Do for Images and Videos? [1-2]. Unlike another Unofficial PyTorch implementation [3], our version is heavily borrowed from the official implementation [4] and TensorFlow implementation[5], and try to keep consistent with them.
Usage
You can access the TokenLearner and TokenLearnerModuleV11 class from the tokenlearner file. You can use this layer with a Vision Transformer, MLPMixer, or Video Vision Transformer as done in the paper.
import torch
from tokenlearner import TokenLearner
tklr = TokenLearner(in_channels=128, num_tokens=8, use_sum_pooling=False)
x = torch.ones(256, 32, 32, 128) # [bs, h, w, c]
y1 = tklr(x)
print(y1.shape) # [256, 8, 128]
You can also use TokenLearnerModuleV11, which aligns with the official implementation.
import torch
from tokenlearner import TokenLearnerModuleV11
tklr_v11 = TokenLearnerModuleV11(in_channels=128, num_tokens=8, num_groups=4, dropout_rate=0.)
tklr_v11.eval() # control droput
x = torch.ones(256, 32, 32, 128) # [bs, h, w, c]
y2 = tklr_v11(x)
print(y2.shape) # [256, 8, 128]
References
[1] TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?; Ryoo et al.; arXiv 2021; https://arxiv.org/abs/2106.11297
[2] TokenLearner: Adaptive Space-Time Tokenization for Videos; Ryoo et al., NeurIPS 2021; https://openreview.net/forum?id=z-l1kpDXs88
6.9k Jan 3, 2023
119 Nov 24, 2022
1.4k Jan 1, 2023
360 Dec 10, 2022
9.2k Jan 2, 2023
0 Jul 13, 2021
8 Nov 21, 2022
157 Dec 11, 2022
即input.shape #[bs,h,w,c]
请问这里的bs代表什么含义呢？
另外在output的位置， 您的注释写成了#[B,n_tokens,c]，这里的B代表什么含义呢？
48 Sep 27, 2022
90 Dec 8, 2022
121 Dec 17, 2022
172 Dec 12, 2022
111 Dec 8, 2022
76 Jan 7, 2023
13 Nov 19, 2022
556 Jan 4, 2023
616 Jan 6, 2023
520 Dec 30, 2022