Hi @chinhsuanwu,

Could you please update your model by adding initialisation code? And apple has released the weights for MobileViTs, could you also please add option to load pre-trained weights?

Many thanks, Yiming

Thanks for your code which helps me a lot. After read your implementation, I found that the head num for attention module is set to 1 for every Transformer module. How did you know that heas num should be set to 1. I can not find any discription about this in the original article.

@chinhsuanwu , thank you so much for uploading the code.

• There are small issues regarding the structure of model that you've designed, I have updated the code accordingly in my repository over here.

• Basically, I referred to the original code present over here . I inferenced the original detector model on some sample data and compared their outputs with a detector designed from your code, at every stage. Then I found that there was a huge disparity between both of these models.

• I have changed your code at some parts, and the results that it gave were similar to the original one, the link to the model file is here.

• Please let me know if you want me to elaborate further on this topic.

• The complete repository can be found here

I test my code using pretrained mobilevit_xs and pretrained HRNet(Deep High-Resolution Representation Learning for Visual Recognition) by my dataset. I checked that mobilevit_xs have less x8 parameters than HRNet and my pretrained weight of mobilevit_xs (85MB) is smaller than pretrained weight of HRNet (254MB).

When i test the model using eval image, the model using mobilevit_xs occured CUDA out of memory on RTX 3070. However, HRNet model work normally.

Why do not work the mobilevit_xs model? Eval image resolution is 1080 x 1920.

Is it because of key, query, value operations?

Error message is below. Traceback (most recent call last): File ".\streaming_points.py", line 213, in writeVideo(args) File ".\streaming_points.py", line 146, in writeVideo model = build_model(args.weight_path) File ".\streaming_points.py", line 114, in build_model _ = model(dummy_tensor) File "C:\Users\user\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "C:\Users\user\anaconda3\envs\pytorch\pythonProject\Chicken Counting\live_code\Networks\MobileViT\mobilevit.py", line 224, in forward x = self.mvit0 File "C:\Users\user\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "C:\Users\user\anaconda3\envs\pytorch\pythonProject\Chicken Counting\live_code\Networks\MobileViT\mobilevit.py", line 161, in forward x = self.transformer(x) File "C:\Users\user\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "C:\Users\user\anaconda3\envs\pytorch\pythonProject\Chicken Counting\live_code\Networks\MobileViT\mobilevit.py", line 89, in forward x = attn(x) + x File "C:\Users\user\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "C:\Users\user\anaconda3\envs\pytorch\pythonProject\Chicken Counting\live_code\Networks\MobileViT\mobilevit.py", line 31, in forward return self.fn(self.norm(x), **kwargs) File "C:\Users\user\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "C:\Users\user\anaconda3\envs\pytorch\pythonProject\Chicken Counting\live_code\Networks\MobileViT\mobilevit.py", line 70, in forward dots = torch.matmul(q, k.transpose(-1, -2)) * self.scale RuntimeError: CUDA out of memory. Tried to allocate 1.99 GiB (GPU 0; 8.00 GiB total capacity; 3.85 GiB already allocated; 1.53 GiB free; 3.87 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_s plit_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

If I want to use Mobilevit in Cifar10, how should I set the parameters?

Because I changed the input size, but the parameters don't match.

For example:

EinopsError: Shape mismatch, can't divide axis of length 2 in chunks of 4

During handling of the above exception, another exception occurred:

EinopsError Traceback (most recent call last) C:\Users\ADMINI~1\AppData\Local\Temp/ipykernel_16468/2105769576.py in 44 torch.cuda.empty_cache() 45 if name == "main": ---> 46 main()

C:\Users\ADMINI~1\AppData\Local\Temp/ipykernel_16468/2105769576.py in main() 31 32 for epoch in range(1, total_epoch+1): ---> 33 train_one_epoch(model, 34 train_dataloader, 35 criterion,

C:\Users\ADMINI~1\AppData\Local\Temp/ipykernel_16468/4152541084.py in train_one_epoch(model, dataloader, criterion, optimizer, epoch, total_epoch, report_freq) 70 label = data[1].to(device) 71 ---> 72 out = model(image) 73 loss = criterion(out, label) 74

D:\anaconda\envs\CoAtnet\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs) 1100 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks 1101 or _global_forward_hooks or _global_forward_pre_hooks): -> 1102 return forward_call(*input, **kwargs) 1103 # Do not call functions when jit is used 1104 full_backward_hooks, non_full_backward_hooks = [], []

E:\code\awesome_lightweight_networks-main\light_cnns\Transformer\mobile_vit.py in forward(self, x) 216 217 x = self.mv25 --> 218 x = self.mvit1 219 220 x = self.mv26

D:\anaconda\envs\CoAtnet\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs) 1100 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks 1101 or _global_forward_hooks or _global_forward_pre_hooks): -> 1102 return forward_call(*input, **kwargs) 1103 # Do not call functions when jit is used 1104 full_backward_hooks, non_full_backward_hooks = [], []

E:\code\awesome_lightweight_networks-main\light_cnns\Transformer\mobile_vit.py in forward(self, x) 162 # Global representations 163 _, _, h, w = x.shape --> 164 x = rearrange(x, 'b d (h ph) (w pw) -> b (ph pw) (h w) d', ph=self.ph, pw=self.pw) 165 x = self.transformer(x) 166 x = rearrange(x, 'b (ph pw) (h w) d -> b d (h ph) (w pw)', h=h // self.ph, w=w // self.pw, ph=self.ph,

D:\anaconda\envs\CoAtnet\lib\site-packages\einops\einops.py in rearrange(tensor, pattern, **axes_lengths) 450 raise TypeError("Rearrange can't be applied to an empty list") 451 tensor = get_backend(tensor[0]).stack_on_zeroth_dimension(tensor) --> 452 return reduce(tensor, pattern, reduction='rearrange', **axes_lengths) 453 454

D:\anaconda\envs\CoAtnet\lib\site-packages\einops\einops.py in reduce(tensor, pattern, reduction, **axes_lengths) 388 message += '\n Input is list. ' 389 message += 'Additional info: {}.'.format(axes_lengths) --> 390 raise EinopsError(message + '\n {}'.format(e)) 391 392

EinopsError: Error while processing rearrange-reduction pattern "b d (h ph) (w pw) -> b (ph pw) (h w) d". Input tensor shape: torch.Size([200, 80, 2, 2]). Additional info: {'ph': 4, 'pw': 4}. Shape mismatch, can't divide axis of length 2 in chunks of 4