Bottleneck Transformers for Visual Recognition

Related tags

Deep Learning transformers imagenet image-classification image-recognition cifar10 bottleneck-transformers
Overview

Bottleneck Transformers for Visual Recognition

Experiments

Model Params (M) Acc (%)
ResNet50 baseline (ref) 23.5M 93.62
BoTNet-50 18.8M 95.11%
BoTNet-S1-50 18.8M 95.67%
BoTNet-S1-59 27.5M 95.98%
BoTNet-S1-77 44.9M wip

Summary

스크린샷 2021-01-28 오후 4 50 19

Usage (example)

  • Model
from model import Model

model = ResNet50(num_classes=1000, resolution=(224, 224))
x = torch.randn([2, 3, 224, 224])
print(model(x).size())
  • Module
from model import MHSA

resolution = 14
mhsa = MHSA(planes, width=resolution, height=resolution)

Reference

  • Paper link
  • Author: Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, Ashish Vaswani
  • Organization: UC Berkeley, Google Research
You might also like...
This is the official pytorch implementation for our ICCV 2021 paper
This is the official pytorch implementation for our ICCV 2021 paper "TRAR: Routing the Attention Spans in Transformers for Visual Question Answering" on VQA Task

🌈 ERASOR (RA-L'21 with ICRA Option) Official page of "ERASOR: Egocentric Ratio of Pseudo Occupancy-based Dynamic Object Removal for Static 3D Point C

Hyungtae Lim 225 Dec 29, 2022
Efficient Training of Visual Transformers with Small Datasets
Efficient Training of Visual Transformers with Small Datasets

Official codes for "Efficient Training of Visual Transformers with Small Datasets", NerIPS 2021.

Yahui Liu 112 Dec 25, 2022
Multivariate Time Series Forecasting with efficient Transformers. Code for the paper
Multivariate Time Series Forecasting with efficient Transformers. Code for the paper "Long-Range Transformers for Dynamic Spatiotemporal Forecasting."

Spacetimeformer Multivariate Forecasting This repository contains the code for the paper, "Long-Range Transformers for Dynamic Spatiotemporal Forecast

QData 440 Jan 2, 2023
[CVPR 2021] Released code for Counterfactual Zero-Shot and Open-Set Visual Recognition
[CVPR 2021] Released code for Counterfactual Zero-Shot and Open-Set Visual Recognition

Counterfactual Zero-Shot and Open-Set Visual Recognition This project provides implementations for our CVPR 2021 paper Counterfactual Zero-S

null 144 Dec 24, 2022
[CVPR 2021] Involution: Inverting the Inherence of Convolution for Visual Recognition, a brand new neural operator
[CVPR 2021] Involution: Inverting the Inherence of Convolution for Visual Recognition, a brand new neural operator

involution Official implementation of a neural operator as described in Involution: Inverting the Inherence of Convolution for Visual Recognition (CVP

Duo Li 1.3k Dec 28, 2022
PyTorch reimplementation of the paper Involution: Inverting the Inherence of Convolution for Visual Recognition [CVPR 2021].

Involution: Inverting the Inherence of Convolution for Visual Recognition Unofficial PyTorch reimplementation of the paper Involution: Inverting the I

Christoph Reich 100 Dec 1, 2022
Implementation of
Implementation of "Distribution Alignment: A Unified Framework for Long-tail Visual Recognition"(CVPR 2021)

Implementation of "Distribution Alignment: A Unified Framework for Long-tail Visual Recognition"(CVPR 2021)

null 105 Nov 7, 2022
Location-Sensitive Visual Recognition with Cross-IOU Loss
Location-Sensitive Visual Recognition with Cross-IOU Loss

The trained models are temporarily unavailable, but you can train the code using reasonable computational resource. Location-Sensitive Visual Recognit

Kaiwen Duan 146 Dec 25, 2022
Conformer: Local Features Coupling Global Representations for Visual Recognition

Conformer: Local Features Coupling Global Representations for Visual Recognition (arxiv) This repository is built upon DeiT and timm Usage First, inst

Zhiliang Peng 378 Jan 8, 2023
Comments
  • A bug shows when the batch_size sets 1

    A bug shows when the batch_size sets 1

    When I set batch_size 1, a bug shows as "ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 512, 1, 1])". I wonder how to solve this problem.

    opened by liuhui0401 1
  • cifar效果达不到啊，直接运行main

    cifar效果达不到啊，直接运行main

    Current Learning Rate: [0.030934962553363768] [Epoch: 251], Loss: 0.085, Acc: 97.030, Correct 12544.0 / Total 12928.0 [Epoch: 251], Loss: 0.084, Acc: 97.104, Correct 24983.0 / Total 25728.0 [Epoch: 251], Loss: 0.081, Acc: 97.194, Correct 37447.0 / Total 38528.0 [Epoch: 251], Acc: 87.820 Current Learning Rate: [0.030032595786498105] [Epoch: 252], Loss: 0.075, Acc: 97.401, Correct 12592.0 / Total 12928.0 [Epoch: 252], Loss: 0.077, Acc: 97.365, Correct 25050.0 / Total 25728.0 [Epoch: 252], Loss: 0.077, Acc: 97.366, Correct 37513.0 / Total 38528.0 [Epoch: 252], Acc: 86.980 Current Learning Rate: [0.029137946110005482] [Epoch: 253], Loss: 0.064, Acc: 97.857, Correct 12651.0 / Total 12928.0 [Epoch: 253], Loss: 0.066, Acc: 97.777, Correct 25156.0 / Total 25728.0 [Epoch: 253], Loss: 0.070, Acc: 97.610, Correct 37607.0 / Total 38528.0 [Epoch: 253], Acc: 87.350 Current Learning Rate: [0.02825135842836657] [Epoch: 254], Loss: 0.073, Acc: 97.563, Correct 12613.0 / Total 12928.0 [Epoch: 254], Loss: 0.074, Acc: 97.477, Correct 25079.0 / Total 25728.0 [Epoch: 254], Loss: 0.070, Acc: 97.560, Correct 37588.0 / Total 38528.0 [Epoch: 254], Acc: 86.640 Current Learning Rate: [0.02737317453800964] [Epoch: 255], Loss: 0.063, Acc: 97.826, Correct 12647.0 / Total 12928.0 [Epoch: 255], Loss: 0.061, Acc: 97.889, Correct 25185.0 / Total 25728.0 [Epoch: 255], Loss: 0.065, Acc: 97.742, Correct 37658.0 / Total 38528.0 [Epoch: 255], Acc: 87.650 Current Learning Rate: [0.026503732995541415] [Epoch: 256], Loss: 0.064, Acc: 97.803, Correct 12644.0 / Total 12928.0 [Epoch: 256], Loss: 0.064, Acc: 97.831, Correct 25170.0 / Total 25728.0 [Epoch: 256], Loss: 0.063, Acc: 97.861, Correct 37704.0 / Total 38528.0 [Epoch: 256], Acc: 87.970 Current Learning Rate: [0.025643368987227095] [Epoch: 257], Loss: 0.066, Acc: 97.788, Correct 12642.0 / Total 12928.0 [Epoch: 257], Loss: 0.066, Acc: 97.804, Correct 25163.0 / Total 25728.0 [Epoch: 257], Loss: 0.065, Acc: 97.835, Correct 37694.0 / Total 38528.0 [Epoch: 257], Acc: 87.380 Current Learning Rate: [0.02479241419976968] [Epoch: 258], Loss: 0.055, Acc: 98.198, Correct 12695.0 / Total 12928.0 [Epoch: 258], Loss: 0.053, Acc: 98.204, Correct 25266.0 / Total 25728.0 [Epoch: 258], Loss: 0.055, Acc: 98.183, Correct 37828.0 / Total 38528.0 [Epoch: 258], Acc: 87.930 Current Learning Rate: [0.023951196692438358] [Epoch: 259], Loss: 0.049, Acc: 98.213, Correct 12697.0 / Total 12928.0 [Epoch: 259], Loss: 0.051, Acc: 98.231, Correct 25273.0 / Total 25728.0 [Epoch: 259], Loss: 0.054, Acc: 98.108, Correct 37799.0 / Total 38528.0 [Epoch: 259], Acc: 87.880 Current Learning Rate: [0.023120040770595558] [Epoch: 260], Loss: 0.046, Acc: 98.430, Correct 12725.0 / Total 12928.0 [Epoch: 260], Loss: 0.050, Acc: 98.340, Correct 25301.0 / Total 25728.0 [Epoch: 260], Loss: 0.051, Acc: 98.271, Correct 37862.0 / Total 38528.0 [Epoch: 260], Acc: 87.700 Current Learning Rate: [0.022299266860670866] [Epoch: 261], Loss: 0.052, Acc: 98.229, Correct 12699.0 / Total 12928.0 [Epoch: 261], Loss: 0.051, Acc: 98.278, Correct 25285.0 / Total 25728.0 [Epoch: 261], Loss: 0.050, Acc: 98.279, Correct 37865.0 / Total 38528.0 [Epoch: 261], Acc: 88.190 Current Learning Rate: [0.021489191386630774] [Epoch: 262], Loss: 0.046, Acc: 98.391, Correct 12720.0 / Total 12928.0 [Epoch: 262], Loss: 0.046, Acc: 98.395, Correct 25315.0 / Total 25728.0 [Epoch: 262], Loss: 0.047, Acc: 98.378, Correct 37903.0 / Total 38528.0 [Epoch: 262], Acc: 87.440 Current Learning Rate: [0.020690126647990973] [Epoch: 263], Loss: 0.041, Acc: 98.577, Correct 12744.0 / Total 12928.0 [Epoch: 263], Loss: 0.043, Acc: 98.496, Correct 25341.0 / Total 25728.0 [Epoch: 263], Loss: 0.045, Acc: 98.435, Correct 37925.0 / Total 38528.0 [Epoch: 263], Acc: 87.740 Current Learning Rate: [0.019902380699419107] [Epoch: 264], Loss: 0.041, Acc: 98.700, Correct 12760.0 / Total 12928.0 [Epoch: 264], Loss: 0.040, Acc: 98.706, Correct 25395.0 / Total 25728.0 [Epoch: 264], Loss: 0.040, Acc: 98.679, Correct 38019.0 / Total 38528.0 [Epoch: 264], Acc: 87.720 Current Learning Rate: [0.019126257231973805] [Epoch: 265], Loss: 0.037, Acc: 98.824, Correct 12776.0 / Total 12928.0 [Epoch: 265], Loss: 0.038, Acc: 98.764, Correct 25410.0 / Total 25728.0 [Epoch: 265], Loss: 0.039, Acc: 98.713, Correct 38032.0 / Total 38528.0 [Epoch: 265], Acc: 88.020 Current Learning Rate: [0.018362055456025896] [Epoch: 266], Loss: 0.034, Acc: 98.971, Correct 12795.0 / Total 12928.0 [Epoch: 266], Loss: 0.034, Acc: 98.865, Correct 25436.0 / Total 25728.0 [Epoch: 266], Loss: 0.038, Acc: 98.731, Correct 38039.0 / Total 38528.0 [Epoch: 266], Acc: 88.280 Current Learning Rate: [0.01761006998590733] [Epoch: 267], Loss: 0.028, Acc: 99.033, Correct 12803.0 / Total 12928.0 [Epoch: 267], Loss: 0.030, Acc: 98.978, Correct 25465.0 / Total 25728.0 [Epoch: 267], Loss: 0.031, Acc: 98.936, Correct 38118.0 / Total 38528.0 [Epoch: 267], Acc: 88.010 Current Learning Rate: [0.016870590726331475] [Epoch: 268], Loss: 0.030, Acc: 99.033, Correct 12803.0 / Total 12928.0 [Epoch: 268], Loss: 0.031, Acc: 98.989, Correct 25468.0 / Total 25728.0 [Epoch: 268], Loss: 0.032, Acc: 98.928, Correct 38115.0 / Total 38528.0 [Epoch: 268], Acc: 88.690 Current Learning Rate: [0.016143902760629568] [Epoch: 269], Loss: 0.028, Acc: 99.087, Correct 12810.0 / Total 12928.0 [Epoch: 269], Loss: 0.027, Acc: 99.122, Correct 25502.0 / Total 25728.0 [Epoch: 269], Loss: 0.028, Acc: 99.110, Correct 38185.0 / Total 38528.0 [Epoch: 269], Acc: 88.200 Current Learning Rate: [0.015430286240845494] [Epoch: 270], Loss: 0.028, Acc: 99.010, Correct 12800.0 / Total 12928.0 [Epoch: 270], Loss: 0.025, Acc: 99.090, Correct 25494.0 / Total 25728.0 [Epoch: 270], Loss: 0.026, Acc: 99.079, Correct 38173.0 / Total 38528.0 [Epoch: 270], Acc: 88.660 Current Learning Rate: [0.014730016279731955] [Epoch: 271], Loss: 0.028, Acc: 99.103, Correct 12812.0 / Total 12928.0 [Epoch: 271], Loss: 0.025, Acc: 99.172, Correct 25515.0 / Total 25728.0 [Epoch: 271], Loss: 0.026, Acc: 99.164, Correct 38206.0 / Total 38528.0 [Epoch: 271], Acc: 88.780 Best Model Saving... Current Learning Rate: [0.014043362844689204] [Epoch: 272], Loss: 0.023, Acc: 99.273, Correct 12834.0 / Total 12928.0 [Epoch: 272], Loss: 0.026, Acc: 99.122, Correct 25502.0 / Total 25728.0 [Epoch: 272], Loss: 0.025, Acc: 99.167, Correct 38207.0 / Total 38528.0 [Epoch: 272], Acc: 88.590 Current Learning Rate: [0.0133705906536875] [Epoch: 273], Loss: 0.024, Acc: 99.188, Correct 12823.0 / Total 12928.0 [Epoch: 273], Loss: 0.020, Acc: 99.386, Correct 25570.0 / Total 25728.0 [Epoch: 273], Loss: 0.019, Acc: 99.403, Correct 38298.0 / Total 38528.0 [Epoch: 273], Acc: 88.260 Current Learning Rate: [0.0127119590732133] [Epoch: 274], Loss: 0.023, Acc: 99.157, Correct 12819.0 / Total 12928.0 [Epoch: 274], Loss: 0.021, Acc: 99.265, Correct 25539.0 / Total 25728.0 [Epoch: 274], Loss: 0.020, Acc: 99.299, Correct 38258.0 / Total 38528.0 [Epoch: 274], Acc: 88.300 Current Learning Rate: [0.012067722018278455] [Epoch: 275], Loss: 0.018, Acc: 99.373, Correct 12847.0 / Total 12928.0 [Epoch: 275], Loss: 0.018, Acc: 99.378, Correct 25568.0 / Total 25728.0 [Epoch: 275], Loss: 0.018, Acc: 99.377, Correct 38288.0 / Total 38528.0 [Epoch: 275], Acc: 88.700 Current Learning Rate: [0.011438127854531303] [Epoch: 276], Loss: 0.015, Acc: 99.590, Correct 12875.0 / Total 12928.0 [Epoch: 276], Loss: 0.015, Acc: 99.569, Correct 25617.0 / Total 25728.0 [Epoch: 276], Loss: 0.015, Acc: 99.564, Correct 38360.0 / Total 38528.0 [Epoch: 276], Acc: 88.500 Current Learning Rate: [0.010823419302506784] [Epoch: 277], Loss: 0.017, Acc: 99.404, Correct 12851.0 / Total 12928.0 [Epoch: 277], Loss: 0.016, Acc: 99.456, Correct 25588.0 / Total 25728.0 [Epoch: 277], Loss: 0.016, Acc: 99.452, Correct 38317.0 / Total 38528.0 [Epoch: 277], Acc: 88.690 Current Learning Rate: [0.010223833344053286] [Epoch: 278], Loss: 0.014, Acc: 99.520, Correct 12866.0 / Total 12928.0 [Epoch: 278], Loss: 0.015, Acc: 99.518, Correct 25604.0 / Total 25728.0 [Epoch: 278], Loss: 0.014, Acc: 99.525, Correct 38345.0 / Total 38528.0 [Epoch: 278], Acc: 89.290 Best Model Saving... Current Learning Rate: [0.00963960113097138] [Epoch: 279], Loss: 0.012, Acc: 99.629, Correct 12880.0 / Total 12928.0 [Epoch: 279], Loss: 0.013, Acc: 99.600, Correct 25625.0 / Total 25728.0 [Epoch: 279], Loss: 0.013, Acc: 99.624, Correct 38383.0 / Total 38528.0 [Epoch: 279], Acc: 89.070 Current Learning Rate: [0.009070947895900596] [Epoch: 280], Loss: 0.011, Acc: 99.675, Correct 12886.0 / Total 12928.0 [Epoch: 280], Loss: 0.011, Acc: 99.666, Correct 25642.0 / Total 25728.0 [Epoch: 280], Loss: 0.011, Acc: 99.678, Correct 38404.0 / Total 38528.0 [Epoch: 280], Acc: 89.250 Current Learning Rate: [0.008518092865487875] [Epoch: 281], Loss: 0.011, Acc: 99.667, Correct 12885.0 / Total 12928.0 [Epoch: 281], Loss: 0.011, Acc: 99.708, Correct 25653.0 / Total 25728.0 [Epoch: 281], Loss: 0.011, Acc: 99.655, Correct 38395.0 / Total 38528.0 [Epoch: 281], Acc: 89.110 Current Learning Rate: [0.007981249175871482] [Epoch: 282], Loss: 0.011, Acc: 99.652, Correct 12883.0 / Total 12928.0 [Epoch: 282], Loss: 0.011, Acc: 99.670, Correct 25643.0 / Total 25728.0 [Epoch: 282], Loss: 0.010, Acc: 99.689, Correct 38408.0 / Total 38528.0 [Epoch: 282], Acc: 89.260 Current Learning Rate: [0.007460623790513096] [Epoch: 283], Loss: 0.008, Acc: 99.737, Correct 12894.0 / Total 12928.0 [Epoch: 283], Loss: 0.009, Acc: 99.740, Correct 25661.0 / Total 25728.0 [Epoch: 283], Loss: 0.009, Acc: 99.725, Correct 38422.0 / Total 38528.0 [Epoch: 283], Acc: 89.420 Best Model Saving... Current Learning Rate: [0.006956417420409298] [Epoch: 284], Loss: 0.010, Acc: 99.683, Correct 12887.0 / Total 12928.0 [Epoch: 284], Loss: 0.010, Acc: 99.697, Correct 25650.0 / Total 25728.0 [Epoch: 284], Loss: 0.010, Acc: 99.689, Correct 38408.0 / Total 38528.0 [Epoch: 284], Acc: 89.370 Current Learning Rate: [0.0064688244467137924] [Epoch: 285], Loss: 0.006, Acc: 99.838, Correct 12907.0 / Total 12928.0 [Epoch: 285], Loss: 0.007, Acc: 99.802, Correct 25677.0 / Total 25728.0 [Epoch: 285], Loss: 0.007, Acc: 99.785, Correct 38445.0 / Total 38528.0 [Epoch: 285], Acc: 89.180 Current Learning Rate: [0.005998032845799671] [Epoch: 286], Loss: 0.006, Acc: 99.845, Correct 12908.0 / Total 12928.0 [Epoch: 286], Loss: 0.006, Acc: 99.817, Correct 25681.0 / Total 25728.0 [Epoch: 286], Loss: 0.006, Acc: 99.795, Correct 38449.0 / Total 38528.0 [Epoch: 286], Acc: 89.140 Current Learning Rate: [0.0055442241167910295] [Epoch: 287], Loss: 0.008, Acc: 99.737, Correct 12894.0 / Total 12928.0 [Epoch: 287], Loss: 0.007, Acc: 99.759, Correct 25666.0 / Total 25728.0 [Epoch: 287], Loss: 0.007, Acc: 99.779, Correct 38443.0 / Total 38528.0 [Epoch: 287], Acc: 89.310 Current Learning Rate: [0.005107573211591536] [Epoch: 288], Loss: 0.007, Acc: 99.729, Correct 12893.0 / Total 12928.0 [Epoch: 288], Loss: 0.007, Acc: 99.759, Correct 25666.0 / Total 25728.0 [Epoch: 288], Loss: 0.007, Acc: 99.756, Correct 38434.0 / Total 38528.0 [Epoch: 288], Acc: 89.330 Current Learning Rate: [0.004688248467437186] [Epoch: 289], Loss: 0.007, Acc: 99.783, Correct 12900.0 / Total 12928.0 [Epoch: 289], Loss: 0.007, Acc: 99.817, Correct 25681.0 / Total 25728.0 [Epoch: 289], Loss: 0.006, Acc: 99.834, Correct 38464.0 / Total 38528.0 [Epoch: 289], Acc: 89.370 Current Learning Rate: [0.004286411541999064] [Epoch: 290], Loss: 0.006, Acc: 99.830, Correct 12906.0 / Total 12928.0 [Epoch: 290], Loss: 0.005, Acc: 99.841, Correct 25687.0 / Total 25728.0 [Epoch: 290], Loss: 0.005, Acc: 99.839, Correct 38466.0 / Total 38528.0 [Epoch: 290], Acc: 89.200 Current Learning Rate: [0.0039022173510612273] [Epoch: 291], Loss: 0.006, Acc: 99.845, Correct 12908.0 / Total 12928.0 [Epoch: 291], Loss: 0.006, Acc: 99.829, Correct 25684.0 / Total 25728.0 [Epoch: 291], Loss: 0.005, Acc: 99.834, Correct 38464.0 / Total 38528.0 [Epoch: 291], Acc: 89.350 Current Learning Rate: [0.003535814008797773] [Epoch: 292], Loss: 0.004, Acc: 99.876, Correct 12912.0 / Total 12928.0 [Epoch: 292], Loss: 0.005, Acc: 99.837, Correct 25686.0 / Total 25728.0 [Epoch: 292], Loss: 0.005, Acc: 99.826, Correct 38461.0 / Total 38528.0 [Epoch: 292], Acc: 89.340 Current Learning Rate: [0.003187342770671916] [Epoch: 293], Loss: 0.004, Acc: 99.930, Correct 12919.0 / Total 12928.0 [Epoch: 293], Loss: 0.004, Acc: 99.880, Correct 25697.0 / Total 25728.0 [Epoch: 293], Loss: 0.004, Acc: 99.875, Correct 38480.0 / Total 38528.0 [Epoch: 293], Acc: 89.560 Best Model Saving... Current Learning Rate: [0.002856937978979447] [Epoch: 294], Loss: 0.004, Acc: 99.884, Correct 12913.0 / Total 12928.0 [Epoch: 294], Loss: 0.004, Acc: 99.883, Correct 25698.0 / Total 25728.0 [Epoch: 294], Loss: 0.004, Acc: 99.873, Correct 38479.0 / Total 38528.0 [Epoch: 294], Acc: 89.490 Current Learning Rate: [0.002544727011057081] [Epoch: 295], Loss: 0.004, Acc: 99.892, Correct 12914.0 / Total 12928.0 [Epoch: 295], Loss: 0.004, Acc: 99.883, Correct 25698.0 / Total 25728.0 [Epoch: 295], Loss: 0.004, Acc: 99.881, Correct 38482.0 / Total 38528.0 [Epoch: 295], Acc: 89.650 Best Model Saving... Current Learning Rate: [0.002250830230176169] [Epoch: 296], Loss: 0.004, Acc: 99.853, Correct 12909.0 / Total 12928.0 [Epoch: 296], Loss: 0.004, Acc: 99.887, Correct 25699.0 / Total 25728.0 [Epoch: 296], Loss: 0.004, Acc: 99.881, Correct 38482.0 / Total 38528.0 [Epoch: 296], Acc: 89.450 Current Learning Rate: [0.001975360939140324] [Epoch: 297], Loss: 0.003, Acc: 99.899, Correct 12915.0 / Total 12928.0 [Epoch: 297], Loss: 0.003, Acc: 99.903, Correct 25703.0 / Total 25728.0 [Epoch: 297], Loss: 0.003, Acc: 99.901, Correct 38490.0 / Total 38528.0 [Epoch: 297], Acc: 89.440 Current Learning Rate: [0.0017184253366050195] [Epoch: 298], Loss: 0.004, Acc: 99.884, Correct 12913.0 / Total 12928.0 [Epoch: 298], Loss: 0.004, Acc: 99.868, Correct 25694.0 / Total 25728.0 [Epoch: 298], Loss: 0.004, Acc: 99.868, Correct 38477.0 / Total 38528.0 [Epoch: 298], Acc: 89.690 Best Model Saving... Current Learning Rate: [0.001480122476136056] [Epoch: 299], Loss: 0.003, Acc: 99.915, Correct 12917.0 / Total 12928.0 [Epoch: 299], Loss: 0.003, Acc: 99.911, Correct 25705.0 / Total 25728.0 [Epoch: 299], Loss: 0.004, Acc: 99.899, Correct 38489.0 / Total 38528.0 [Epoch: 299], Acc: 89.680 Current Learning Rate: [0.0012605442280224245] [Epoch: 300], Loss: 0.003, Acc: 99.930, Correct 12919.0 / Total 12928.0 [Epoch: 300], Loss: 0.003, Acc: 99.934, Correct 25711.0 / Total 25728.0 [Epoch: 300], Loss: 0.003, Acc: 99.922, Correct 38498.0 / Total 38528.0 [Epoch: 300], Acc: 89.660 Current Learning Rate: [0.00105977524385864] [Epoch: 301], Loss: 0.002, Acc: 99.954, Correct 12922.0 / Total 12928.0 [Epoch: 301], Loss: 0.002, Acc: 99.934, Correct 25711.0 / Total 25728.0 [Epoch: 301], Loss: 0.003, Acc: 99.907, Correct 38492.0 / Total 38528.0 [Epoch: 301], Acc: 89.690 Current Learning Rate: [0.0008778929239099148] [Epoch: 302], Loss: 0.003, Acc: 99.907, Correct 12916.0 / Total 12928.0 [Epoch: 302], Loss: 0.003, Acc: 99.918, Correct 25707.0 / Total 25728.0 [Epoch: 302], Loss: 0.003, Acc: 99.920, Correct 38497.0 / Total 38528.0 [Epoch: 302], Acc: 89.660 Current Learning Rate: [0.000714967387272874] [Epoch: 303], Loss: 0.003, Acc: 99.884, Correct 12913.0 / Total 12928.0 [Epoch: 303], Loss: 0.003, Acc: 99.899, Correct 25702.0 / Total 25728.0 [Epoch: 303], Loss: 0.003, Acc: 99.904, Correct 38491.0 / Total 38528.0 [Epoch: 303], Acc: 89.600 Current Learning Rate: [0.0005710614448433164] [Epoch: 304], Loss: 0.004, Acc: 99.899, Correct 12915.0 / Total 12928.0 [Epoch: 304], Loss: 0.003, Acc: 99.911, Correct 25705.0 / Total 25728.0 [Epoch: 304], Loss: 0.003, Acc: 99.914, Correct 38495.0 / Total 38528.0 [Epoch: 304], Acc: 89.660 Current Learning Rate: [0.0004462305751014317] [Epoch: 305], Loss: 0.003, Acc: 99.892, Correct 12914.0 / Total 12928.0 [Epoch: 305], Loss: 0.003, Acc: 99.914, Correct 25706.0 / Total 25728.0 [Epoch: 305], Loss: 0.003, Acc: 99.927, Correct 38500.0 / Total 38528.0 [Epoch: 305], Acc: 89.630 Current Learning Rate: [0.00034052290272376895] [Epoch: 306], Loss: 0.002, Acc: 99.954, Correct 12922.0 / Total 12928.0 [Epoch: 306], Loss: 0.002, Acc: 99.934, Correct 25711.0 / Total 25728.0 [Epoch: 306], Loss: 0.003, Acc: 99.912, Correct 38494.0 / Total 38528.0 [Epoch: 306], Acc: 89.620 Current Learning Rate: [0.0002539791800302582] [Epoch: 307], Loss: 0.002, Acc: 99.946, Correct 12921.0 / Total 12928.0 [Epoch: 307], Loss: 0.002, Acc: 99.949, Correct 25715.0 / Total 25728.0 [Epoch: 307], Loss: 0.002, Acc: 99.943, Correct 38506.0 / Total 38528.0 [Epoch: 307], Acc: 89.630 Current Learning Rate: [0.00018663277127344463] [Epoch: 308], Loss: 0.003, Acc: 99.884, Correct 12913.0 / Total 12928.0 [Epoch: 308], Loss: 0.003, Acc: 99.883, Correct 25698.0 / Total 25728.0 [Epoch: 308], Loss: 0.003, Acc: 99.891, Correct 38486.0 / Total 38528.0 [Epoch: 308], Acc: 89.550 Current Learning Rate: [0.0001385096397758911] [Epoch: 309], Loss: 0.003, Acc: 99.938, Correct 12920.0 / Total 12928.0 [Epoch: 309], Loss: 0.003, Acc: 99.930, Correct 25710.0 / Total 25728.0 [Epoch: 309], Loss: 0.003, Acc: 99.920, Correct 38497.0 / Total 38528.0 [Epoch: 309], Acc: 89.700 Best Model Saving... Current Learning Rate: [0.00010962833792086233] [Epoch: 310], Loss: 0.002, Acc: 99.961, Correct 12923.0 / Total 12928.0 [Epoch: 310], Loss: 0.002, Acc: 99.953, Correct 25716.0 / Total 25728.0 [Epoch: 310], Loss: 0.003, Acc: 99.945, Correct 38507.0 / Total 38528.0 [Epoch: 310], Acc: 89.670 Current Learning Rate: [0.1]

    opened by HHEjie123 4
  • A question about the size of the vector input to resnet and MHSA

    A question about the size of the vector input to resnet and MHSA

    Hello, I would like to ask whether H and W of input vectors must be fixed when using your Resnet50 and MHSA modules?Is it possible to automatically adjust the image size according to each batsize?？

    opened by xiaosa96 1
  • RuntimeError: The size of tensor a (4) must match the size of tensor b (196) at non-singleton dimension 2

    RuntimeError: The size of tensor a (4) must match the size of tensor b (196) at non-singleton dimension 2

    RuntimeError: The size of tensor a (4) must match the size of tensor b (196) at non-singleton dimension 2

    I'm using your code. And I didn't make any changes

    opened by YePG 3
Owner
Myeongjun Kim
Computer Vision Research using Deep Learning
Myeongjun Kim
GitHub
Implementation of Bottleneck Transformer in Pytorch

Bottleneck Transformer - Pytorch Implementation of Bottleneck Transformer, SotA visual recognition model with convolution + attention that outperforms

Phil Wang 621 Jan 6, 2023
codes for Image Inpainting with External-internal Learning and Monochromic Bottleneck

Image Inpainting with External-internal Learning and Monochromic Bottleneck This repository is for the CVPR 2021 paper: 'Image Inpainting with Externa

null 97 Nov 29, 2022
Code for the paper: "On the Bottleneck of Graph Neural Networks and Its Practical Implications"

On the Bottleneck of Graph Neural Networks and its Practical Implications This is the official implementation of the paper: On the Bottleneck of Graph

null 75 Dec 22, 2022
A self-supervised 3D representation learning framework named viewpoint bottleneck.

Pointly-supervised 3D Scene Parsing with Viewpoint Bottleneck Paper Created by Liyi Luo, Beiwen Tian, Hao Zhao and Guyue Zhou from Institute for AI In

null 63 Aug 11, 2022
A self-supervised 3D representation learning framework named viewpoint bottleneck.

Pointly-supervised 3D Scene Parsing with Viewpoint Bottleneck Paper Created by Liyi Luo, Beiwen Tian, Hao Zhao and Guyue Zhou from Institute for AI In

null 42 Sep 24, 2021
Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation (NeurIPS 2021)

Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation (NeurIPS 2021) The implementation of Reducing Infromation Bottleneck for W

Jungbeom Lee 81 Dec 16, 2022
Official PyTorch Implementation for InfoSwap: Information Bottleneck Disentanglement for Identity Swapping

InfoSwap: Information Bottleneck Disentanglement for Identity Swapping Code usage Please check out the user manual page. Paper Gege Gao, Huaibo Huang,

Grace Hešeri 56 Dec 20, 2022
improvement of CLIP features over the traditional resnet features on the visual question answering, image captioning, navigation and visual entailment tasks.

CLIP-ViL In our paper "How Much Can CLIP Benefit Vision-and-Language Tasks?", we show the improvement of CLIP features over the traditional resnet fea

null 310 Dec 28, 2022
Official implementation of the paper Visual Parser: Representing Part-whole Hierarchies with Transformers

Visual Parser (ViP) This is the official implementation of the paper Visual Parser: Representing Part-whole Hierarchies with Transformers. Key Feature

Shuyang Sun 117 Dec 11, 2022
Pytorch implementation for our ICCV 2021 paper "TRAR: Routing the Attention Spans in Transformers for Visual Question Answering".

TRAnsformer Routing Networks (TRAR) This is an official implementation for ICCV 2021 paper "TRAR: Routing the Attention Spans in Transformers for Visu

Ren Tianhe 49 Nov 10, 2022