Hi Author,

I need to add some extra output tensors which are used for inference. These tensors are not referenced during training but just for inference after the conversion to tflite. My naive intention is to put some operations in forward as many as possible so as to relief the loading of post processing which has to be implemented by c/c++ code.

For instance, some matrix ops such as reshape/sigmoid/multiply better be done by GPU/NPU instead of with CPU.

I add some logic in forward to implement this requirement and the training goes well but the conversion to tflite fails with following error message: File "/usr/local/lib/python3.6/dist-packages/torch/nn/quantized/modules/functional_modules.py", line 160, in mul r = ops.quantized.mul(x, y, scale=self.scale, zero_point=self.zero_point) RuntimeError: Mul operands should have same data type.

Is there any feasible way or workaround for this scenario? The script attached. Thanks. movenet_qat.zip

Please see the below colab that I am using to convert mobilet v2 from pytorch to tflite(int8) https://colab.research.google.com/drive/1eW-I0RDzB3L6Zbz364t5lkI4fxgvpGbI#scrollTo=5YtQg5Ga2wmq

Getting the below erro]rs Traceback (most recent call last): File "./examples/converter/convert.py", line 9, in from examples.models.cifar10.mobilenet import DEFAULT_STATE_DICT, Mobilenet ModuleNotFoundError: No module named 'examples.models'

@peterjc123 Could you please help us to convert below tiny english pytorch model to tflite model, Colab notebook would help me alot. https://openaipublic.azureedge.net/main/whisper/models/d3dd57d32accea0b295c96e26691aa14d8822fac7d9d27d5dc00b4ca2826dd03/tiny.en.pt

Thanks in Advance..

Hi, sorry to bother. This issue may not be directly related to tinynn. The scenario on my side is described as following:

1. I perform int8 QAT based on an official float32 model.
2. Only to fine tune some layers, I set requires_grad to False for all the other layers.
3. The training goes normally and the conversion to tflite is also done.
4. Use tf2onnx to dequantize the generated tflite and check the weights/bias. The weights/bias of the frozen layers are pretty different compared to the original model.

My target is to keep the weights/bias the same with the original model for the frozen layers after dequantization. Not sure if it's feasible. As you know, the official float32 model performs well for most cases and I just want to fine tune a few layers. Meanwhile, considering the inference speed on some edge devices, I have to perform QAT since the post-training quantization is really worse.

Thanks for your time.

I got a QAT int8 per-channel tflite model. To check the accuracy, I compare the inference results between it and the de-quantized onnx model.

1. python3.6 -m tf2onnx.convert --opset 11 --tflite test.tflite --output temp.onnx --dequantize python3.6 -m onnxsim temp.onnx test.onnx #test.onnx then holds float32 weights/bias
2. Run test.onnx and test.tflite respectively, compare the inference results based on the same input. There are big differences between the results. The onnx model has much better inference results.

I'm not pretty clear about the inference flow of a QAT tflite. Not sure such situation is normal or not. It's supposed to have similar results to that of onnx model.

Attach onnx and tflite model for reference. test.zip

Hi, my qat training goes well based on your sample code. But the conversion to tflite fails with the following error message:

RuntimeError: Quantized copy only works with contiguous Tensors

Based on the core trace, the error comes from: converter.convert()

My codes are almost the same with the sample code:

print("Start converting the model to TFLite") with torch.no_grad(): qat_model.eval() qat_model.cpu()

# The step below converts the model to an actual quantized model, which uses the quantized kernels.
qat_model = torch.quantization.convert(qat_model)                                                      

# When converting quantized models to TFLite, please ensure the quantization backend is QNNPACK.
torch.backends.quantized.engine = 'qnnpack'

# The code section below is used to convert the model to the TFLite format
converter = TFLiteConverter(qat_model, dummy_input, tflite_path='out/qat_model.tflite')
converer.convert()

By the way, I am curious how the trained qat model affect the conversion. Looks like the conversion could be done even without training. Not familiar with that, appreciate your help.Thanks.

There is a kind of quantization that quantizes model with dynamic fixed point int16. Is there any chance to support int16 quantization for QAT and mixed QAT?

Hi, this is really great work, thanks!

I am able to convert the LiteHRNet model to TFLite without running into any issues. However, the outputs don't match up.

Here is the output from sending ones through the network. Output is of shape [1,17,96,72]. I am just showing here output[0,0,0] from both pytorch and tflite:

When I convert to tflite via the onnx route, the outputs do match. So my guess is that some of the transpose/reshapes for NHWC is not happening correctly but I am not sure. Looking for some insight into what would be the best way to debug this?

Models: LiteHRNet pt trace LiteHRNet tiny tflite

非常棒的Repo，但我想使用convert.py将PyTorch训练好的3DCNN分类模型转换为.tflite模型，发生了错误：

问题1：x = F.avg_pool3d(x, x.data.size()[-3:])不支持

ERROR (tinynn.converter.base) Unsupported ops: aten::avg_pool3d

问题2：我将Temporal通道Squeeze掉了，但还是发生了错误：

Traceback (most recent call last):
  File "/home/leovin/Dynamic Gesture Detection/3D-CNNs/TinyNeuralNetwork/examples/converter/convert_for_3DCNNs.py", line 50, in <module>
    main_worker()
  File "/home/leovin/Dynamic Gesture Detection/3D-CNNs/TinyNeuralNetwork/examples/converter/convert_for_3DCNNs.py", line 46, in main_worker
    converter.convert()
  File "/home/leovin/anaconda3/envs/dgr/lib/python3.9/site-packages/TinyNeuralNetwork-0.1.20220310142810-py3.9.egg/tinynn/converter/base.py", line 347, in convert
    optimizer.optimize()
  File "/home/leovin/anaconda3/envs/dgr/lib/python3.9/site-packages/TinyNeuralNetwork-0.1.20220310142810-py3.9.egg/tinynn/converter/operators/optimize.py", line 1377, in optimize
    self.transform_graph()
  File "/home/leovin/anaconda3/envs/dgr/lib/python3.9/site-packages/TinyNeuralNetwork-0.1.20220310142810-py3.9.egg/tinynn/converter/operators/optimize.py", line 210, in transform_graph
    op.transform(self.graph, mapping)
  File "/home/leovin/anaconda3/envs/dgr/lib/python3.9/site-packages/TinyNeuralNetwork-0.1.20220310142810-py3.9.egg/tinynn/converter/operators/tflite/transformable.py", line 235, in transform
    assert False, "Only Conv[Transpose]1d/2d is supported"
AssertionError: Only Conv[Transpose]1d/2d is supported

Process finished with exit code 1

Only Conv[Transpose]1d/2d is supported请问是因为不支持Conv3d吗？

问题3：如果支持Conv3d，我该怎么做？

期待您的回答！

Thanks for your excellent work. It will definitely help when converting the model from Pytorch to TFLite.

When I try to convert the MoveNet (https://github.com/lee-man/movenet-pytorch) to TFLite using tinynn converter, the log reported some unsupported ops: argmax, expand and gather. Hope you can support these ops later:)

Hello author. I'm testing some encoder-decoder models here. After some experiments, it seems that implemented pruner is not working with interpolate layer.

from tinynn.prune import OneShotChannelPruner
import torch
from torch import Tensor
from torch import nn
from torch.nn import functional as F

class Interpolate(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return F.interpolate(x, (24, 24), mode='bilinear', align_corners=False)

class Conv2d_Interpolate(nn.Module):
    def __init__(self):
        super().__init__()

        self.conv2d = nn.Conv2d(3, 24, 3, padding = 1)
        self.interpolate = Interpolate()

    def forward(self, inTensor: Tensor):
        x = self.conv2d(inTensor)
        x = self.interpolate(x)
        return x

model = Conv2d_Interpolate()

dummy_input = torch.rand((1, 3, 192, 192))

_ = model(dummy_input)

pruner = OneShotChannelPruner(model, dummy_input, {"sparsity": {"default" : 0.25}, "metrics": "l2_norm"})

running above codes gives error below.

INFO (tinynn.graph.modifier) Start tracking tensor dimension changes...
ERROR (tinynn.graph.modifier) error modifier = interpolate_0_f, type = <class 'tinynn.graph.tracer.TraceFunction'>, kind = 
interpolate
...
RuntimeError: The size of tensor a (24) must match the size of tensor b (192) at non-singleton dimension 3

oneShotChannelPruner is not currently support interpolate layer?

Below are the PyTorch operators that are yet to be supported.

Unclassfied (New)

N/A

Primitives (Python operators)

• [x] aten::len

Very easy (Constant generation or aliasing)

• [x] aten::clamp_min
• [x] aten::clamp_max
• [x] aten::expand_as

Easy (Direct mapping)

Medium (Composite of multiple operators)

• [x] aten::im2col https://github.com/alibaba/TinyNeuralNetwork/issues/69
• [x] aten::col2im https://github.com/alibaba/TinyNeuralNetwork/issues/69
• [x] aten::mish
• [x] aten::group_norm
• [ ] torchvision::nms https://github.com/alibaba/TinyNeuralNetwork/issues/16

Hard (No mapping or the mapping is too complex)

• [ ] aten::grid_sample https://github.com/alibaba/TinyNeuralNetwork/issues/69
• [ ] quantized::instance_norm
• [ ] quantized::layer_norm

Sorry for bothering again, when I use QAT on yolov5, the conversion of the model will cause significant errors. I compare them layer by layer and find out it starts to cause errors from the very first layer and I have no idea how to fix it.

The generated script is here autoshape_qat.zip

device = "cpu"
dummy_input = torch.rand(1, 3, 640, 640)
model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True)

for k, m in model.named_modules():
    if isinstance(m, Detect):
        m.inplace = False
        m.onnx_dynamic = True
        m.export = True

quantizer = QATQuantizer(model, dummy_input, work_dir='out', config={'asymmetric': True, 'per_tensor': True})
qat_model = quantizer.quantize()
qat_model.to(device=device)

with torch.no_grad():  
    qat_model.cpu()
    qat_model = torch.quantization.convert(qat_model)
    torch.backends.quantized.engine = quantizer.backend
    convert_and_compare(qat_model, './output/qat_model.tflite', dummy_input)

My use case: Apply post training quantization to a pth model and convert to tflite. The generated tflite model fails to pass benchmark test with following error message: STARTING! Log parameter values verbosely: [0] Graph: [out/ptq_model.tflite] Loaded model out/ptq_model.tflite ERROR: tensorflow/lite/kernels/concatenation.cc:179 t->params.scale != output->params.scale (3 != -657359264) ERROR: Node number 154 (CONCATENATION) failed to prepare. Failed to allocate tensors! Benchmarking failed.

Pls refer to the attachment. Thanks. test.zip

Hi author，

For a certain model, oneshotpruner fails when there's channel padding with following error message:

ERROR (tinynn.graph.modifier) All node's sparsity in one subgraph must be the same

Please let me know if you need the model description file. Thanks.