inference discrepancy between onnxruntime and trt8.6(trt10.11 too)

[zzqiuzz]

Before submitting an issue, please make sure it hasn't been already addressed by searching through the existing and past issues.

Describe the bug

• I'd used tensort8.6.1 to convert an existing onnx model with qdq nodes, say a model exported using Model Optimizer Toolkit, to run on a orin-equiped platform. However, the inference result of the engine is not good as expected, it showed a large discrepancy compared with the original qdq onnx. For convenience, i again tried trt8.6.1 and trt10.11 on my local x86-64 workstation to convert the onnx into engine, to verify if i can see the same phenomenon. Unfortunately, yes. I calculated cosine similarity between qdq onnx and generated engine according to trt8.6 and trt10.11 respectively.

As depicted above, we can indeed see a large discrepancy.

Steps/Code to reproduce #bug

• convert command:
  trtexec --onnx=quant0206.onnx --saveEngine=quant_0206.engine --dumpProfile=true --best --verbose=true

Expected behavior

Who can help?

• I really appreciate if anyone can help solve the problem, or some instructive advice on how to solve it. I guess the problem may be caused by the conversion process from onnx to trt in which precision lost occurred in some operators.

System information

• Container used (if applicable): nvcr.io/nvidia/tensorrt-llm/release:1.0.0
• OS (e.g., Ubuntu 22.04, CentOS 7, Windows 10): ? Ubuntu 20.04
• CPU architecture (x86_64, aarch64): x86_64
• GPU name (e.g. H100, A100, L40S): GT3060
• GPU memory size: 12G
• Number of GPUs: 1
• Library versions (if applicable):
  • Python: 3.9
  • ModelOpt version or commit hash: 0.40.0
  • CUDA: nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2021 NVIDIA Corporation
    Built on Wed_Jun__2_19:15:15_PDT_2021
    Cuda compilation tools, release 11.4, V11.4.48
    Build cuda_11.4.r11.4/compiler.30033411_0
  • PyTorch: 2.0
  • Transformers: not used
  • TensorRT-LLM: not used
  • ONNXRuntime: 1.19.2-gpu
  • TensorRT: trt8.6 or trt10.11
• Any other details that may help:

[zzqiuzz]

quant0206.zip

[zzqiuzz]

@ajrasane hello, same issue mentioned @ NVIDIA/TensorRT#4708

[zzqiuzz]

I again used tensort 8.6.1.6 to quantize the model implicitly in the same quantization setting, say the same calibration data, calibration observer. The quantized engine outputs better results than that model opt does. Is this normal? any updates here? @yeyu-nvidia Thanks.

[yeyu-nvidia]

@ajrasane Can you please take a look?

[gcunhase]

@zzqiuzz thank you for bringing this to our attention.

I see that you provided the quantized model. Can you also please provide the following files?

1. Non-quantized ONNX model;
2. ORT inference script;
3. Cosine similarity script.

Thanks!

[zzqiuzz]

@gcunhase Thank you for your help! The zip file attatched below involved a non-quantized onnx model and an inference script including onnx inference, trt inference and result comparison using cosine similarity. You can refer it for more details.

for_issue.zip