result mismatch between qdq-onnx and tensorrt engine.

[zzqiuzz]

Description

i tried to use tensort8.6.1 to convert a onnx model with qdq nodes, however, the output of the converted engine dismatched with that of original onnx.

Environment

TensorRT Version:8.6.1 GA

NVIDIA GPU: Geforce 3060

NVIDIA Driver Version:Driver Version: 535.230.02

CUDA Version: nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

CUDNN Version: 8.6.0

Operating System: ubuntu20.04

Python Version (if applicable):3.9.23

Tensorflow Version (if applicable):

PyTorch Version (if applicable):

Baremetal or Container (if so, version):

Relevant Files

Steps To Reproduce

1. trtexec --onnx=quant_base.onnx --saveEngine=quant_base.plan --dumpProfile=true --int8 --fp16 --verbose=true > log.log 2>&1
   quant_base.onnx
2. python _infer.py
   i use randomly generated input to forward onnx and engine, compare their results by calculating cosine similarity.

[Saad-Mallebhari]

@zzqiuzz
QDQ output mismatch between ONNX Runtime and TensorRT is a known pain point and can have several causes , mixed precision fusion, QDQ folding differences, per-channel vs per-tensor scale handling, or QDQ importer limitations in TRT 8.6.

One common cause - mixed precision fusion

Your trtexec command uses both --int8 and --fp16, which allows TensorRT to fuse QDQ node pairs and selectively run layers in FP16 rather than INT8. This can shift quantization boundaries and accumulate numerical differences compared to the strict QDQ interpretation in ONNX Runtime.

Steps to isolate the cause:

1. Remove --fp16 first - run with --int8 only to see if the mismatch reduces:

trtexec --onnx=quant_base.onnx \
        --saveEngine=quant_base.plan \
        --int8 --verbose > log.log 2>&1

2. Add --dumpLayerInfo - this shows how QDQ nodes were fused during engine building, which helps confirm whether fusion is changing quantization boundaries:

trtexec --onnx=quant_base.onnx \
        --saveEngine=quant_base.plan \
        --int8 --dumpLayerInfo --verbose > log.log 2>&1

3. Use Polygraphy to compare layer by layer - pinpoints exactly where divergence starts:

polygraphy run quant_base.onnx \
    --onnxrt --trt \
    --int8 \
    --atol 1e-2 --rtol 1e-2 \
    --verbose

4. Check TRT version - TRT 8.6 has known QDQ handling differences compared to newer releases. Testing on TRT 10.x would help confirm whether this is version-specific behavior.

Could you share the cosine similarity values you're seeing? That would help determine whether this is a small numerical drift or a larger structural mismatch.

[zzqiuzz]

Thank you for your reply!
For Step 1: I didn't see any changes among cosine similarity of outputs when i removed -fp16;
For Step2: the flag ---dumpLayerInfo is from my point of view likey to show layer fusion in more details to see if any op fusion into int8 kernel is wrong. Thus, i visualize the engine by using Nvidia Trex from TensorRT package. The generated SVG file is as follows. And it seems see no wrongings from the visualization? log file is here.
log.log

For Step3: Yes, previously I also used polygraphy to try to identify intermediate layers from which originally the error occured. However, polygraphy couldn't successfully convert qdq onnx into trt engines with each op outputs tensor inserted .

For Step4: Immediately when i found the result mismatch, engine conversion using trt10.11 was conducted. It showed the same phenomenon as trt8.6.1.6. But i'll verify it again to ensure if it's trt version specific.