TensorRT v10.x some functions get worse

[lix19937]

The function of engine information visualization (such as memory layout) has regressed and is not as good as TensorRT v8. x

• v10.10 trex info

• v8.6 trex info

[lix19937]

@nvpohanh @zerollzeng

[nvpohanh]

could you share repro steps?

[lix19937]

@nvpohanh , There are some case:
1, The information of layerinfo from TensorRT 10.x is not enough like TensorRT 8.x
2, I found TensorRT 10.x has completely restructured the underlying logic of the --best option, resulting in its inability to consistently find the optimal PTQ inference configuration as in version TensorRT 8.x.
It may even lead to situations where the --best or --int8 --fp16 option is slower than pure --fp16. So, how to find a systematic solution for optimal PTQ configuration like TensorRT 8.x ?
Thanks.

[nvpohanh]

We are moving towards using Model Optimizer for quantization: https://github.com/NVIDIA/Model-Optimizer/tree/main/modelopt/onnx/quantization

You give ModelOpt the unquantized onnx with the calibration data, and then ModelOpt will calibration and insert the Q/DQ ops with optimal scales for preserve the accuracy, then the quantized ONNX can be loaded back into TensorRT in StronglyTyped mode to build the engine.

[lix19937]

@nvpohanh I have already use model-opt, like follow

from modelopt.onnx.quantization import quantize

quantize(
    onnx_path="mt_260206_no_plugin.float.onnx",
    quantize_mode="int8",  
    calibration_data="calib.npy",
    calibration_method="entropy",   # max, entropy, awq_clip, rtn_dq  
    output_path="mt_260206.quant.onnx",
)

trtexec --onnx=mt_260206.quant.onnx  \
--stronglyTyped                \
--verbose                      \
--dumpProfile                  \
--separateProfileRun           \
--saveEngine=$plan_file        \
--exportProfile=$profile_file  \
--exportLayerInfo=$json_file   \
--profilingVerbosity=detailed  \
--useCudaGraph                 \
--dumpLayerInfo                \
--builderOptimizationLevel=5   \
--tilingOptimizationLevel=3    \
--warmUp=500                   \
--duration=15                  \
--useSpinWait                  \
--noDataTransfers  

In this way , Can I get the fastest time of this model on thor-u ? I prioritize infer time.

[nvpohanh]

Could you share the onnx file so that we can measure the perf?