Skip to content

failure of TensorRT 10.15 when running utils/process_engine.py on GPU A30 #4724

@LinGeLin

Description

@LinGeLin

Description

When I attempted to run this script tools/experimental/trt-engine-explorer/utils/process_engine.py for model performance analysis, an error occurred.

python utils/process_engine.py model.onnx ./output_dir fp16

error msg:

Profiling metadata: generated output file ./output_dir_opset13/model.onnx.engine.profile.metadata.json
Generating graph diagram: ./output_dir_opset13/model.onnx.engine.graph.json
Error: model.onnx.engine.graph.json.gv: syntax error in line 432 scanning a quoted string (missing endquote? longer than 16384?)
String starting:"LayerType:kgen\nMetadata:[ONNX Layer: ReduceSumSquare__2367]+[ONNX Layer: Div__2
Traceback (most recent call last):
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/env_trex/lib/python3.12/site-packages/graphviz/backend/execute.py", line 88, in run_check
    proc.check_returncode()
  File "/usr/lib/python3.12/subprocess.py", line 502, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '[PosixPath('dot'), '-Kdot', '-Tsvg', '-o', 'model.onnx.engine.graph.json.svg', 'model.onnx.engine.graph.json.gv']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/utils/process_engine.py", line 351, in <module>
    do_work(args)
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/utils/process_engine.py", line 345, in do_work
    process_engine(args, build, profile, draw)
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/utils/process_engine.py", line 279, in process_engine
    success = generate_engine_svg(args, add_suffix=build)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/utils/process_engine.py", line 250, in generate_engine_svg
    draw_engine(graph_json_fname, profiling_json_fname)
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/utils/draw_engine.py", line 179, in draw_engine
    trex.graphing.render_dot(graph, engine_json_fname, output_format)
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/trex/graphing.py", line 409, in render_dot
    dot_graph.render(outfile=output_fname, view=False, overwrite_source=False)
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/env_trex/lib/python3.12/site-packages/graphviz/_tools.py", line 185, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/env_trex/lib/python3.12/site-packages/graphviz/rendering.py", line 122, in render
    rendered = self._render(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/env_trex/lib/python3.12/site-packages/graphviz/_tools.py", line 185, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/env_trex/lib/python3.12/site-packages/graphviz/backend/rendering.py", line 326, in render
    execute.run_check(cmd,
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/env_trex/lib/python3.12/site-packages/graphviz/backend/execute.py", line 90, in run_check
    raise CalledProcessError(*e.args)
graphviz.backend.execute.CalledProcessError: Command '[PosixPath('dot'), '-Kdot', '-Tsvg', '-o', 'model.onnx.engine.graph.json.svg', 'model.onnx.engine.graph.json.gv']' returned non-zero exit status 1. [stderr: b'Error: model.onnx.engine.graph.json.gv: syntax error in line 432 scanning a quoted string (missing endquote? longer than 16384?)\nString starting:"LayerType:kgen\\nMetadata:[ONNX Layer: ReduceSumSquare__2367]+[ONNX Layer: Div__2\n']

Environment

TensorRT Version :10.15:

NVIDIA GPU : A30:

NVIDIA Driver Version : 550.144.03:

CUDA Version : 13.1:

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions