[Bug] --loadInputs not working: input name mismatch when Flatten is the input node #1990
Comments
|
It looks like TRT has done some optimization around the Flatten op and the input tensor name has changed. Could you try the following?
|
|
@nvpohanh Sure, I tried the script and its output is |
|
The full output: and the script: import tensorrt as trt
logger = trt.Logger(trt.Logger.VERBOSE)
with open("output.engine", "rb") as f:
serialized_engine = f.read()
with trt.Runtime(logger) as runtime:
engine = runtime.deserialize_cuda_engine(serialized_engine)
print(engine.get_binding_name(0)) # print out the input tensor name |
|
yes, could you now try: |
|
@nvpohanh Yeah that works. I just realized that I may not be clear: actually the issue is just intended to report a bug and no urgency is needed (since both the unsqueeze one and your solution work as workarounds). Sorry if I didn't make it clear :-). |
lazycal
changed the title
lazycal
changed the title
|
I see. Thanks for the feedback. We can take a look at why the input tensor name is changed. One question: does the input tensor name change if the network contain more than one op? Such as a normal ONNX file containing the whole network? |
|
@nvpohanh Yes. import numpy as np
import onnx
from onnx import helper
from onnx import AttributeProto, TensorProto, GraphProto
import struct
N = 2
i0 = helper.make_tensor_value_info('i0', TensorProto.FLOAT, [N])
o0 = helper.make_tensor_value_info('o0', TensorProto.FLOAT, [1, N])
values = np.array([0], dtype=np.int64)
const = onnx.helper.make_node(
'Constant',
inputs=[],
outputs=['axes'],
value=onnx.helper.make_tensor(
name='const_tensor',
data_type=onnx.TensorProto.INT64,
dims=values.shape,
vals=values,
),
)
node = onnx.helper.make_node(
'Flatten',
inputs=['i0'],
outputs=['x'],
axis=0,
)
# node = helper.make_node(
# 'Unsqueeze',
# inputs=['i0', 'axes'],
# outputs=['o0'],
# )
relu_node = helper.make_node(
'Relu',
inputs=['x'],
outputs=['o0'],
)
graph_def = helper.make_graph(
[node, relu_node], # nodes
'test-model', # name
[i0], # inputs
[o0], # outputs
)
# Create the model (ModelProto)
model_def = helper.make_model(graph_def, producer_name='onnx-example')
print('The model is:\n{}'.format(model_def))
onnx.checker.check_model(model_def, full_check=True)
print('The model is checked!')
onnx.save(model_def, "output.onnx")
with open("id.bin", "wb") as f:
for i in range(N):
d = float(i)
f.write(struct.pack('<f', d))
from subprocess import check_call
check_call(
'trtexec --onnx=output.onnx --saveEngine=output.engine --buildOnly', shell=True)
import tensorrt as trt
logger = trt.Logger(trt.Logger.VERBOSE)
with open("output.engine", "rb") as f:
serialized_engine = f.read()
with trt.Runtime(logger) as runtime:
engine = runtime.deserialize_cuda_engine(serialized_engine)
print(engine.get_binding_name(0)) # print out the input tensor nameI also tried changing N or the rank of |
|
Internal bug tracking id: 3660435 We are investigating the issue. Will keep you updated. Thanks |
|
This has been fixed internally and will be available in next TRT release. Thanks for reporting this issue! |
|
Hi, which version of TensorRT will fix this? I met a similar problem in TensorRT 8.5.1.7 |
|
This was supposed to be fixed in TRT 8.5.1.7. @zhangtaoshan could you share your ONNX file? |
Description
I created a model with an input named
i0. However, when I tried to pass an input with--loadInputs=i0:id.bintotrtexecto run the model, I got the following error:Cannot find input tensor with name "i0" in the engine bindings! Please make sure the input tensor names are correct.Below is my code snippet to create the model and input:I tried adding more layers afterwards and the issue still exists, but after replacing the
Flattenwith other op will not trigger this error. For example, the code below replaces it withUnsqueezeand passed:P.S. PyTorch's onnx converter is buggy at handling
Flatten, so I have to make the ONNX with onnx helper manually, instead of from PyTorch, so if this is worth investigating, it might be a good idea to avoid building model from PyTorch.Environment
TensorRT Version: 8.4.0.6
NVIDIA GPU: RTX2080
NVIDIA Driver Version: 495.29.05
CUDA Version: 11.5
CUDNN Version: 8.3.0
Operating System: 18.04.1-Ubuntu
Python Version (if applicable): 3.8.10
Tensorflow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if so, version): Container built atop nvcr.io/nvidia/tensorrt 21.11-py3
Relevant Files
output.onnx 3.zip
Steps To Reproduce
Run
trtexec --onnx=output.onnx --loadInputs='i0':id.binThe full output log:
The text was updated successfully, but these errors were encountered: