Question: Disable Optimizations for TensorRT

[YixuanSeanZhou]

Question

Because there are so many optimizations that TRT performs, sometimes it is very hard to isolate the issue if we see regression in model accuracy. I know we have the builder_optimization_level flag, but it seems to only control which kernel is used when executing the model.

I wonder if there is more fine-grained control? For example, I want to prevent fusions, or prevent removing dead code.

To give more context: In my specific use case, I am interested in isolating whether resolving Q/DQ nodes can causes regression in model. What I am interested to achieve is to only enable Q/DQ resolution and disable all other optimizations. Is this achievable?

Thanks in advance

[lix19937]

For example, I want to prevent fusions, or prevent removing dead code.

For your ref, try to use follow.

polygraphy run spec.onnx --trt   --best   --trt-outputs mark all 

[YixuanSeanZhou]

Hi @lix19937 thanks for your response!

So is this CLI option supposedly to skip all the optimizations TRT does?

Also, I think --best is not the correct option. I tried to use --int8, but i got this errer:

[E] 2: Assertion static_cast<size_t>(c) < mSet.size() failed. 
[E] 2: [cgraph.h::assertIsValidSubscript::161] Error Code 2: Internal Error (Assertion static_cast<size_t>(c) < mSet.size() failed. )

The corresponding onnx file was able to be built with TRT python API. I also have ran the polygraphy surgeon sanitize --fold-constants before building this onnx file

Thanks in advance!

[poweiw]

TRT will not fuse ops or remove dead nodes that produces graph outputs so marking them with --trt-outputs mark all will stop the fusion. For other questions let me get back with someone with better relevant knowledge 👍

[YixuanSeanZhou]

@poweiw Thank you! And is there a similar API we can leverage when running using the Python interface to compile TRT?

[poweiw]

You could try to use this code to mark all outputs to onnx first and then build the TRT network. This is also how polygraphy --mark-outputs does in the background.

import onnx
from polygraphy.backend.onnx import * 
from polygraphy import constants


model = onnx.load('model.onnx')
print(model.graph.output)
model = modify_outputs(model, outputs=constants.MARK_ALL)
print(model.graph.output)
onnx.save(model, 'out.onnx')