Torch Compile 高级用法

此交互式脚本旨在概述 torch_tensorrt.compile(…, ir=”torch_compile”, …) 的工作流程，以及它如何与 torch.compile API 集成。

导入和模型定义

import torch
import torch_tensorrt

# We begin by defining a model
class Model(torch.nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.relu = torch.nn.ReLU()

    def forward(self, x: torch.Tensor, y: torch.Tensor):
        x_out = self.relu(x)
        y_out = self.relu(y)
        x_y_out = x_out + y_out
        return torch.mean(x_y_out)

使用默认设置使用 torch.compile 进行编译

# Define sample float inputs and initialize model
sample_inputs = [torch.rand((5, 7)).cuda(), torch.rand((5, 7)).cuda()]
model = Model().eval().cuda()

# Next, we compile the model using torch.compile
# For the default settings, we can simply call torch.compile
# with the backend "torch_tensorrt", and run the model on an
# input to cause compilation, as so:
optimized_model = torch.compile(model, backend="torch_tensorrt", dynamic=False)
optimized_model(*sample_inputs)

使用自定义设置使用 torch.compile 进行编译

# First, we use Torch utilities to clean up the workspace
# after the previous compile invocation
torch._dynamo.reset()

# Define sample half inputs and initialize model
sample_inputs_half = [
    torch.rand((5, 7)).half().cuda(),
    torch.rand((5, 7)).half().cuda(),
]
model_half = Model().eval().cuda()

# If we want to customize certain options in the backend,
# but still use the torch.compile call directly, we can provide
# custom options to the backend via the "options" keyword
# which takes in a dictionary mapping options to values.
#
# For accepted backend options, see the CompilationSettings dataclass:
# py/torch_tensorrt/dynamo/_settings.py
backend_kwargs = {
    "enabled_precisions": {torch.half},
    "debug": True,
    "min_block_size": 2,
    "torch_executed_ops": {"torch.ops.aten.sub.Tensor"},
    "optimization_level": 4,
    "use_python_runtime": False,
}

# Run the model on an input to cause compilation, as so:
optimized_model_custom = torch.compile(
    model_half,
    backend="torch_tensorrt",
    options=backend_kwargs,
    dynamic=False,
)
optimized_model_custom(*sample_inputs_half)

清理

# Finally, we use Torch utilities to clean up the workspace
torch._dynamo.reset()

Cuda 驱动程序错误说明

有时，在使用 torch_tensorrt 完成 Dynamo 编译后退出 Python 运行时时，可能会遇到 Cuda 驱动程序错误。此问题与 https://github.com/NVIDIA/TensorRT/issues/2052 相关，可以通过将编译/推理包装在函数中并使用范围限定调用来解决，如下所示

if __name__ == '__main__':
    compile_engine_and_infer()