ONNX Runtime

ONNX Runtime is a cross-platform inference runtime that optimizes and executes ONNX models. It is not a kernel compiler: it does not generate new GPU kernels, but instead applies graph-level optimizations (constant folding, node elimination, operator fusion) and partitions the graph across execution providers. The execution provider framework is its key strength: the same model can run on CUDA, TensorRT, OpenVINO, DirectML, or CPU by simply changing the provider configuration. For production deployment, ONNX Runtime is most valuable when cross-platform support is required (e.g., Windows + Linux + mobile). For pure GPU inference on NVIDIA hardware, TensorRT or Triton-based engines typically outperform ONNX Runtime. The graph optimizations are effective but limited compared to what Triton or Inductor can achieve: ONNX Runtime can fuse GEMM+Activation or Attention, but cannot fuse arbitrary sequences of operations into a single kernel.