torch.compile / TorchInductor

torch.compile is PyTorch's recommended path for improving model performance. It consists of TorchDynamo (a Python bytecode tracer that captures the computation graph) and TorchInductor (a compiler that generates optimized code, primarily Triton kernels for GPU). The main failure modes are: (1) Graph breaks: when Dynamo encounters an operation it cannot trace (e.g., data-dependent control flow, unsupported Python constructs), it breaks the graph and falls back to eager mode for that segment. This can cause significant performance degradation. (2) Recompilation storms: Dynamo specializes on Python values (e.g., integer arguments), so each new value triggers recompilation. For inference with varying batch sizes, this can lead to dozens of recompilations. The recompile_limit (default 8) and accumulated_recompile_limit (default 256) control when compilation fails. (3) Dynamic shapes: by default, Dynamo specializes on tensor shapes, so each new shape triggers recompilation. The automatic_dynamic_shapes config can help, but it is not enabled by default. For production inference, torch.compile is most effective when the workload has fixed shapes (e.g., fixed batch size, fixed sequence length). For variable-length workloads, the recompilation overhead can dominate.