Hello Intel Gaudi Team,
I am currently working with Intel Gaudi accelerators and attempting to integrate them with vLLM, a popular open-source LLM inference engine.
During my integration, I have identified a compatibility gap between the Gaudi supported PyTorch version and the requirements of vLLM:
Environment
Gaudi SW stack version: 25.0.1
PyTorch version (Gaudi official support): 2.6.0 (per Support Matrix)
vLLM versions tested: 0.8.x (works), 0.9.0 and above (fails)
Issue
Starting from vLLM 0.9.0, the project requires PyTorch 2.7.x.
This was introduced after PR #18056, which removed legacy typing.List usage in favor of Python 3.9+ native list[...] annotations.
In PyTorch 2.6.0, some functions (e.g., infer_schema) still expect List[Tensor].
With vLLM ≥0.9.0 (built against torch 2.7.x), only list[Tensor] signatures exist.
This mismatch causes schema inference errors and prevents vLLM from running on Gaudi with PyTorch 2.6.0.
Impact
Users on Gaudi cannot run the latest vLLM (0.9.x and newer).
Only older versions (≤0.8.x) or Habana’s internal fork are usable, which prevents access to important features and fixes from the mainline vLLM project.
Request
Could you please share if there is a roadmap or plan for Gaudi to support PyTorch 2.7.x?
If not yet planned, would it be possible to provide a temporary patch/workaround for PyTorch 2.6.0 compatibility with vLLM ≥0.9.0?
Any guidance for the community on the recommended path forward would be very helpful.
Thank you for your support.
