topic Is parallelism possible in OpenVINO? in Intel® Distribution of OpenVINO™ Toolkit

Is parallelism possible in OpenVINO?

jic5760 — Fri, 09 May 2025 08:09:58 GMT

Hello,

I want to run large LLM models using multiple Intel GPUs.

Is there a way to split a large model and run it on multiple GPUs?

Ref: https://huggingface.co/docs/transformers/v4.13.0/parallelism

Thank you

Re:Is parallelism possible in OpenVINO?

Zulkifli_Intel — Fri, 09 May 2025 18:25:59 GMT

Hi jic5760,

Thank you for reaching out.

To distribute inference across multiple GPUs, you can use the heterogeneous plugin in OpenVINO, which allows you to simultaneously leverage multiple inference devices (e.g., CPU, GPU, NPU) in one model. You can refer to pipeline-parallelism for multiple devices' execution in OpenVINO.

Regards,

Zul

Re:Is parallelism possible in OpenVINO?

Zulkifli_Intel — Tue, 20 May 2025 10:43:48 GMT

This thread will no longer be monitored since we have provided a solution. If you need any additional information from Intel, please submit a new question