topic Re: Multi GPU clCreateBuffer failure on single context in GPU Compute Software

Multi GPU clCreateBuffer failure on single context

shekar — Thu, 04 May 2023 02:38:20 GMT

I have two Intel ARC A770 GPUs and I am seeing this behavior (test code attached).

1. I am trying to allocate memory using clCreateBuffer (1 MB each).

2. I have two GPU devices.

3. The test code creates a context, loads a kernel program, builds it.

4. The test program creates only one context (and can do that for 1 device or 2 devices).

5. When I run the code in 1 GPU mode, I am able to allocate memory (over 10000 allocations). But when I run the code in 2 GPU mode, I run out of memory (out of host memory after 1000 allocations -- around 1 GB memory). OpenCL error code -6.

The GPUs have 16 GB RAM and my host has 192 GB RAM.

Any idea what I am doing wrong?

The command to build the code is:

gcc -D CL_TARGET_OPENCL_VERSION=220 -g -Wall -o OpenCLMulti OpenCLMulti.cpp -lOpenCL -lm

Re: Multi GPU clCreateBuffer failure on single context

KonstantyMisiak — Fri, 05 May 2023 10:26:38 GMT

Hi, I'm looking into your issue.
Could you provide more details about your setup, such as OS distribution, kernel version, OneAPI/OpenCL driver version?

Re: Multi GPU clCreateBuffer failure on single context

shekar — Fri, 05 May 2023 10:35:16 GMT

Hi @KonstantyMisiak

Here is the output of "uname -a"

Linux gordian-2 5.19.0-35-generic #36~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb 17 15:17:25 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

I am attaching the output of clinfo.

Also, another interesting thing is that once in a while, Xorg will just crash and put me back in the login screen. This started happening after I installed the two A770 GPUs (I used to have a single A750 and everything was working okay -- though frankly it was an ordeal to get that up and running).

Anyway, thanks for looking into this issue. Appreciate your help.

Re: Multi GPU clCreateBuffer failure on single context

shekar — Mon, 08 May 2023 13:17:06 GMT

@KonstantyMisiak

Do you have everything you need from me to look into the issue? Thanks.

Re: Multi GPU clCreateBuffer failure on single context

KonstantyMisiak — Mon, 08 May 2023 13:56:11 GMT

@shekar yes, this issue is internally tracked and me and my team are looking into it.

Re: Multi GPU clCreateBuffer failure on single context

shekar — Mon, 15 May 2023 15:24:26 GMT

@KonstantyMisiak Any update on this issue? Thanks.

Re: Multi GPU clCreateBuffer failure on single context

KonstantyMisiak — Mon, 22 May 2023 11:55:44 GMT

@shekar We couldn't reproduce your issue on our standard setup. We are working on recreating the closest setup to yours and reproducing this issue.

Re: Multi GPU clCreateBuffer failure on single context

shekar — Mon, 22 May 2023 12:43:10 GMT

@KonstantyMisiak What is the standard setup you have? Thanks.

Re: Multi GPU clCreateBuffer failure on single context

shekar — Tue, 30 May 2023 18:41:30 GMT

@KonstantyMisiak Can you tell me the standard setup? This issue has become a huge blocker for me and I really need to get this done. Thanks.

Re: Multi GPU clCreateBuffer failure on single context

KonstantyMisiak — Wed, 31 May 2023 10:04:22 GMT

The biggest difference seems to be that our standard setup uses linux kernel with internal patches which can affect the issue

Re: Multi GPU clCreateBuffer failure on single context

shekar — Wed, 31 May 2023 10:12:43 GMT

@KonstantyMisiak You mean kernel patches that only you have access to or something that I can download too? Please provide more clarity on this. If you think this is a "bug" in the driver, and will require an upgrade, can you provide some information on when such an upgrade will be available? Thanks.

Re: Multi GPU clCreateBuffer failure on single context

KonstantyMisiak — Wed, 31 May 2023 10:16:48 GMT

Yes, these patches are internal.
I have to kindly ask for your patience while we are trying to reproduce your issue. If we establish that this is the bug in the driver we will need to pinpoint which component is at fault so we can direct it to the proper subteam.

Re: Multi GPU clCreateBuffer failure on single context

shekar — Mon, 12 Jun 2023 04:09:50 GMT

@KonstantyMisiak Can you give me some idea as to when you are going to look at my issue? Just to be clear, I installed the exact driver and Linux kernel recommended in your documentation. So, I am surprised to hear that you don't have a machine lying around with that same configuration. If you don't think this issue can be solved, please say so and I will make contingency plans (like upgrading to NVIDIA GPUs). Though, I would prefer to not change my hardware.

Re: Multi GPU clCreateBuffer failure on single context

MaciejPlewka — Mon, 19 Jun 2023 10:17:06 GMT

Hi @shekar

I was able to reproduce your issue with OpenCL driver in version 23.05.25593.18 .
When I updated driver to 23.13.26032.26 version looks like issue is gone.

Can you please test it with latest driver?

Re: Multi GPU clCreateBuffer failure on single context

shekar — Mon, 19 Jun 2023 13:45:42 GMT

@MaciejPlewkaI upgraded the driver and now clinfo doesn't recognize the platform. I reinstalled Ubuntu 22.04 and while following the instructions given here: https://dgpu-docs.intel.com/installation-guides/ubuntu/ubuntu-jammy-arc.html, I am getting this error:

sudo apt-get update && sudo apt-get install -y --install-suggests linux-image-5.19.0-35-generic
Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:5 https://repositories.intel.com/graphics/ubuntu jammy InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-image-5.19.0-35-generic
E: Couldn't find any package by glob 'linux-image-5.19.0-35-generic'
E: Couldn't find any package by regex 'linux-image-5.19.0-35-generic'

What do I do now?

Re: Multi GPU clCreateBuffer failure on single context

MaciejPlewka — Tue, 20 Jun 2023 06:39:46 GMT

I was also using this instruction and met the same problem. There is no linux-image-5.19.0-35-generic in added repositories, but I found that linux-image-5.19.0-37 should be there.