GPU works with 2021.3 on Ubuntu20.04 but not Ubuntu 22.04 and 2024.3

[wb666greene]

I have a couple of N4020 Celeron with Mesa UHD 600 graphics (GLK2) systems running Ubuntu 20.04 and OpenVINO 2021.3 that run Darknet Yolo4 on the iGPU.  I updated one of them to Ubuntu 22.04 and pip installed openvino 2024.3 in a VENV virtual environment to run Ultralytics Yolo8 on the iGPU.  But running code that works fine on other systems I get this error:

[INFO] Using OpenVINO: 2024.2.0-15519-5c0f38f83f6-releases/2024/2
Available devices:
   CPU: Intel(R) Celeron(R) N4020 CPU @ 1.10GHz
   GPU: Intel(R) UHD Graphics 600 [0x3185] (iGPU)
Traceback (most recent call last):
  File "/home/ai/LearnTestOV/yolo8ov.py", line 45, in <module>
    det_compiled_model = core.compile_model(det_ov_model, "GPU", ov_config)
  File "/home/ai/xyz/lib/python3.10/site-packages/openvino/runtime/ie_api.py", line 543, in compile_model
    super().compile_model(model, device_name, {} if config is None else config),
RuntimeError: Exception from src/inference/src/cpp/core.cpp:104:
Exception from src/inference/src/dev/plugin.cpp:54:
Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:185:
[GPU] ProgramBuilder build failed!
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_stream.cpp:429:
[GPU] clWaitForEvents, error code: -14

Note: I saved this error message from 2024.2, but upgrading to 2024.3 didn't change anything, so I'm posting this question now since installing all these "neo" driver packages and rebooting didn't change anything either:

https://github.com/intel/compute-runtime/releases/tag/24.31.30508.7 

Edit:  Should mention that running kernel:

6.8.0-40-generic #40~22.04.3-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 30 17:30:19 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

[Witold_Intel]

Hi Wally,

I was able to discuss your case with a graphics driver expert. We would require two more files for further diagnosis.

1. Kernel logs, normally located under /var/log/kern.log . Please note that those are typically retained for four weeks, so you may need to reproduce the error once more.
2. Disk image for reproduction. This can be obtained by running "dd" command ( sudo dd if=/dev/sdX of=/path/to/backup.img bs=64K status=progress ) or proprietary software like Clonezilla. 

Would you be willing and able to provide those for us?

[wb666greene]

I will try.  The logs should be intact since I shut the system down after the last items I uploaded.  I have no trouble doing the dd command to clone a system, just it is a pretty large SSD, let me know where to upload the image, I will try to have it all for you on Monday.

[Witold_Intel]

That's great. The portal normally allows large attachments. If there's a problem to upload, please let me know, I will look for a solution.

[Witold_Intel]

Hi Wally,

Were you able to generate the files?

[wb666greene]

Here is the kernel log, I booted up and ran my test code.  Not much in it.  My root file-system is like 40GB with 25GB used but I don't think it will be bootable without the /boot partition as well (which is another 536 MB.  Where is this "portal" where I can upload a 40GB image?  With more effort I can shrink the partition with gparted and get it to about 26 GB.

I need some details about uploading such a large image.

[Witold_Intel]

Many thanks for the kernel log, I passed it to graphics debug server team. I will also ask my colleague about the most convenient channel to upload the image. By portal I meant the customer portal but it seems it is not applicable to OpenVino customers, sorry.

[Witold_Intel]

Hello Wally,

My colleague from graphics server debug team has checked the kernel log. Apparently, there is a GPU reset happening.

Please note that OpenVINO 2021.3 is supported by below OSes:

Supported Operating System

Python* Version (64-bit)

Ubuntu* 18.04 long-term support (LTS), 64-bit

3.6, 3.7

Ubuntu* 20.04 long-term support (LTS), 64-bit

3.6, 3.7

https://pypi.org/project/openvino-dev/2021.3.0/#:~:text=NOTE:%20This%20package%20can%20be,versions%20above%20are%20fully%20validated.

At this time it can be either a driver issue or APP hanging your GPU because of incompatibility behaviors. I will consult my colleague who is an expert on OpenVino github cases if we still require a system image from you.

[Witold_Intel]

I was in touch with my colleague from Graphics Debug team and we concluded that a partition image is not necessary for this issue.

Has the reference to supported versions helped you? Do you require further support?

[Witold_Intel]

Hi Wally,

we haven't heard from you for a while. Maybe you could check the GPU graphics driver version used on both systems again (working on Ubuntu 20.04 and not working on Ubuntu 22.04)?

The commands to run and gather this info are here:

export gfx_version=$(dpkg-query --showformat='${Version}' --show intel-opencl-icd)

echo $gfx_version

Other than that, can we support you further?

[wb666greene]

Sorry I was away for Thanksgiving.  Curious the 20.04 system running OpenVINO 2021.3 that uses the GPU for yolo4 returns this for your gfx_version probe:

ai@HB2a:~$ export gfx_version=$(dpkg-query --showformat='${Version}' --show intel-opencl-icd)
dpkg-query: no packages found matching intel-opencl-icd

Running kernel:

ai@HB2a:~$ uname -a
Linux HB2a 5.15.0-117-generic #127~20.04.1-Ubuntu SMP Thu Jul 11 15:36:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

I never had any issues setting up the 20.04 system and yolo4 following these instructions with 2021.3 OpenVINO:

https://github.com/TNTWEN/OpenVINO-YOLOV4 

I'm at a loss here.  This was to be a backup to the 20.04 system and was tested and held in reserve, later I installed Ubuntu 22.04 on it to upgrade to OpenVINO 2024.3 and use yolo8 on the GPU.  I run MobilenetSSD_v2 on an M.2 Coral TPU and when a person is detected I "zoom in" on the detection box and pass it to the yolo model, this cascade gets the false positive rate very close to zero, with an increase in the false negative rate of ~3-5%, which is insignificant if the frame rate is decent.  

I have a workaround using a USB Coral to run the yolo8 model, but this is a significant increase to the cost of the system.

I may install Ubuntu 24.04 and the latest OpenVINO (2024.5, correct? assuming it supports 24.04) and try again. 

But I've since learned that refurbished I3 or i5 "business class" laptops (easily found on-line for ~$150) end up the most economical as I can run the MobilenetSSD on the CPU and the yolo8 on the GPU.

So at this point if you have no further suggestions, we can consider this closed.

[Witold_Intel]

Hi Wally,

were you able to see my previous message? Thank you.

[Witold_Intel]

Hi Wally,

have you been able to see my previous comment with driver check-up command? Thank you.

[Witold_Intel]

Thank you for your patience. Indeed, Ubuntu 24.04 long-term support (LTS), 64-bit (Kernel 6.8+) is supported by OpenVINO 2024.5 in preview.

More information: https://docs.openvino.ai/2024/about-openvino/release-notes-openvino/system-requirements.html

I'm glad to hear that you found a workaround. In this case I will deescalate this issue and pass your feedback to the graphics driver expert that investigated the case for future reference.

[Aznie_Intel]

Hi Wally,

This thread will no longer be monitored since this issue has been resolved. If you need any additional information from Intel, please submit a new question. 

Regards,

Aznie