Graphics Driver 32.0.101.8626 Issue with training AI Models

[LukeT]

I recently updated my Intel(R) Arc(TM) Graphics integrated 8 Core 9GB VRAM / (GB Shared) driver from 32.0.101.8509 to 32.0.101.8626. However after the update, I could no longer train my AI Models. I would receive the following error "RuntimeError: level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)". Once I rolled back to  32.0.101.8509 the training would succeed again. The Driver update, which broke my models training also included an update to Intel OneAPI too, so it may have been that.

[VivekS_Intel]

Hello LukeT

I hope you are doing well.

Thank you for posting your query on the Intel Community Forum!

We understand that updating to driver version 32.0.101.8626 and the new Intel oneAPI version has resulted in a UR_RESULT_ERROR_DEVICE_LOST error during AI model training, and that rolling back to 32.0.101.8509 currently serves as your workaround.

To help us better understand the root cause and investigate this potential regression, could you please provide the following details:

1. Training Framework: Which AI framework and version are you using (e.g., PyTorch with Intel Extension for PyTorch, TensorFlow, etc.)?
2. System Memory & Allocation: Since the integrated GPU shares system RAM, have you noticed if the error occurs specifically when VRAM/System memory usage reaches a certain threshold?
3. oneAPI Components: Could you confirm the specific version of the Intel oneAPI Base Toolkit you updated to? 
4. Model Specifics: Does this error occur with all models you attempt to train, or is it isolated to a specific architecture or large batch size?

Additionally, to help us review your hardware and driver configuration, please provide an SSU (System Support Utility) log file. When generating the report, please ensure that the Network tab is unchecked before clicking scan. You can find the tool and a guide on how to use it here: How to get the Intel® System Support Utility Logs.

This information will be very helpful for us to replicate the environment and identify if the issue stems from the driver or the updated oneAPI libraries.

Best regards,

VivekS

Intel Customer Support Technician

[VivekS_Intel]

Hello LukeT,

I hope you’re doing well.

I’m checking in to see if you’ve had a moment to review my previous message regarding the DEVICE_LOST error you encountered. I know how disruptive it is to have your workflow interrupted by a driver update, and I want to make sure we get this sorted out for you.

When you have a chance, could you please provide the SSU log and the framework details I mentioned? This information is essential for us to replicate the issue and find a permanent fix so you aren't stuck on an older driver version.

Best regards,

VivekS

Intel Customer Support Technician 

[LukeT]

Hi VivekS,

I haven't had chance to get back on this yet, when I next encounter an issue with a driver update I'll check-back in here.

[VivekS_Intel]

Hello LukeT,

Thank you for the update. I completely understand that you're sticking with the stable version for now to maintain your workflow.

Whenever you do decide to test a newer release, I wanted to let you know that a new driver is now available: Version 32.0.101.8724 (Latest).

If you decide to give this version a try, I recommend the following steps to ensure the best results:

• Clean Installation: You can download the latest driver here. Performing a "Clean Install" during the setup process is highly recommended to clear out any potential conflicts from previous versions.
• System Optimization: We also suggest checking your motherboard manufacturer’s website for any BIOS updates and using the Intel® Driver & Support Assistant (IDSA) to keep the rest of your system drivers current.

We will keep this thread open for now. If you encounter the same error on this latest version, please let us know and share the SSU log. When generating the report, please ensure that the Network tab is unchecked before clicking scan. This will help us investigate further.

Best regards,

VivekS

Intel Customer Support Technician  

[Chawan_Intel]

Hello LukeT,

I hope you are doing well.

We are following up on our previous message regarding the latest available graphics driver version 32.0.101.8724 and the recommended troubleshooting steps shared earlier.

Kindly let us know if you have had an opportunity to test the latest driver version using the clean installation method, and whether the issue still persists after the update.

If the behavior is still being observed, please share the latest Intel® System Support Utility for Windows* for further analysis. When generating the report, kindly ensure that the Network tab is unchecked before scanning.

This will help us continue the investigation and assist you further with the next steps.

We look forward to your response.

Best regards,

Chawan

Intel Customer Support Technician  

[Chawan_Intel]

Hello LukeT,

I hope you are doing well.

I wanted to follow up regarding our previous communication about the graphics driver version 32.0.101.8724 and the recommended troubleshooting steps.

Could you please let us know if you had the opportunity to test the latest driver using the clean installation method, and whether the issue still persists after the update?

If the issue is still observed, kindly share the latest Intel® System Support Utility for Windows* report for further analysis. When generating the report, please ensure that the Network tab is unchecked before scanning.

This will help us continue the investigation and assist you with the next steps.

Thank you for your time and cooperation. I look forward to your response.

Best regards,

Chawan

Intel Customer Support Technician  

[Chawan_Intel]

Hi LukeT, 

I hope you are doing well.

I wanted to follow up again regarding our previous communication about the graphics driver version 32.0.101.8724 and the recommended troubleshooting steps, as we have not yet received a response or update from your end after multiple follow-ups.

As there has been no further update, we will proceed with closing this thread for now. However, please note that you can always reach out to us by opening a new thread, and we will be happy to assist you further.

If you still require assistance on this issue, feel free to reopen or create a new case at your convenience.

Thank you for your understanding.

Best regards,

Chawan

Intel Customer Support Technician