DirectML kernel creating is very slow compared to other platforms

[jacobgorm]

Hi,

I am building a product using a couple of CNNs of the MobileNets family, which use a GEMM op as the main workhorse operation.

I've shipped this tech on IOS and MacOS for years, using their MPS kernels, have alwasy been able to start my app instantly, as is a hard requirement especially on Mobile.

However, on DirectML on Intel, with a cold shader cache, I am seeing extremely long compilation times on first install or after a graphics driver upgrade. Typically over 600ms per kernel in a release build, and up to 16,000ms(!) when running under Windbg. A MobileNet needs a pretty varied collection of kernels, so this all adds up to very long first-start times for my app.

I've filed a bug about this with Microsoft two years ago, and they confirmed that this problem exists and is specific to Intel (NVIDIA is typically well under 5ms per kernel, and AMD is plenty fast too) and promised to follow up with Intel. Yet nothing has happened. See the discussion and benchmark results here: https://github.com/microsoft/DirectML/issues/212

There is no way I can ship a product that takes minutes to start up the first time or after a driver update, rendering DirectML useless in practice, and forcing me to write and use a slower fallback kernel on Intel, or recommend that my users upgrade to AMD or NVIDIA GPUs that have better DirectML support.

Could somebody help?

Best regards,

Jacob Gorm Hansen

[Vasudha_Intel]

Thank you for posting your question, can you provide details of the CPU (model name and generation).

[jacobgorm]

Yes: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz 2.42 GHz

Best,
Jacob

[VonM_Intel]

Hi, jacobgorm.

Thank you for posting in the Intel Communities. Let me help you with that.

However, we will need to do further research/checking and review on this matter and post the response on this thread once available.

Best regards,

Von M.

Intel Customer Support Technician

[VonM_Intel]

Hi, jacobgorm,

Thank you for waiting. To better understand and address your concern, we would like to ask for the following details: 

• What are the steps that you have taken to troubleshoot the issue?
• When did the issue first occur?
• Was this working fine before?
• Were there any recent changes to the system or the software?
• How did you install the latest graphics driver for the computer?

Moreover, kindly provide us with your Intel SSU, as it would really help us if we could get more information about your system.

Kindly do the following: 

1. Download the Intel® System Support Utility and save the application to your system. 

https://www.intel.com/content/www/us/en/download/18377/intel-system-support-utility-for-windows.html 

2. Open the application, tick Everything, and click Scan to see system and device information. The Intel® System Support Utility defaults to the Summary View on the output screen following the scan. Click the menu where it says summary to change to Detailed View. 

3. To save your scan, click Next and click Save. You can save the file to any accessible location on your computer. 

4. Attach the text file here.

Intel® System Support Utility for Windows*

This download is for version 3.0.0.2 of the Intel® System Support Utility (Intel® SSU) for Windows*.

Best Regards,

Von M.

Intel Customer Support Technician

[VonM_Intel]

Hi, jacobgorm.

We hope you are doing fine.

Were you able to check the previous post?

Kindly let us know if you still need assistance regarding our questions.

Best regards,

Von M.

Intel Customer Support Technician

[jacobgorm · Accepted solution]

Hi Von M,

first all the detail you asked about:

Hi,

I've gotten independent confirmation on this issue from the DirectML developers at Microsoft, who also tested against NVIDIA's driver and found that the slowness occurs at their Intel systems too, but not in their NVIDIA systems. See the Github issue linked above for the full details.

The issue first occurred in early 2022 when I ported my code to DirectML.

It has never worked fine on any Intel system, including those of my customers.

I've updated the drivers by both by downloading directly from Intel and via the ARC GUI and from the Intel Driver & Support Assistant.

These are the details about my machine from the Intel Driver & Support Assistant:

LENOVO 20W0002UMX
BIOS
Version
N34ET59W (1.59 )
Date
11/7/2023
Motherboard
Manufacturer
LENOVO
Model
20W0002UMX
Version
SDK0J40697 WIN
Operating System
Edition
Microsoft Windows 11 Pro (64-bit)
Version (Build)
23H2 (10.0.22631)
Devices and Drivers
Processor
11th Gen Intel® Core™ i5-1135G7 @ 2.40GHz
Graphics
Intel® Iris® Xe Graphics
Audio
Realtek High Definition Audio(SST)
Intel® Smart Sound teknologi til digitale mikrofoner
Intel® Smart Sound teknologi til USB audio
USB Audio Device
Networking and I/O
Intel® Ethernet Connection (13) I219-V
Intel® Wi-Fi 6E AX210 160MHz
Bluetooth Device (Personal Area Network)
Thunderbolt Controller 1
Thunderbolt Controller 2
Intel® Wireless Bluetooth®
Memory
16 GB
Storage
KXG50ZNV512G NVMe TOSHIBA 512GB

However, the good news is that after updating to version 31.0.101.5186 of the Intel drivers that were released last week, the issue seems to finally have been resolved!

I am going to leave this bug open for a few more days, in case it reappears.

Thank you

Jacob

[VonM_Intel]

Hi, jacobgorm.

Thank you for providing these details.

I acknowledged that and we will keep on monitoring the DirectML kernel issues as we are constantly striving to improve in making better products and services.

Best regards,

Von M.

Intel Customer Support Technician

[VonM_Intel]

Hi, jacobgorm.

We are glad to know you found a solution, thank you for taking the time to share this fix, and hopefully, it may help other community members experiencing similar behavior. Since the thread is now solved, we will proceed to close it. Please keep in mind that this thread will no longer be monitored by Intel.

Best regards,

Von M.

Intel Customer Support Technician