benchmark_app fails to run on specified OS with parameter "-infer_precision CPU:bf16"

[ClinB]

Hi all:

Sorry for asking so many questions about benchmark_app failure.

benchmark_app fails when running with "-infer_precision CPU:bf16"

benchmark_app fails on WindowsServer2016/2019 with 4th Xeon CPU 

I have designed several experiments with advice from community, but all results were failed.

My environments:

Env1:

• Amazon Machine Image: ami-065b889ab5c33720e  
• OS: Windows Server 2019, version 1809, OS build 17763.4737
• CPU: Intel Xeon Platinum 8488C (4th Xeon), 1 core, 2 threads
• Python: 3.8, from official website
• OpenVINO: 2022.3
  • pip install openvino==2022.3
  • pip install openvino-dev==2022.3
•  Model:  asl-recognition-0004 (FP16)
  • omz_downloader --name asl-recognition-0004 --precisions FP16 --output_dir myLocalPath

Env2:

• Basically, same as Env1
• Model: asl-recognition-0004 (FP32)

Env3:

• Basically, same as Env1
• OpenVINO: 2023.0

Env4:

• Basically, same as Env2
• OpenVINO: 2023.0

Env5:

• Amazon Machine Image: ami-09301a37d119fe4c5 
• OS: Windows Server 2022, version 21H2, OS build 20348.1906
• Others same as Env1

Env6:

• Basically, same as Env5
• Model: asl-recognition-0004 (FP32)

My experiments:

Test1

• Env1
• benchmark_app -infer_precision CPU:bf16 -m myLocalPath\asl-recognition-0004.xml
• benchmark_app failed
• INFERENCE_PERCISION_HINT is "bfloat16"

Test2

• Env2
• benchmark_app -infer_precision CPU:bf16 -m myLocalPath\asl-recognition-0004.xml
• benchmark_app failed
• INFERENCE_PERCISION_HINT is "bfloat16"

Test3

• Env3
• benchmark_app -infer_precision CPU:bf16 -m myLocalPath\asl-recognition-0004.xml
• benchmark_app failed
• INFERENCE_PERCISION_HINT is "bfloat16"

Test4

• Env4
• benchmark_app -infer_precision CPU:bf16 -m myLocalPath\asl-recognition-0004.xml
• benchmark_app failed
• INFERENCE_PERCISION_HINT is "bfloat16"

Test5

• Env5
• benchmark_app -infer_precision CPU:bf16 -m myLocalPath\asl-recognition-0004.xml
• benchmark_app succeeded
• INFERENCE_PERCISION_HINT is "bfloat16"

Test6

• Env6
• benchmark_app -infer_precision CPU:bf16 -m myLocalPath\asl-recognition-0004.xml
• benchmark_app succeeded
• INFERENCE_PERCISION_HINT is "bfloat16"

My Investigations & Questions:

• According to reply from community of my last question :
  • OpenVINO2023.0 benchmark_app is succeeded with bf16, the INFERENCE_PERCISION_HINT is "float32"
  • Why does it different from mine, by using same model and calling same command?
• I have checked source code of OpenVINO and call stack  of benchmark_app 
  • 
  • It crashed in jit_avx512_core_amx_convolution_fwd_t, which is the convolution layer processer.
  • It seems "ldtilecfg" is an AMX instruction. reference: ldtilecfg 
  • When parameter "-infer_precision CPU:bf16" is used, OpenVINO inner variable "enforceBF16" will be set as true. 
  • It will lead OpenVINO infer with AMX instruction set, which is a new instruction set in 4th Generation Intel CPU.
  • However, currently in my test environments, such as WindowsServer2019, it leads to an “Illegal Instruction” crash.
  • I wonder if my issue only occurs on specified Windows kernel, and maybe there is any Windows kernel update or any Windows hotfix fixed the issue.
    • So community members could not reproduce my issue just because of your latest OS version/kernel.
    • But I can't test it because I don't know how to upgrade OS on aws under limitation of Amazon.
  • How can I make benchmark_app running successful and KEEP my current OS without installing any upgrade nor hotfix?

Best Regards

Clin B

[Aznie_Intel]

Hi Chill B,

Intel® Distribution of OpenVINO™ Toolkit is officially supported and validated on Windows 10/11 only. There might be kernel conflict that cause the Benchmark_app failed with your server.

On another note, devices similar to the ones we have used for benchmarking can be accessed using Intel® DevCloud for the Edge, a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of OpenVINO™ Toolkit. Learn more or Register here.

Hope this helps.

Regards,

Aznie

[gerald2]

I have the same issue, same cpu, on aws machine.

Is there a way to detect that the ldtilecfg instruction is not supported and then not execute this code, or disable the avx512 in that case?

[Aznie_Intel]

Hi Chill and Gerald,

I would suggest choosing an AWS EC2 instance that fulfills Intel® Distribution of OpenVINO™ Toolkit System Requirements to use OpenVINO. For the AMX setting, it only can be done by changing the Bios setting.

This thread will no longer be monitored since we have provided information. If you need any additional information from Intel, please submit a new question.

Regards,

Aznie