Failed to run intel advisor on openvino benchmarkapp

[yaranwu]

        Recently, I tried  to use intel advisor to dump each kernel  roofline model of OpenVino and to check if we can have any chance to optimize key kernels in YoloV5m. But i met some errors. Can you share any suggestions on this error?

when I run advisor for openvino, I get the error GTPin ERROR: mit != _moduleMap.end()

advisor -collect=roofline -profile-gpu --project-dir=./advi -- $POR_MODELS_HOME/openvino/framework/bin/intel64/Release/benchmark_app -m $POR_MODELS_HOME/model/ov_models/yolov5-m/int8/yolo_v5m.xml -d GPU.$ZE_AFFINITY_MASK -b 1 -hint none -t 50 -nstreams 1  -use_device_mem -shape [1,3,416,416] -nireq 2

// Output

Intel(R) Advisor Command Line Tool

Copyright (C) 2009-2024 Intel Corporation. All rights reserved.

advisor: Collection started. To stop the collection, either press CTRL-C or enter from another console window: advisor -r /home/gta/advi/e000/hs000 -command stop.

[Step 1/11] Parsing and validating input arguments

[ INFO ] Parsing input parameters

[Step 2/11] Loading OpenVINO Runtime

[ INFO ] OpenVINO:

[ INFO ] Build ................................. 2023.2.0-13121-4c31a9bef77-HEAD

[ INFO ]

[ INFO ] Device info:

[ INFO ] GPU

[ INFO ] Build ................................. 2023.2.0-13121-4c31a9bef77-HEAD

[ INFO ]

[ INFO ]

[Step 3/11] Setting device configuration

[ WARNING ] Batch size is set. Auto batching will be disabled

[Step 4/11] Reading model files

[ INFO ] Loading model files

[ INFO ] Read model took 25.13 ms

[ INFO ] Original model I/O parameters:

[ INFO ] Network inputs:

[ INFO ]     images (node: images) : f32 / [N,C,H,W] / [1,3,640,640]

[ INFO ] Network outputs:

[ INFO ]     output (node: output) : f32 / [...] / [1,3,80,80,85]

[ INFO ]     1052 (node: 1052) : f32 / [...] / [1,3,40,40,85]

[ INFO ]     1072 (node: 1072) : f32 / [...] / [1,3,20,20,85]

[Step 5/11] Resizing model to match image sizes and given batch

[ INFO ] Reshaping model: 'images': [1,3,416,416]

[ INFO ] Reshape model took 5.64 ms

[Step 6/11] Configuring input of the model

[ INFO ] Model batch size: 1

[ INFO ] Network inputs:

[ INFO ]     images (node: images) : u8 / [N,C,H,W] / [1,3,416,416]

[ INFO ] Network outputs:

[ INFO ]     output (node: output) : f32 / [...] / [1,3,52,52,85]

[ INFO ]     1052 (node: 1052) : f32 / [...] / [1,3,26,26,85]

[ INFO ]     1072 (node: 1072) : f32 / [...] / [1,3,13,13,85]

[Step 7/11] Loading the model to the device

GTPin ERROR: mit != _moduleMap.end()

at: CreateKernel : 832

advisor: Error: [Instrumentation Engine]: GTPin:  mit != _moduleMap.end()

advisor: Collection stopped.

advisor: Warning: The application returned a non-zero exit value.

advisor: Opening result 19 % Loading 'userapicollector-183840-7d92003c.trace' f

advisor: Error: Cannot load data file `/home/gta/advi/e000/hs000/data.0/userapicollector-183840-7d92003c.modmap' ().

advisor: Error: Cannot load data file `/home/gta/advi/e000/hs000/data.0/183835-183840.0.trace' (Data file is corrupted).

advisor: Error: Cannot load data file `/home/gta/advi/e000/hs000/data.0/userapicollector-183840-7d92003c.trace' (Data file is corrupted).

advisor: Opening result 39 % Processing profile metrics and debug information

advisor: Error: Error 0x4000002a (Database interface error) -- Cannot run data transformation `Add Fake Loop Data'.

advisor: Opening result 100 % done

advisor: Error: Data loading failed.

advisor: Error: No data is collected. Possible reasons:

- Workload is too small. No samples are collected.

- The application environment is not specified correctly. Shared libraries are not found.

- The command-line parameters required to profile the application are missing.

Consider checking the collection log for additional information.

[yuzhang3_intel]

Please get Advisor version using command below:

$ advisor -version

And get $ sycl-ls output,  thanks.

[briannguyen]

Hi, I am working with Yaran on this.

Here are the respective outputs.

$ advisor --version

Intel(R) Advisor 2024.1.0 (build 615301) Command Line Tool

Copyright (C) 2009-2024 Intel Corporation. All rights reserved.

$ sycl-ls

[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device OpenCL 1.2  [2024.17.3.0.08_160000]

[opencl:cpu:1] Intel(R) OpenCL, Intel(R) Xeon(R) Gold 6336Y CPU @ 2.40GHz OpenCL 3.0 (Build 0) [2024.17.3.0.08_160000]

[opencl:gpu:2] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Flex 170 OpenCL 3.0 NEO  [23.43.27642.21]

[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Data Center GPU Flex 170 1.3 [1.3.27642]

[yuzhang3_intel]

Is the OS Ubuntu? Which version? Thanks.

The NEO version is old, please update the NEO to 23.52.28202.14 and try again.

https://github.com/intel/compute-runtime

[briannguyen]

Yes, I am running Ubuntu 22.04.4 LTS.

$ uname -sr

Linux 5.15.0-73-generic

After updating the NEO to 23.52.28202.14, the same issue of GTPin ERROR: mit != _moduleMap.end() still occurs.

$ sycl-ls

[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device OpenCL 1.2  [2024.17.3.0.08_160000]

[opencl:cpu:1] Intel(R) OpenCL, Intel(R) Xeon(R) Gold 6336Y CPU @ 2.40GHz OpenCL 3.0 (Build 0) [2024.17.3.0.08_160000]

[opencl:gpu:2] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Flex 170 OpenCL 3.0 NEO  [23.52.28202.14]

[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Data Center GPU Flex 170 1.3 [1.3.28202]

$ advisor --version

Intel(R) Advisor 2024.1.0 (build 615301) Command Line Tool

Copyright (C) 2009-2024 Intel Corporation. All rights reserved.

$ advisor --collect=roofline --profile-gpu -- $POR_MODELS_HOME/openvino/framework/bin/intel64/Release/benchmark_app -m $POR_MODELS_HOME/model/ov_models/yolov5-m/int8/yolo_v5m.xml -d GPU.$ZE_AFFINITY_MASK -b 1 -hint none -t 50 -nstreams 1  -use_device_mem -shape [1,3,416,416] -nireq 2

advisor: The 'roofline' is a special batch mode for data collection. It runs several analyses one by one.

advisor: Starting command line: advisor --collect=survey --profile-gpu -- /home/gta/client/code/openvino/23ww47-release-2/openvino/framework/bin/intel64/Release/benchmark_app -m /home/gta/client/code/openvino/23ww47-release-2/model/ov_models/yolov5-m/int8/yolo_v5m.xml -d GPU. -b 1 -hint none -t 50 -nstreams 1 -use_device_mem -shape [1,3,416,416] -nireq 2

Intel(R) Advisor Command Line Tool

Copyright (C) 2009-2024 Intel Corporation. All rights reserved.

advisor: Collection started. To stop the collection, either press CTRL-C or enter from another console window: advisor -r /home/gta/e000/hs000 -command stop.

[Step 1/11] Parsing and validating input arguments

[ INFO ] Parsing input parameters

[Step 2/11] Loading OpenVINO Runtime

[ INFO ] OpenVINO:

[ INFO ] Build ................................. 2023.2.0-13121-4c31a9bef77-HEAD

[ INFO ]

[ INFO ] Device info:

[ INFO ] GPU

[ INFO ] Build ................................. 2023.2.0-13121-4c31a9bef77-HEAD

[ INFO ]

[ INFO ]

[Step 3/11] Setting device configuration

[ WARNING ] Batch size is set. Auto batching will be disabled

[Step 4/11] Reading model files

[ INFO ] Loading model files

[ INFO ] Read model took 24.61 ms

[ INFO ] Original model I/O parameters:

[ INFO ] Network inputs:

[ INFO ]     images (node: images) : f32 / [N,C,H,W] / [1,3,640,640]

[ INFO ] Network outputs:

[ INFO ]     output (node: output) : f32 / [...] / [1,3,80,80,85]

[ INFO ]     1052 (node: 1052) : f32 / [...] / [1,3,40,40,85]

[ INFO ]     1072 (node: 1072) : f32 / [...] / [1,3,20,20,85]

[Step 5/11] Resizing model to match image sizes and given batch

[ INFO ] Reshaping model: 'images': [1,3,416,416]

[ INFO ] Reshape model took 5.68 ms

[Step 6/11] Configuring input of the model

[ INFO ] Model batch size: 1

[ INFO ] Network inputs:

[ INFO ]     images (node: images) : u8 / [N,C,H,W] / [1,3,416,416]

[ INFO ] Network outputs:

[ INFO ]     output (node: output) : f32 / [...] / [1,3,52,52,85]

[ INFO ]     1052 (node: 1052) : f32 / [...] / [1,3,26,26,85]

[ INFO ]     1072 (node: 1072) : f32 / [...] / [1,3,13,13,85]

[Step 7/11] Loading the model to the device

GTPin ERROR: mit != _moduleMap.end()

at: CreateKernel : 832

advisor: Error: [Instrumentation Engine]: GTPin:  mit != _moduleMap.end()

advisor: Collection stopped.

advisor: Warning: The application returned a non-zero exit value.

advisor: Opening result 19 % Loading 'userapicollector-75235-f6f54349.trace' fi

advisor: Error: Cannot load data file `/home/gta/e000/hs000/data.0/userapicollector-75235-f6f54349.modmap' ().

advisor: Error: Cannot load data file `/home/gta/e000/hs000/data.0/75230-75235.0.trace' (Data file is corrupted).

advisor: Error: Cannot load data file `/home/gta/e000/hs000/data.0/userapicollector-75235-f6f54349.trace' (Data file is corrupted).

advisor: Opening result 39 % Processing profile metrics and debug information

advisor: Error: Error 0x4000002a (Database interface error) -- Cannot run data transformation `Add Fake Loop Data'.

advisor: Opening result 100 % done

advisor: Error: Data loading failed.

advisor: Error: No data is collected. Possible reasons:

 - Workload is too small. No samples are collected.

 - The application environment is not specified correctly. Shared libraries are not found.

 - The command-line parameters required to profile the application are missing.

 Consider checking the collection log for additional information.

[yuzhang3_intel]

Thanks for the feedback! Let me check if the GTPin issue is fixed in the next release, there is one similar issue that is monitored.

[yuzhang3_intel]

Just confirmed that the GTPin issue will be fixed in the next release.