Re: Intel Advisor Internal Error with survey/roofline analysis

sham1810 · ‎08-23-2023

Hi,

I downloaded the latest Intel advisor and tried collecting roofline data with the following command for a sample matrix multiplication application:

./advixe-cl --collect=rooflilne --enable-cache-simulation --project-dir=/home/intel/sham/advisor-proj-dir -- /home/intel/sham/apps/ScimarkStable/scimark2_64static

The survey analysis never ends and keeps going, cannot stop with Ctrl-C, have to kill the process (sample application) in order to stop it.

After stopping, if i try to open the project in GUI, there is no data.

Console output:

intel@intel:/opt/intel/oneapi/advisor/2023.2.0/bin64$ ./advixe-cl --collect=roofline --enable-cache-simulation --project-dir=/home/intel/sham/advisor-proj-dir/ -- /home/intel/sham/apps/ScimarkStable/scimark2_64static
advisor: Warning: The Roofline is a special batch mode of data collection. It runs two analyses one by one. There are Survey Analysis and Trip Counts Analysis with FLOP respectively.
advisor: Starting command line: ./advixe-cl --collect survey --enable-cache-simulation --project-dir /home/intel/sham/advisor-proj-dir/ -- /home/intel/sham/apps/ScimarkStable/scimark2_64static
Intel(R) Advisor Command Line Tool
Copyright (C) 2009-2023 Intel Corporation. All rights reserved.
advisor: Collection started. To stop the collection, either press CTRL-C or enter from another console window: advisor -r /home/intel/sham/advisor-proj-dir/e000/hs000 -command stop.

-- After killing the application --

advisor: Collection stopped.
advisor: Warning: The application returned a non-zero exit value.
advisor: Warning: The application exit code can indicate an error while loading shared libraries, e.g. OpenMP runtime.

Possible Solution: Configure the PATH environment variable. If you use an Intel compiler, you can run the following configuration script:

source /[install-dir]/bin/compilervars.sh -arch [arch] -platform [platform]
Example: source /opt/intel/bin/compilervars.sh intel64
advisor: Opening result 19 % Loading '59014-59019.0.trace' file
advisor: Error: Cannot load data file `/home/intel/sham/advisor-proj-dir/e000/hs000/data.0/59014-59019.0.trace' (Data file is corrupted).
advisor: Opening result 31 % Processing profile metrics and debug information
advisor: Error: Error 0x4000002a (Database interface error) -- Cannot run data transformation `Add Fake Loop Data'.
advisor: Opening result 100 % done
advisor: Error: Data loading failed.
advisor: Error: No data is collected. Possible reasons:
- Workload is too small. No samples are collected.
- The application environment is not specified correctly. Shared libraries are not found.
- The command-line parameters required to profile the application are missing.
Consider checking the collection log for additional information.
intel@intel:/opt/intel/oneapi/advisor/2023.2.0/bin64$

Other types of collection "map, tripcounts" do not seem to have this problem.

I tried this command and it seems to work fine

./advixe-cl --collect=map --enable-cache-simulation --project-dir=/home/intel/sham/advisor-proj-dir/ -- /home/intel/sham/apps/ScimarkStable/scimark2_64static

intel@intel:/opt/intel/oneapi/advisor/2023.2.0/bin64$ ./advixe-cl --collect=map --enable-cache-simulation --project-dir=/home/intel/sham/advisor-proj-dir/ -- /home/intel/sham/apps/ScimarkStable/scimark2_64static
Intel(R) Advisor Command Line Tool
Copyright (C) 2009-2023 Intel Corporation. All rights reserved.
advisor: Collection started. To stop the collection, either press CTRL-C or enter from another console window: advisor -r /home/intel/sham/advisor-proj-dir/e000/mp000 -command stop.
** **
** SciMark2 Numeric Benchmark, see http://math.nist.gov/scimark **
** for details. (Results can be submitted to pozo@nist.gov) **
** **
Using 2.00 seconds min time per kernel.
FFT kernel cycles: 65536
SOR kernel cycles: 65536
Monte Carlo kernel cycles: 268435456
MatMult kernel cycles: 524288
LU decomp kernel cycles: 16384
Composite Score: 698.35
FFT Mflops: 732.87 (N=1024)
SOR Mflops: 1052.19 (100 x 100)
MonteCarlo: Mflops: 36.17
Sparse matmult Mflops: 787.62 (N=1000, nz=5000)
LU Mflops: 882.88 (M=100, N=100)
advisor: Warning: No SITE annotations were encountered, so no stride/alignment data can be reported.
advisor: Loading result... 25 % done
intel@intel:/opt/intel/oneapi/advisor/2023.2.0/bin64$

JaideepK_Intel · ‎08-25-2023

Hi,

Thank you for posting in Intel Communities.

Please share the processor details and Linux distribution you are using (Ubuntu, Centos, RHEL, Fedora.. etc.), and also please share the exact steps to build scimark2_64static binary.

An alternate way to run a roofline analysis, try running a survey and trip count analysis first (via CLI), and also please share the output for each analysis. Please follow the below steps.

To run survey analysis run the below command:

advixe-cl -collect survey -project-dir MyResults --enable-cache-simulation -- ./scimark2

To run Tripcount run the below command:

advixe-cl -collect tripcounts --enable-cache-simulation -flop -project-dir MyResults -- ./scimark2

Now we can run roofline analysis with the below command:

advixe-cl -collect roofline --enable-cache-simulation -project-dir MyResults -- ./scimark2

If this resolves your issue, make sure to accept this as a solution. This would help others with similar issues.

Thanks,

Jaideep

sham1810 · ‎08-25-2023

Hello,

As I've said, I tried collecting both (survey and tripcounts) and I am having trouble with survey analysis. Survey analysis never ends, keeps going on forever. Please refer to my above post.

Can you help me diagnose the problem here?

Thanks

Shreyank

JaideepK_Intel · ‎08-25-2023

Hi,

From the above log, when you run the roofline analysis survey and tripcount analysis are going to run. I want to know if, when you run only survey analysis, you are facing the same issue or not.

If scimark2_64static and scimark binaries are the same, then we are able to run the roofline analysis without any issue

Please share the processor details and Linux distribution you are using (Ubuntu, Centos, RHEL, Fedora.. etc.), and also please share the exact steps to build scimark2_64static binary.

Thanks,

Jaideep

sham1810 · ‎08-28-2023

Hello,

I face this issue even if I just run the survey analysis.

Here is the console output:

intel@intel:/opt/intel/oneapi/advisor/2023.2.0/bin64$ ./advixe-cl -collect survey -project-dir /home/intel/sham/advisor-proj-dir/ --enable-cache-simulation -- /home/intel/sham/apps/ScimarkStable/scimark2_64static
Intel(R) Advisor Command Line Tool
Copyright (C) 2009-2023 Intel Corporation. All rights reserved.
advisor: Collection started. To stop the collection, either press CTRL-C or enter from another console window: advisor -r /home/intel/sham/advisor-proj-dir/e000/hs000 -command stop.

-------------- After I kill the target application process (after waiting for like 10 minutes) -------------------------------

advisor: Collection stopped.
advisor: Warning: The application returned a non-zero exit value.
advisor: Warning: The application exit code can indicate an error while loading shared libraries, e.g. OpenMP runtime.

Possible Solution: Configure the PATH environment variable. If you use an Intel compiler, you can run the following configuration script:

source /[install-dir]/bin/compilervars.sh -arch [arch] -platform [platform]
Example: source /opt/intel/bin/compilervars.sh intel64
advisor: Opening result 19 % Loading '74945-74950.0.trace' file
advisor: Error: Cannot load data file `/home/intel/sham/advisor-proj-dir/e000/hs000/data.0/74945-74950.0.trace' (Data file is corrupted).
advisor: Opening result 31 % Processing profile metrics and debug information
advisor: Error: Error 0x4000002a (Database interface error) -- Cannot run data transformation `Add Fake Loop Data'.
advisor: Opening result 100 % done
advisor: Error: Data loading failed.
advisor: Error: No data is collected. Possible reasons:
- Workload is too small. No samples are collected.
- The application environment is not specified correctly. Shared libraries are not found.
- The command-line parameters required to profile the application are missing.
Consider checking the collection log for additional information.

I have attached the scimark folder

There is a makefile in the main folder

Use:

make scimark2_64static

to build the executable

./scimark2_64_static

to launch the application

OS and CPU information:

intel@intel:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

intel@intel:~$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 112
On-line CPU(s) list: 0-111
Thread(s) per core: 2
Core(s) per socket: 28
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz
Stepping: 7
CPU MHz: 1000.007
CPU max MHz: 4000.0000
CPU min MHz: 1000.0000
BogoMIPS: 5400.00
Virtualization: VT-x
L1d cache: 1.8 MiB
L1i cache: 1.8 MiB
L2 cache: 56 MiB
L3 cache: 77 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84
,86,88,90,92,94,96,98,100,102,104,106,108,110
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79,81,83,85
,87,89,91,93,95,97,99,101,103,105,107,109,111
Vulnerability Itlb multihit: KVM: Vulnerable
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Retbleed: Mitigation; Enhanced IBRS
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; TSX disabled
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe sys
call nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni p
clmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc
_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin
ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi
2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt
xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_cl
ear flush_l1d arch_capabilities

JaideepK_Intel · ‎08-29-2023

Hi,

Thank you for sharing your machine/processor details, we were able to reproduce your issue on Intel(R) Xeon(R) Platinum 8260M CPU @ 2.40GHz(Cascade Lake) and we are working on this internally and get back to you with an update.

we found that the collection was getting hanged/stuck and we manually killed the process with process ID.

Observations:

With the above sample we were able to run roofline analysis on Alderlake (Core processor) without any issues.

Thanks,

Jaideep

sham1810 · ‎08-31-2023

Hi,

"With the above sample we were able to run roofline analysis on Alderlake (Core processor) without any issues."

- Okay but I don't have an Alderlake system available. Is there an older version of Intel Advisor that works for the above scenario that I can use with my system while you debug this issue?

Thanks

Shreyank

JaideepK_Intel · ‎09-06-2023

Hi,

In the meantime, can you please try to build Scimark2 with the below binary? It is working for us without any issues.

link: https://math.nist.gov/scimark2/download_c.html

Thanks,

Jaideep

JaideepK_Intel · ‎09-14-2023

Hi,

We are working on this internally and we will get back to you with an update.

Thanks,

Jaideep

JaideepK_Intel · ‎09-29-2023

Hi,

We are looking into your issue, In the meantime have you tried to build the Scimark2 binary from the above link that we shared?

Thanks,

Jaideep