Analyzers
Talk to fellow users of Intel Analyzer tools (Intel VTune™ Profiler, Intel Advisor)
5132 Discussions

FLOP measurements differ between VTune and Advisor

jgwohlbier
Beginner
3,326 Views

I ran the same code through both vtune and advisor and the reported flops numbers are different by two orders of magnitude. Vtune reports 108 SP GFLOPS and Advisor reports 1.39 GFLOPS. How do I proceed to determine which, if either, is correct. 

 

Using 2019 update 1.

jgwohlbier@sdh01:.../ipnsw$ advixe-cl --version
Intel(R) Advisor 2019 Update 1 (build 579143) Command Line Tool
Copyright (C) 2009-2018 Intel Corporation. All rights reserved.

Labels (2)
0 Kudos
13 Replies
Ruslan_M_Intel
Employee
3,321 Views

Could you please share more details? It would be nice if you shared your projects  

0 Kudos
Dmitry_P_Intel1
Employee
3,303 Views

Hello,

It also makes sense to update the tools to more recent versions and recheck. Both products might have had fixes related to FLOPS counting since 2018.

Or if it is possible to provide the code we can do the check from our side and triage the issue if it still persists. 

Thanks & Regards, Dmitry

 

0 Kudos
Gopika_Intel
Moderator
3,298 Views

Hi


Thank you for using the Analyzers forum. Could you please share us with the reproducer code and steps to reproduce it, so that we can try it from our end.


Regards

Gopika


0 Kudos
jgwohlbier
Beginner
3,287 Views

All, thanks for the replies. I did not have access to the 2020 tools, but I should get access today. I will rerun the tests and see if the issue persists. Creating a reproducer should be doable if the issue persists, but I will have to obtain permission to do so.

Thanks again!

jgw

 

0 Kudos
jgwohlbier
Beginner
3,274 Views

I got access to the 2020 tools. vtune gave the same result. Advisor did not complete the second step of the roofline data collection.

OMP_DISPLAY_ENV=true OMP_NUM_THREADS=12 KMP_HW_SUBSET=1s advixe-cl -data-limit=10000 -collect tripcounts -flop -project-dir ./advisor -- python -u main.py
Intel(R) Advisor Command Line Tool
Copyright (C) 2009-2020 Intel Corporation. All rights reserved.
advixe: Opening result 25 % done
advixe: Preparing frequently used data 38 % done
advixe: Preparing frequently used data 100 % done
advixe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: advixe-cl -r /DATA/SDH/jgwohlbier/SDH/prog-eval/convnet/advisor/e000/trc000 -command stop.

OPENMP DISPLAY ENVIRONMENT BEGIN
_OPENMP = '201511'
OMP_DYNAMIC = 'FALSE'
OMP_NESTED = 'FALSE'
OMP_NUM_THREADS = '12'
OMP_SCHEDULE = 'DYNAMIC'
OMP_PROC_BIND = 'FALSE'
OMP_PLACES = ''
OMP_STACKSIZE = '0'
OMP_WAIT_POLICY = 'PASSIVE'
OMP_THREAD_LIMIT = '4294967295'
OMP_MAX_ACTIVE_LEVELS = '2147483647'
OMP_CANCELLATION = 'FALSE'
OMP_DEFAULT_DEVICE = '0'
OMP_MAX_TASK_PRIORITY = '0'
OMP_DISPLAY_AFFINITY = 'FALSE'
OMP_AFFINITY_FORMAT = 'level %L thread %i affinity %A'
OPENMP DISPLAY ENVIRONMENT END

OPENMP DISPLAY ENVIRONMENT BEGIN
_OPENMP='201611'
[host] OMP_AFFINITY_FORMAT='OMP: pid %P tid %i thread %n bound to OS proc set {%A}'
[host] OMP_ALLOCATOR='omp_default_mem_alloc'
[host] OMP_CANCELLATION='FALSE'
[host] OMP_DEBUG='disabled'
[host] OMP_DEFAULT_DEVICE='0'
[host] OMP_DISPLAY_AFFINITY='FALSE'
[host] OMP_DISPLAY_ENV='TRUE'
[host] OMP_DYNAMIC='FALSE'
[host] OMP_MAX_ACTIVE_LEVELS='1'
[host] OMP_MAX_TASK_PRIORITY='0'
[host] OMP_NESTED: deprecated; max-active-levels-var=1
[host] OMP_NUM_TEAMS='0'
[host] OMP_NUM_THREADS='12'
[host] OMP_PLACES: value is not defined
[host] OMP_PROC_BIND='false'
[host] OMP_SCHEDULE='static'
[host] OMP_STACKSIZE='8M'
[host] OMP_TARGET_OFFLOAD=DEFAULT
[host] OMP_TEAMS_THREAD_LIMIT='0'
[host] OMP_THREAD_LIMIT='2147483647'
[host] OMP_TOOL='enabled'
[host] OMP_TOOL_LIBRARIES: value is not defined
[host] OMP_WAIT_POLICY='PASSIVE'
OPENMP DISPLAY ENVIRONMENT END

100%|███████████████████████████████████████████| 79/79 [01:33<00:00, 1.19s/it]
{"epoch": 0, "test_acc": 0.855, "time": 97.50701689720154}
100%|███████████████████████████████████████████| 79/79 [01:34<00:00, 1.19s/it]
{"epoch": 1, "test_acc": 0.9465, "time": 195.63309955596924}
100%|███████████████████████████████████████████| 79/79 [01:19<00:00, 1.01s/it]
{"epoch": 2, "test_acc": 0.954, "time": 278.2721538543701}
19%|████████▏ | 15/79 [00:15<01:03, 1.00it/s]advixe: Warning: The application returned a non-zero exit value.
advixe: Opening result 71 % Loading 'tripcounts_58828_1.tcs' file
advixe: Error: Cannot load data file `/DATA/SDH/jgwohlbier/SDH/prog-eval/convnet/advisor/e000/hs000/data.1/tripcounts_58817_0.tcs' (Data file is corrupted).
advixe: Error: Cannot load data file `/DATA/SDH/jgwohlbier/SDH/prog-eval/convnet/advisor/e000/hs000/data.1/tripcounts_58822_0.tcs' (Data file is corrupted).
advixe: Error: Cannot load data file `/DATA/SDH/jgwohlbier/SDH/prog-eval/convnet/advisor/e000/hs000/data.1/tripcounts_58828_0.tcs' (Data file is corrupted).
advixe: Opening result 99 % done
advixe: Preparing frequently used data 38 % done
advixe: Preparing frequently used data 100 % done

0 Kudos
Zakhar_M_Intel1
Employee
3,258 Views

Hello

So your question is about FLOPS counting in Python code.

Which Python version do you use? Also, could you please share your python application (which you profile)? 

Also, if you don't use many 3rd parties, it would be interesting to repeat your test with the variant of python3 which is packaged with Advisor. Just replace "python" with "advixe-python" in your profiling (survey and tripcounts) steps and let us know the output.

 

Few more notes:

1) The log shared by you doesn't necessary indicate that FLOP step was failing. 

2) In general to get reasonable FLOPS data for python application one has to add up --stacks flag in tripcounts stage

But i'd not stress items (1) and (2) for now, because I agree that most likely there was an issue and therefore it would be nice to obtain the reproducer.

Thanks in advance! 

0 Kudos
Gopika_Intel
Moderator
3,243 Views

Hi,

We haven’t heard back from you, was the error resolved, let us know your updates.

Regards

Gopika


0 Kudos
jgwohlbier
Beginner
3,237 Views

I did get some results using the latest IPS. In terms of the original post the discrepancy between FLOPS between VTune and Advisor dropped from a factor of 100 to a factor of 2. I will have to go with this result for now.

Advisor still intermittently appears to crash. I don't have time at the moment to create a reproducer. I was doing a couple different types of runs. A simple convolutional neural network and a graph neural network, both using PyTorch. There are examples that are similar within pytorch_geometric, e.g., https://github.com/rusty1s/pytorch_geometric/blob/master/examples/graph_sage_unsup.py

I would be interested to see how Advisor does on that example. I may have a chance to test it at a later time.

0 Kudos
Gopika_Intel
Moderator
3,213 Views

Hi,


We’re forwarding this thread to subject matter experts who can help you out.


Regards

Gopika


0 Kudos
DMITRY_T_Intel
Employee
3,201 Views

Hi John,

I see that initial issue has been resolved. Could you please describe current issue with Advisor? Please attach archive with Advisor results directory. Please attach a reproducer and describe steps for reproducing the issue. Please tell me the operating system version and Advisor version. Thank you!


0 Kudos
DMITRY_T_Intel
Employee
3,082 Views

Hi John,

Could you please tell us the status of this issue? Do you need further assistance? If not then we will close the ticket and will not monitor this forum topic.

Thank you!


0 Kudos
jgwohlbier
Beginner
3,065 Views

Hi, I don't have time to work on it at the moment. You can close it and I will reopen if needed.

jgw

 

0 Kudos
DMITRY_T_Intel
Employee
3,060 Views

Hi John,

Thank you for this information. I'm closing the ticket and will not monitor this forum topic. Please feel free to create a new forum topic and we will assist in the issue investigation.

Thank you!


0 Kudos
Reply