- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am trying to use advisor (2019 update 4) to profile the roofline, but encountered the internal error issue and the advisor hung (not crash). Do you have any suggestions on that? Thanks!
Here attaches the log:
# advixe-cl -collect roofline --project-dir roofline -target-pid=426780 -stop-after=10
advixe: Warning: The Roofline is a special batch mode of data collection. It runs two analyses one by one. There are Survey Analysis and Trip Counts Analysis with FLOP respectively.
advixe: Starting command line: advixe-cl --collect survey --project-dir roofline --stop-after 10 --target-pid 426780 --
Intel(R) Advisor Command Line Tool
Copyright (C) 2009-2019 Intel Corporation. All rights reserved.
advixe: Opening result 25 % done
advixe: Preparing frequently used data 0 % done
advixe: Preparing frequently used data 100 % done
advixe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: advixe-cl -r /home/pujiang/tensorflow/roofline/e000/hs002 -command stop.
^Cadvixe: Collection detached.
advixe: Collection stopped.
advixe: Opening result 19 % Loading '418800-426780.0.trace' file
advixe: Error: Cannot load data file `/home/pujiang/tensorflow/roofline/e000/hs002/data.1/.import' ().
advixe: Opening result 21 % Resolving information for `libm.so.6'
advixe: Warning: Cannot locate debugging information for file `/lib64/libgomp.so.1'.
advixe: Warning: Cannot locate debugging information for file `/lib64/libpthread.so.0'.
advixe: Warning: Cannot locate debugging information for file `/lib64/libstdc++.so.6'.
advixe: Warning: Cannot locate debugging information for file `/lib64/libm.so.6'.
advixe: Warning: Cannot locate debugging information for file `/lib64/libc.so.6'.
advixe: Opening result 24 % Resolving information for `libtensorflow_framework.
advixe: Warning: Cannot locate debugging information for file `/root/.cache/bazel/_bazel_root/3a774584f77b95d5cd1e53968137dc39/execroot/org_tensorflow/bazel-out/k8-opt/bin/_solib_k8/_U_S_Stensorflow_Scc_Sexample_Cexample___Utensorflow/libtensorflow_framework.so'.
advixe: Opening result 25 % Resolving information for `example'
advixe: Warning: Cannot locate debugging information for file `/root/.cache/bazel/_bazel_root/3a774584f77b95d5cd1e53968137dc39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/cc/example/example'.
advixe: Opening result 99 % done
advixe: Preparing frequently used data 0 % done
advixe: Preparing frequently used data 100 % done
Elapsed Time: 7.28s
Total CPU time: 149.93
Time in 5 vectorized loops: 3.72
advixe: Starting command line: advixe-cl --collect tripcounts --project-dir roofline --stop-after 10 --target-pid 426780 --flop --no-trip-counts --
Intel(R) Advisor Command Line Tool
Copyright (C) 2009-2019 Intel Corporation. All rights reserved.
advixe: Opening result 25 % done
advixe: Preparing frequently used data 0 % done
advixe: Preparing frequently used data 100 % done
advixe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: advixe-cl -r /home/pujiang/tensorflow/roofline/e000/trc001 -command stop.
advixe: Error: An internal error has occurred. Our apologies for this inconvenience. Please gather a description of the steps leading up to the problem and contact the Intel customer support team.
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Could you please share your project?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I've managed to reproduce the issue and see that attach/detach functionality doesn't work as expected for roofline collection. That's why the error happens (tripcounts collection failed to attach to the same pid due to survey collection doesn't detach correctly). Need to investigate more.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Got. Then before the fix, I will start the process in the command line.
BTW, ask a question here: I see there are many points for small functions in roofline graph, but how to get the roofline for a big function which across multiple modules? (I am new to advisor)
Thanks!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Are you talking about callchains? If so you can try "roofline" collection with stacks enabled (-stacks)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
Here you can find more details about roofline with callstacks
https://software.intel.com/en-us/articles/roofline-with-callstacks
https://stackoverflow.com/questions/55135055/intel-advisor-inspect-method-including-all-submethods
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Roofline is combination of two collections: Survey and Tripcounts/Flops. Any collection could to attach to a process separately, But it is impossible to attach to a process which has been already profiled (with detach) by one another collector. This moment is missed in Advisor User Guide for a now. BTW, roofline in "attach to process" mode is a lot of not common situation, because of Survey collection collect basic information and Tripcounts/Flops collect extended information which would be matched on Survey results. If we try to collect roofline in "attach to process" mode, then we have situation when Survey collect information for one application state and Tripcounts/Flops - for one another, so we will have situation when Tripcounts/Flops data will not be able to be matched on Survey results.
Note: Tripcounts/Flop collector doe not support "-stacks" data collection in "attach-to-process" mode.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
何普江 (Intel) wrote:how to get the roofline for a big function which across multiple modules?
You can use "pause-resume" feature: put itt_resume at the beginning of the function and itt_pause at the end of the function. Now it needs to start collection in "start-paused" mode. Result: Advisor will collect data only for function of interest.

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page