Resolving symbols on remote Linux machine

[Nathan_K_5]

I'm using VTune 2017 Beta to make remote measurements from one Linux machine to another.  I've found it very difficult figure to get all the symbols to resolve correctly.  The documentation seems really weak on explaining what's actually happening.   

Eventually I found enough information here in the forums to get things working from the command line (on either the host or target), but I still haven't figured out how to do what I want from the GUI.  It's possible that I should just give up and stick with the command line, but I feel like I'm close enough that maybe someone can tell me what I'm missing.  I'm not certain exactly what's actually happening when you do a remote collection from the GUI. First the GUI on the host launches amplxe-cl on the host, which then executes amplxe-runss on the target to collect the data.  On the target, amplxe-runss then calls "sep" on the target (because it's called with -collect-with runsa?).

I don't understand though which files get transferred from the target to the host, and whether they are transferred immediately or during "finalization"?   But the important thing I was missing for a long time is that all symbol resolution happens on the host machine, and that all the -search-dir arguments (adjusted in the GUI with the Binary/Symbol search button in Analysis Target) apply only to the Host Machine, and that despite the text at the top of the page, the "symbol server paths" are not available for Linux.   Thus if you want the debugging symbols to be available, you need to make them locally accessible through the file system: https://software.intel.com/en-us/forums/intel-vtune-amplifier-xe/topic/534965

Things got better once I mounted the target /usr/lib/debug directory on the host with sshfs, and then specified this directory in the GUI as a place to search.   But still I was missing a lot of important symbols, like those for libc6 (the host and target are running different versions).   Maybe the problem is that the debugging symbols in Ubuntu's .dbg and .dbgsym files are not necessarily laid out quite as VTune expect?  But when I ran from the command line locally on the target, it worked, so I know the information must be there.  

The "Search Order" page (https://software.intel.com/en-us/node/544099) helped understand that certain directories are treated differently, and that the hardcoded /usr/lib/debug works differently that a /usr/lib/debug/remote specified with -search-dir (where "remote" is the remote mounted /usr/lib/debug).  Using the hints here https://software.intel.com/en-us/forums/intel-vtune-amplifier-xe/topic/279763, I was able to figure out that switching to "-search-dir  sym:pr=/usr/lib/debug/remote" (added "r" for recursive) works!  But contrary to that comment, I can find no "check box at the end of the line with the search path" that allows me to specify this in the GUI.

1) Is there any way to specify a recursive symbol search directory from within the GUI? 

2) Is the format of the argument to -search-dir documented anywhere but the forums?

3) Is there some better way to resolve remote symbols that doesn't require remote mounting directories, running amplxe-cl with strace to figure out what it's doing, and sifting through years of forum questions for undocumented features?  It seems like this would be a major feature that's used very regularly, but it feels like I'm going down an untrodden path.

[Denis_M_Intel]

Hello,

1) Is there any way to specify a recursive symbol search directory from within the GUI? 

It is not possible.

2) Is the format of the argument to -search-dir documented anywhere but the forums?

It has only one argument - directory path. "sym:pr" syntax is deprecated. You can find description for this option in "./amplxe-cl -help finalize" output or VTune Help.

3) Is there some better way to resolve remote symbols that doesn't require remote mounting directories, running amplxe-cl with strace to figure out what it's doing, and sifting through years of forum questions for undocumented features?  It seems like this would be a major feature that's used very regularly, but it feels like I'm going down an untrodden path.

The only way currently available is to have symbol files on the host file system and specify binary/symbol search directory for each symbol file if they are located in different subdirectories. For example, if path is "/dir1/dir2/symbol_file.dbg", correct option is "-search-dir /dir1/dir2".

[Nathan_K_5]

1) Is there any way to specify a recursive symbol search directory from within the GUI? 

It is not possible.

Thanks, that's nicely clear.

2) Is the format of the argument to -search-dir documented anywhere but the forums?

It has only one argument - directory path. "sym:pr" syntax is deprecated. You can find description for this option in "./amplxe-cl -help finalize" output or VTune Help.

Unfortunately a little less clear.  Do you mean that "sym:", "bin:" and "all:" are deprecated, that the flags after them are deprecated, or that just the "r" for recursive flag is deprecated?

The online help (https://software.intel.com/en-us/node/544283) does say to use a bare DIR, but "amplxe-cl -help finalize" still refers to the non-specified <string> and the example at the bottom of the page is "amplxe-cl -finalize -r r001hs -search-dir all=example/bin".   If this hasn't already been updated between 2017 Beta and 2017 Final, perhaps this could be fixed to agree with the online docs?

Also, the code generated by the GUI (viewed with the "Command Line" popup) still uses "--search-dir sym:p=/usr/lib/debug/remote" syntax.  Since this seems like a canonical reference, if this syntax has been deprecated perhaps this could be changed to use whatever is now correct?

3) Is there some better way to resolve remote symbols that doesn't require remote mounting directories, running amplxe-cl with strace to figure out what it's doing, and sifting through years of forum questions for undocumented features?  It seems like this would be a major feature that's used very regularly, but it feels like I'm going down an untrodden path.

The only way currently available is to have symbol files on the host file system and specify binary/symbol search directory for each symbol file if they are located in different subdirectories. For example, if path is "/dir1/dir2/symbol_file.dbg", correct option is "-search-dir /dir1/dir2".

OK, I can work with that.  Having to specify each individual directory is an awkward interface, but I can get 95% of what I want with these: 

# sshfs user@remote:/usr/lib/debug /usr/lib/debug/remote

# amplxe-cl -finalize -r r081hpc1 --search-dir /usr/lib/debug/remote --search-dir /usr/lib/debug/remote/lib/x86_64-linux-gnu --search-dir /usr/lib/debug/remote/usr/lib/x86_64-linux-gnu  --search-dir /usr/lib/debug/remote/lib/modules/4.4.0-38-generic/kernel/drivers/scsi/megaraid --search-dir /usr/lib/debug/remote/lib/modules/4.4.0-22-generic-real/kernel/drivers/net/bonding

Thanks for your help.  

[Denis_M_Intel]

 All the flags "sym, bin, all, r" are deprecated. They still works but that could be changed in future.
We will update documentation, sorry for the inconvenience.