Instrumentation for call graph takes too long when working in RDC model

avicheckpoint_com · ‎12-21-2008

Hello,

I have a VTune Analyzer (V. 9.0u11) installed on a Windows machine, and VTune agent (of same version) installed on a remote Linux machine. Collecting sampling information works fine. However, when trying to collect call graph information, instrumentation is extremely slow. It takes about 3 minutes toinstrument everyshared library(as appears in the /tmp/vtune_/Cache directory), and 3 more minutes to have the appropriate message appearing in the Analyzer. Thus, having more then 10shared objects for the executable,call graph collection is impractical.

Apparently the problem is not with the VTune agent running on the Linux machine but rather with the Analyzer. Using the same VTune agent on the same machine, collecting call graph information from an Analyzer (of same version) installed on another Windows machine, runs smoothly. Instrumentation of all binaries takes a few seconds.

It seems, therefore, that there is a configuration/installation issue on my machine. It would be worthwhile to mention that the Analyzer used to work properly. For some reason I face this problem recently. Uninstalling and re-installing the Analyzer, cleaning the projects folder and the cache directories had no effect.

I would be thankful if you could advice regarding this issue.

Regards,

Avi

Peter_W_Intel · ‎12-21-2008

Hi,

I may not understand enough clear for "Apparently the problem is not with the VTune agent running on the Linux machine but rather with the Analyzer. Using the same VTune agent on the same machine, collecting call graph information from an Analyzer (of same version) installed on another Windows machine, runs smoothly. Instrumentation of all binaries takes a few seconds." Did you mean that other Windows host with same VTune Linux agent has no this problem?

VTune Linux Remote agent will instrument associated all system libraries and your user's libraries, and send them to your $cache-directory-remote (default is "/tmp/vtune_user/Cache") in host machine. Probably you can change instrumentation level to reduce time.

1) Use vtl command in command prompt, use "vtl global-options" to know current settings, suggest to do

a) vtl global-options system-library-instrimentation-level="minimal"

b) vtl global-options user-library-instrumentation-level="exports"

-OR-

2) You can adjust(change) similar configurations on "Configure Call graphAdvance" dialog.

Regards, Peter

avicheckpoint_com · ‎12-24-2008

Hi Peter,

Thanks for your reply.

Indeed, the other Windows host working with the same VTune agent running on the very same Linux machine has no problem.

Instrumented files created when activity is initiated from my machine are in /tmp/vtune_myuser/Cache while instrumented files created when activity is initiated from the other machine are in /tmp/vtune_otheruser/Cache. The second Cache is being filled quite quickly for the same task.

We also compared global options for both Analyzers to realize same settings on both. In both executable-instrumentation-level=all-functions, system-library-instrumentation-level=exports, user-library-instrumentation-level=all-functions. Indeed, this can be reduced to minimal,minimal,export respecitivy, however in the other desktop the same settings yields reasonable performance.

It is wierd, since the degradation when running the activity from my Windows machine is experienced in the agent as well, not only in the desktop. The cache in /tmp/vtune_myuser/Cache is being filled extremely slowly (as said about 3 minutes per .so file). On the same machine /tmp/vtune_otheruser/Cache is being filled quickly (a few seconds for all files) when activity is run from the other Windows machine.

Setting cache-directory-remote to /tmp/vtune_otheruser/Cache may be a temporary workaround. However, this is not useful as a guideline I canoffer to all VTune users in my organization.

As said before, uninstalling, cleaning, re-installing the Analyzer on my machine had no effect.

Please advice.

Thanks,

Avi

Peter_W_Intel · ‎12-24-2008

Hi Avi,

Please ignore the instrumentation level changing idea if it doesn't meet your request.

I'm just curious to alternate the "Cache directory on remote machine" setting:

1. Can you please verify if you change to "/tmp/vtune_otheruser/Cache" on slow Windows* machine?

2. Canyou please verify if you change to "/tmp/vtune_myuser/Cache" on quick Windows* machine?

What happens?I want toidentfy the problem was caused by Windows client? or on Linux?

Thanks, Peter

avicheckpoint_com · ‎01-04-2009

Quoting - Zhen Yu Wang (Intel)

Hi Avi,

Please ignore the instrumentation level changing idea if it doesn't meet your request.

I'm just curious to alternate the "Cache directory on remote machine" setting:

1. Can you please verify if you change to "/tmp/vtune_otheruser/Cache" on slow Windows* machine?

2. Canyou please verify if you change to "/tmp/vtune_myuser/Cache" on quick Windows* machine?

What happens?I want toidentfy the problem was caused by Windows client? or on Linux?

Thanks, Peter

Hi Peter,

Theother machine was not available for a while, therefore the delay in my posting.

I followed your suggestion #1 and changed the Cache onmy machine to /tmp/vtune_otheruser/Cache. I run the activity. Instrumentation time was indeed faster then before, but still took an extremely long time. Follows an excerpt from VTune output:

Sun Jan 04 13:36:33 2009 Static instrumentation started
Sun Jan 04 13:39:16 2009 Minimal instrumentation of module "my_lightweight_exe" was skipped.
Sun Jan 04 13:39:37 2009 Exports instrumentation of module "libxxx.so" was skipped.
...
Sun Jan 04 14:00:07 2009 Exports instrumentation of module "libstdc++.so.5.0.3" was skipped.
Sun Jan 04 14:03:11 2009 Static instrumentation done
Sun Jan 04 14:09:05 2009 Data collection started...
Sun Jan 04 14:09:09 2009 Process Call Graph (MT) Data Collection, id "17459", started writing Call Graph data.
Sun Jan 04 14:09:30 2009 Process Call Graph (MT) Data Collection, id "17459", finished writing Call Graph data.

As you can see the "skipping instrumentation" phase took about half an hour. Data collection started 6 minutes later. Now it is 14:23 and I'm still waiting to the call graph results to be drawn in the Analyzer window.

(On the other machine, Instrumentation + data collection + drawing the call graph results took 10 seconds.)

Please advice.

Avi

avicheckpoint_com · ‎01-11-2009

Hi,

Any new ideas?

Maybe something related to floating license? I have verified in the license serverthat one license (out fo the floating licenses pool) is used by my machine during run. But yet, I suspect whether license may be an issue here.

What else can explain this weird behavior? Have you got any idea how to fix this?

Thanks,

Avi

Peter_W_Intel · ‎01-11-2009

Quoting - avicheckpoint.com

Hi Peter,

Theother machine was not available for a while, therefore the delay in my posting.

I followed your suggestion #1 and changed the Cache onmy machine to /tmp/vtune_otheruser/Cache. I run the activity. Instrumentation time was indeed faster then before, but still took an extremely long time. Follows an excerpt from VTune output:

Sun Jan 04 13:36:33 2009 Static instrumentation started
Sun Jan 04 13:39:16 2009 Minimal instrumentation of module "my_lightweight_exe" was skipped.
Sun Jan 04 13:39:37 2009 Exports instrumentation of module "libxxx.so" was skipped.
...
Sun Jan 04 14:00:07 2009 Exports instrumentation of module "libstdc++.so.5.0.3" was skipped.
Sun Jan 04 14:03:11 2009 Static instrumentation done
Sun Jan 04 14:09:05 2009 Data collection started...
Sun Jan 04 14:09:09 2009 Process Call Graph (MT) Data Collection, id "17459", started writing Call Graph data.
Sun Jan 04 14:09:30 2009 Process Call Graph (MT) Data Collection, id "17459", finished writing Call Graph data.

As you can see the "skipping instrumentation" phase took about half an hour. Data collection started 6 minutes later. Now it is 14:23 and I'm still waiting to the call graph results to be drawn in the Analyzer window.

(On the other machine, Instrumentation + data collection + drawing the call graph results took 10 seconds.)

Please advice.

Avi

Avi,

The problem is so complicated. But based on your data - the problem was caused by your Windows* host.

You said un-installing/re-installing the product is not effective. If you doubt it was caused by frequent floating-license checking, much time spent during "module's instrumentation skip", 1 hour? You can try single 30 days evaluation license file to compare -https://registrationcenter.intel.com/RegCenter/AutoGen.aspx?ProductID=1249&AccountID=&EmailID=&ProgramID=&RequestDt=&rm=EVAL〈=

Re-install the product with trial license file, then compare the result. If this is the reason, you may re-install Intel License Manager for FLEXlm*?

Regards, Peter