Seems ITT task event in user application could not be tracking in "profile system" mode

[yidu]

Hi,

I'm trying to optimize my application's performance on a FreeBSD system. I've installed VTune 2024.2.0 on my Ubuntu laptop and I'm using remote SSH to analyze the performance. I successfully performed hotspot analysis using hardware event-based sampling.

I created a sample application with ITT API enabled, deployed it to a remote FreeBSD machine, and added the INTEL_LIBITTNOTIFY64 environment variable in the startup script. When I use the "Launch Application" feature with this startup script, everything works fine, and I can see the user task events in the results. However, when I use the "Profile System" feature to instrument and manually run the same script, I can't see these user events at all.

Is this behavior by design, or is there something wrong with my environment?

[yuzhang3_intel]

I integrated your sampling code into a main.c file,  and executed the main binary in one console window. When the binary stopped, I got the process ID and specified it to VTune command line in another console, and then ran the VTune for profiling, input any key in console window 1 to continue the target process. All works fine. 

From the running steps, it looks like you ran one script instead of one binary, as I did.  The process ID you specify to VTune command line is the same process inside which the __itt  API is called, right?

[yidu]

Yes, you are right.

The process ID I provided is correct; if it wasn't, I wouldn't be able to press the run button in the VTune UI.

My application is running under FreeBSD's Linux binary compatibility mode. That is said, The application is a Linux binary, and the INTEL_LIBITTNOTIFY64 environment variable references a Linux library also, but it's running on FreeBSD OS.

To Running the application directly works fine, but attaching the process fails with same script and application and same references library, I don't think this is the cause.

Would you please double-check for me,  if ITT can worked with in remote Linux OS and with "process attach" mode?

thanks a lots

[yuzhang3_intel]

Yes, we support both FreeBSD lunch/attach modes for ITTAPI collection.

Can you provide me logs, result for this problematic collection?

Also, it would be nice to enable additional log collection by setting the following environment variables:

export USERAPICOLLECTOR_DEBUG=1

export USERAPICOLLECTOR_LOG_LEVEL=TRACE

[yidu]

Hi @yuzhang3_intel ,

Sure, Should I export these environment at remote target (freebsd) during collect process?

And how to get these logs out.

please make some instruction in detail.

thanks

[yuzhang3_intel]

First, set the two environment variables. Secondly, Run VTune on your host to remotely attach one process on the target device as you did before. When the process is done, there are log files stored in the folder of  /tmp/amplxe-log-xxx, please share the log files and VTune data as well. 

[yidu]

Hi @yuzhang3_intel,

here is the logs, use proc ID attach ways as before.

thanks a lots.

[yuzhang3_intel]

Could you also share the amplxe log file stored in the path of /tmp ?

[yidu]

Hi @yuzhang3_intel,

Found these log folder in my remote machine, but seems most of them is empty.

thanks.

[yuzhang3_intel]

From the log, it looks like the env vars are not set correctly.

perfrun.itt <> - attach itt controller to pid #52582 failed, 

Please check again environment variable. It should be set on target machine from the same terminal were the application running and this path should be valid and point to the ittnotify-collector.so lib from the installed VTune target package

[yidu]

Hi @yuzhang3_intel ,

I thinks root cause should be,

My ittmain is Linux application but running at Freebsd with linux compat support.

So in my launch script, the INTEL_LIBITTNOTIFY64 referenced ittnotify-collector.so is linux based not for freebsd.

ittnotify-collector.so in linux version should to check /proc/self/exe to get application process stuff. but there is no /proc/self/exe in our freebsd system. it must be failed.

For launch application case, it should avoid these kinds of code at all .

see these logs.

187 [0x37f48213500] WARN perfrun.vdso <> - can not resolve /proc/self/exe: No such file or directory, at file: vcs/perfrun1/plugins/vdso/src/vdso_collector.cpp:128
188 [0x37f48213500] WARN perfrun.vdso <> - can not resolve /proc/self/exe: No such file or directory, at file: vcs/perfrun1/plugins/vdso/src/vdso_collector.cpp:128
190 [0x37f48213500] WARN perfrun.itt <> - attach itt controller to pid #88753failed, probably not configured, at file: vcs/perfrun1/plugins/itt/src/controller.cpp:258

thanks

[yuzhang3_intel]

In VTune installation folder, you can get target package on freebsd, which contains ITT associated libraries. You can use them on target device.

yuzhang3@yuzhang3-10710:/opt/intel/oneapi/vtune/latest/target/freebsd$ tree -a
.
├── vtune_profiler_target_sep_x86_64.tgz
└── vtune_profiler_target_x86_64.tgz

[yidu]

Hi @yuzhang3_intel 

Update INTEL_LIBITTNOTIFY64 to freebsd version, problem existed.

From logs of application , the INTEL_LIBITTNOTIFY64 is updated to freebsd version.

go env INTEL_LIBITTNOTIFY64: /home/yidu/vtune/lib64/runtime/libittnotify_collector.so
Main task begins
collector state 0
Creating worker thread 0
Creating worker thread 1
Creating worker thread 2

But still have these logs.

185 [0x3f1a00413500] WARN perfrun.vdso <> - can not resolve /proc/self/exe: No such file or directory, at file: vcs/perfrun1/plugins/vdso/src/vdso_collector.cpp:128
187 [0x3f1a00413500] WARN perfrun.vdso <> - can not resolve /proc/self/exe: No such file or directory, at file: vcs/perfrun1/plugins/vdso/src/vdso_collector.cpp:128
188 [0x3f1a00413500] WARN perfrun.itt <> - attach itt controller to pid #22730failed, probably not configured, at file: vcs/perfrun1/plugins/itt/src/controller.cpp:258

thanks

[yuzhang3_intel]

Thanks for replying quickly, we need to do a further investigation into the issue.

[yuzhang3_intel]

On FreeBSD 14.0-RELEASE-p3 with VTune Profiler 2024.2.0.628218,
I was able to successfully get ITT user tasks in the following scenarios
(verified with vtune -report summary r@@@hs)

export INTEL_LIBITTNOTIFY64=/opt/intel/vtune_profiler_target/lib64/runtime/libittnotify_collector.so

1) export INTEL_LIBITTNOTIFY64=...; /opt/intel/vtune_profiler_target/bin64/vtune -collect hotspots -knob sampling-mode=hw -knob enable-stack-collection=true – app

2) export INTEL_LIBITTNOTIFY64=...; /opt/intel/vtune_profiler_target/bin64/vtune -collect hotspots -knob sampling-mode=hw -knob enable-stack-collection=true -analyze-system – app

3) export INTEL_LIBITTNOTIFY64=...; start application in background; /opt/intel/vtune_profiler_target/bin64/vtune -collect hotspots -knob sampling-mode=hw -knob enable-stack-collection=true -target-pid=<PID> -analyze-system

Is it possible that you can't set INTEL_LIBITTNOTIFY64 before starting the instrumented application (i.e., only setting it before starting the vtune collection)?

If I repeat #3 and #4 without setting INTEL_LIBITTNOTIFY64 before starting the application, and setting it before starting the vtune collection, I do not see ITT API user tasks, as expected.

[yidu]

Hi @yuzhang3_intel ,

Seems we need export INTEL_LIBITTNOTIFY64 into /opt/intel/vtune_profiler_target/bin64/vtune's process, when it begin the process on final collection..

Sure, I will try these solution and feedback later.

thanks a lots.

[yuzhang3_intel]

Yes, you must export INTEL_LIBITTNOTIFY64  before running the application. Below is the command line and result.

root@renge-FreeBSD:/opt/intel/bin64 # ./vtune -collect hotspots -knob sampling-mode=hw --target-pid 1733 -analyze-system

root@renge-FreeBSD:/opt/intel/bin64 # ./vtune -R summary ,/r001hs
vtune: Using result path `/opt/intel/bin64/r001hs'
vtune: Executing actions 75 % Generating a report Elapsed Time: 171.993s
CPU Time: 1.294s
Effective Time: 1.294s
Spin Time: 0s
Overhead Time: 0s
Instructions Retired: 406,800,000
Microarchitecture Usage: 13.8% of Pipeline Slots
| You code efficiency on this platform is too low.
|
| Possible cause: memory stalls, instruction starvation, branch
| misprediction or long latency instructions.
|
| Next steps: Run Microarchitecture Exploration analysis to identify the
| cause of the low microarchitecture usage efficiency.
|
CPI Rate: 3.654
| The CPI may be too high. This could be caused by issues such as
| memory stalls, instruction starvation, branch misprediction or long
| latency instructions. Explore the other hardware-related metrics to
| identify what is causing high CPI.
|
Total Thread Count: 45
Paused Time: 0s

Top Hotspots
Function Module CPU Time % of CPU Time(%)
------------------- ------ -------- ----------------
acpi_cpu_idle_mwait kernel 0.793s 61.3%
kbdc_data_ready kernel 0.024s 1.9%
rdtsc kernel 0.020s 1.5%
cpu_search kernel 0.019s 1.5%
cpu_search kernel 0.017s 1.3%
[Others] N/A 0.421s 32.5%

Top Tasks
Task Type Task Time Task Count Average Task Time
------------ --------- ---------- -----------------
work 39.805s 250 0.159s
main 10.034s 1 10.034s
CreateThread 0.000s 4 0.000s

[yidu]

Hi @yuzhang3_intel,

I build another ittmain in pure freebsd version.

it's worked fine under attach process ID scenario under vtune-gui control.

and I also try it under your suggestions ways.

#1 #2 #3 worked as expected.  but #4 still could not get Task information.

For deeper investigation, I found it failed to connect domain socket provide by server side. it should be by design as your previous information. 

connect(4,{ AF_UNIX "/tmp/_server_ittcol_helper1619" },106) ERR#2 'No such file or directory'

for my scenario, the linux application do have limitation on itt api. 

thanks.

[yuzhang3_intel · Accepted solution]

Please ignore the #4. I hope the ITT on FreeBSD support can be helpful to you.

[yidu]

Hi @yuzhang3_intel ,

Thanks a lot for your help on this issue.