VTune Jit profiling no information when attach to process, even with INTEL_LIBITTNOTIFY64 set

[ymao]

Hello,

I'm working on a C project with dynamic code generation at startup. Windows 11, VTune 2024.02.

We use jitprofiling.h & jitprofiling.lib to provide dynamic code information once jit finished, which works well when using VTune to launch the application. However, when we launch the application separately and attach VTune by process name or pid, VTune does not display dynamic code information (other code information such as C code are still valid).

Some trouble shooting talk about INTEL_LIBITTNOTIFY64 and INTEL_JIT_PROFILER64 (note that we do not use libittnotify in the C code, and linked to libittnotify.lib in addition does not help either). Here is my setup:

INTEL_LIBITTNOTIFY32=C:\Program Files (x86)\Intel\oneAPI\vtune\2024.2\bin32\runtime\ittnotify_collector.dll
INTEL_LIBITTNOTIFY64=C:\Program Files (x86)\Intel\oneAPI\vtune\2024.2\bin64\runtime\ittnotify_collector.dll
INTEL_JIT_PROFILER32=C:\Program Files (x86)\Intel\oneAPI\vtune\2024.2\bin32\runtime\ittnotify_collector.dll
INTEL_JIT_PROFILER64=C:\Program Files (x86)\Intel\oneAPI\vtune\2024.2\bin64\runtime\ittnotify_collector.dll

With these environment variables set (especially INTEL_JIT_PROFILER64), on attach to process, iJIT_IsProfilingActive retuns iJIT_SAMPLING_ON as expected. However, 

iJIT_NotifyEvent(iJVM_EVENT_TYPE_METHOD_LOAD_FINISHED,(void*)&data);

Does not return 0 (the value returned when we're in launch mode), but some value similar to an address. And VTune does not have dynamic code information (e.g. hotspots mode, bottom-up, no function listed for generated code, probably in [Outside any known module]).

It's very confusing and greatly impact what we're able to do with VTune, as most of the runtime code are dynamic generated. Some team members say it worked before - I don't know when, but VTune support code are added in 2023 in the project.

I would like to know if I'm missing something? Or is it a VTune bug? I can provide more information if needed.

[yuzhang3_intel]

Could you provide the reproducer? Including sample code and full environment setup, thanks.

[ymao]

Hello yuzhang3!

Thank you for your reply. Here is a sample setup, please do not hesitate to ask if you need any further information.

The C program: https://github.com/HaxeFoundation/hashlink. You can find VTune related code here: https://github.com/HaxeFoundation/hashlink/blob/31cee2a1ebfb11a1bc27a28ffa6b655dae88d138/src/module.c#L475-L534

Compile with hl.sln in the project, Visual Studio 2019, Release-x64, Windows 11. This sample only need libhl.dll and hl.exe.

Hashlink is a VM that read a bytecode and translate it into dynamic code, then run the dynamic code generated. I attached a bytecode generated in the following ways for your convenience, and here is how I generated it.

The bytecode is generated by Haxe: https://haxe.org/download/ (we used a nightly build which can be found here https://build.haxe.org/builds/haxe/windows64/haxe_2024-06-30_development_7b92f74.zip )

Here is a sample code Main.hx:

class Main {
    static public function main() {
        var i = 0;
        var j = 2;
        while (true) {
            i++;
            trace(i);
            for(a in 0...1000000000) {
                j = j * (j + i);
            }
        }
        trace("" + i + j);
    }
}

It can be build with haxe.exe -cp . -main Main -hl test.hl .

The only additional environment setup is set those environment variable for VTune (INTEL_JIT_PROFILER64 etc), in system environment variable (or in visual studio debugging environment and launch without debugging).

Once you have hashlink and bytecode, you can run hl.exe test.hl. 

If you launch it in VTune, you'll be able to see a function $Main::main and source file Main.hx, which is not the case when attach to process (only [Outside any known module]).

[yuzhang3_intel]

As you said, JIT profiling can work for launch mode, It would be nice if you could attach the associated VTune data. Thanks.

[ymao]

Sure, here is VTune data:

[yuzhang3_intel]

Can you double-confirm when you use VTune to attach the process, the JIT profiling doesn't happen? I mean, the collection should happen later.

Maybe you can add getchar() to pause the process, and then VTune attaches the process and finally continues it when one character is input.

[ymao]

I don't really understand your question, here is some possible related questions and my reply

• Is iJIT_IsProfilingActive / iJIT_NotifyEvent called before attach VTune to the program? Yes. All Jit code are generated and send to iJIT_NotifyEvent in the first seconds of launch and before jump into Main.hx:main(). When attaching VTune, I first launch hl.exe test.hl, and wait a few seconds until I see couples of print in main(), then I click begin attach to pid.
• Is the iJIT_NotifyEvent call after VTune attach? No. I don't know how to detect VTune attach event and the control flow is totally transferred to generated code after launch, though I can't handle this case.
• When attach VTune to the process, does I begin with paused mode? No, but it doesn't seems to make any difference and "Pause command is not supported for managed code profiling.".
• Am I sure that iJIT_IsProfilingActive / iJIT_NotifyEvent are called if I launch directly hl.exe? Yes. I made a print in the function to ensure this, that's also why I'm seeing difference in return value from iJIT_NotifyEvent.

[yuzhang3_intel]

Did you run the process firstly and then run VTune to profile the process? I mean you need to guarantee VTune can receive iJIT_NotifyEvent when profiling starts in attach mode.

[ymao]

Yes, I always

• first run the process and ensure that iJIT_IsProfilingActive / iJIT_NotifyEvent are called and I'm in main() of the generated code,
• (usually I wait for another 10 seconds the time I found the pid and fill in VTune)
• then I run VTune to attach to process.

[yuzhang3_intel]

So iJIT_IsProfilingActive / iJIT_NotifyEvent  called in the process happens after the VTune starts the profiling in attach mode, right?

[ymao]

No, iJIT_IsProfilingActive / iJIT_NotifyEvent are called in the process before the VTune starts the profiling in attach mode

[yuzhang3_intel]

iJIT_IsProfilingActive / iJIT_NotifyEvent  must be called in the process after VTune starts the profiling in attach mode, this can guarantee that VTune can receive the JIT event for JIT profiling.

(Linux as an example here)

Besides that, I suggest you set 4 environment variables as below to try.

export INTEL_LIBITTNOTIFY64=<vtune>/lib64/runtime/libittnotify_collector.so

export INTEL_LIBITTNOTIFY32=<vtune>/lib32/runtime/libittnotify_collector.so

export INTEL_JIT_PROFILER64=<vtune>/lib64/runtime/libittnotify_collector.so

export INTEL_JIT_PROFILER32=<vtune>/lib32/runtime/libittnotify_collector.so

[ymao]

Is there any way that I can detect VTune attach event in C code, so that I'll be able detect and call iJIT_IsProfilingActive / iJIT_NotifyEvent after VTune starts the profiling in attach mode?

Note: iJIT_IsProfilingActive return iJIT_SAMPLING_ON when INTEL_JIT_PROFILER64 is set, even if VTune is not launched. So it can't be used to poll VTune attach event.

Note2: Further more, the doc for iJIT_NotifyEvent(iJVM_EVENT_TYPE_METHOD_LOAD_FINISHED) says "Send this notification after a JITted method has been loaded into memory, and possibly JIT compiled, but before the code is executed. " As we build dynamic code at startup and transfer control flow to generated code, calling iJIT_IsProfilingActive / iJIT_NotifyEvent later will not respect this description.

For those environment variables, there is no <vtune>/lib64/runtime and I'm on Windows, so there is no .so either. I have set those variables as described in the original post, which points to for example <vtune>/bin64/runtime/ittnotify_collector.dll.

[yuzhang3_intel]

I think I get your point, let me check locally and then give you an update.

[ymao]

Ok, thank you!

[yuzhang3_intel]

After confirmation from engineering team, the issue is upgraded to one new feature request, and will be delivered in future release. Thanks for reporting the issue.

[ymao]

Thank you yuzhang3! Looking forward to the new feature!

[ymao]

Hello yuzhang3,

I'm trying to follow your previous suggestion

iJIT_IsProfilingActive / iJIT_NotifyEvent must be called in the process after VTune starts the profiling in attach mode, this can guarantee that VTune can receive the JIT event for JIT profiling.

I tested the following steps but it did not work as expected either:

1. Having environment variables INTEL_LIBITTNOTIFY64 etc

2. Start the program - generate dynamic code, and transfer the control flow to generated code

3. (wait some time of execution)

4. Attach VTune by pid and see time counter increasing

5. From running generated code, call native function iJIT_IsProfilingActive / iJIT_NotifyEvent. Observe that  iJIT_IsProfilingActive return false.

6. Stop collection in VTune, observe that generated code are not detected and are shown as [Outside of any known module]

Note: if VTune is not attached in the step 4, iJIT_IsProfilingActive return true in step 5.

Note: if launch with VTune in step 2 (instead of attach in step 4), VTune detect generated code correctly.

[yuzhang3_intel]

Could you call the iJIT_NotifyEvent API before step 3?

The JIT (Just-In-Time) Profiling API provides functionality to report information about just-in-time generated code that can be used by performance tools. You need to insert JIT Profiling API calls in the code generator to report information before JIT-compiled code goes to execution. This information is collected at runtime and used by tools like Intel® VTune™ Profiler to display performance metrics associated with JIT-compiled code.

iJVM_EVENT_TYPE_METHOD_LOAD_FINISHED

Send this notification after a JITted method has been loaded into memory, and possibly JIT compiled, but before the code is executed. Use the iJIT_Method_Load structure for EventSpecificData. The return value of iJIT_NotifyEvent is undefined.

https://www.intel.com/content/www/us/en/docs/vtune-profiler/user-guide/2023-0/ijit-notifyevent.html

[ymao]

Thank you for your reply! I tried as you said:

2.1 Start the program

2.2 Generate dynamic code

2.3 Pause the execution with a Sleep

2.4 Attach VTune by pid and see time counter increasing

2.5 Resume from Sleep, call iJIT_NotifyEvent API, typically with iJVM_EVENT_TYPE_METHOD_LOAD_FINISHED

2.6 Transfer the control flow to generated code

But in step 2.5, iJIT_IsProfilingActive returns false, iJIT_NotifyEvent returns 0 (in launch mode it returns 1), and VTune does not detect any generated code at the end.

[yuzhang3_intel]

From my test, the value of iJIT_IsProfilingActive returns correct value, once INTEL_JIT_PROFILER64 is set - it becomes active even without VTune. This means you can't use iJIT_IsProfilingActive as condition jumping.

Your jit code executes after step 2.5, right?