I am doing profiling of a Qt-based application in Windows10 using Intel VTune Amplifier 2019 (Update 8).
The problem is that the apllication stops to react and to respond when profiling is on.
The profiler is run paused and then resumed at the particular operation to be tested.
After resuming the application does not react and the operation stops at some stage (say 50%) and does not finish, but in Task Manager the application loads CPU quite extensively (i.e. something happens).
Without profiling, the operation goes well. The same problem happens under the same conditions but on another machine.
Could you recommend something what can help?
Thank you in advance.
Thank you for posting in Intel Forum.
Could you please share the reproducer code , so that we can try to reproduce the issue from our end. Also, let us know the type of analysis that you are trying to execute using VTune .
thank you very much for the feedback. Excuse me, I can't provide you the code... I can only say that it is a large program, and I test one of its multiple functionalities, which are tested daily. That's why the profiler is paused until I do all necessary actions before the particular operation beintg tested.
I perform the Hotspot analysis by collecting CPU sampling data with stacks and the 5ms interval. All the other options are set by default.
By summarizing, the operation works well without profiler. If it runs with Amplifier, it does not complete, stops at some point and does not respond. This does not depend on input file. All other known operations were tested well.
Maybe you can assume, why the profiler "blocks" the application and/or why there is a strange interaction? What can be the reason?
For example, I can speculate, it is due to the access permissions (hardly), memory (it should be enough), etc.
Could you tell me, if it is possible to see where the application stops by performing a kind of debugging while profiling? Maybe I could play with different options related to the hotspot analysis.
Thank you in adveance.
Thanks for the reply, no, I can't, I am applying what is provided.
Could you recommend some kind of analysis or a method to identify and find a location in the code where the profile gets stuck? How to find the reason?
Thanks for the response.
Make sure that VTune self check script is running fine to validate that appropriate drivers are installed and the system is set up properly to collect performance data.
On windows :
See whether its working fine and share the log.
Also, could you please attach the entire result folder so that we can debug the issue correctly.
Thank you very much for your reply and for the advice.
Unfortunately I can't find the file, but the idea is nice, it can show some diagnostic information.
Actually the file is not a problem, because it is just a command call, as far as I understand.
Could you tell me where I can download this file or what the command does check?
As for data of the performance results, I need to clarify this question.
The self-check file for windows “vtune-self-checker.bat” file will be there in the <vtune-installation-directory>/bin64 or <vtune-installation-directory>/bin32 folder if the VTune installation was done successfully. Or check for “amplxe-self-checker.bat “ .It is not possible to explicitly download the same.
· A self-check script is available to validate that appropriate drivers are installed and the system is set up properly to collect performance data
· The script runs several analysis types on a sample. After the script completes, it produces a log file and gives diagnostics on the success or failure of the checks.
· Finds possible collection limitations, missing permissions or outdated drivers and lists steps to overcome the limitations.
If you couldn’t find the self-checker script , then try profiling sample matrix app in the folder <vtune-installation-directory>\samples\en\C++\matrix and see whether you are able to get the results in VTune successfully.
Also, Please share the following details for better understanding of the issue.
a) After configuring the analysis in GUI , use Command Line Generator ( bottom right corner button) to generate CLI for the configuration you have made and share the same.( Refer attached Image1)
b) The result directory generated or the screenshot of collection log.
Thank you very much for the information and for the advice.
Excuse me for the delayed reply, it took some time to gether something to be reported, although I can't provide for now all the extensive information you required.
Meanwhile, the second operation was found with a similar effect while performing profile: the operation does not complete and the application stops to respond after some step while profiling. And this is very strange and a kind of restriction, the operation is successfully performed without profiling.
Nevertheless, the amplifier is an important tool, and I will try to complete carefully what you suggested to be sure that it works correct.
For now I can say definitely that I do not have the bat-files you wrote about: neither vtune-self-checker.bat, nor amplxe-self-checker.bat.
Could you provide the content of the file which you have as an example so that I have at least an overview what is done there? This is just a script, it should not be difficult.
Thank you for your patience. We installed the exact version of VTune you are using and tried to profile sample matrix application and we got the result successfully. We observed that the self-check file that we were talking about is not present in VTune 2019 update 8 folder and we apologize for asking you to check the same.
Could you please try upgrading to a latest version or we might need reproducer code or result folder for further debugging as we were able to profile samples.
Excuse me for the delayed answer, I had to collect some info which can be useful in searching a solution.
First of all, I found one notification which appears when I change the type of sampling in the hotspot analysis.
There are two types:
(1) user-mode smapling;
(2) hardware event-based sampling.
I usually apply type 1, i.e. user-mode, but then I tried type 2, hardware event-based, and I got a notification:
"This analysis type is not applicable to the system because VTune Amplifier cannot recognize the processor. If this is a new Intel processor, please check for an updated version of VTune Amplifier. If this is an unreleased Intel processor, please contact Online Service Center for an NDA product package. This analysis type is not applicable to the current machine microarchitecture."
The processor of the PC where profiling is done: Intel(R) Core(TM) i9-10900 CPU @ 2.80GHz 2.81 GHz
At that time a new version of Amplifier, namely Intel oneAPI VTune Profiler 2021.1.2 Patch 1 (the old one I used before was Amplifier 2019 as I mentioned at the very beginning).
With this new version I do not have the above notification about the processor recognition, but I still have similar problems. For some operations being profiled, the application stops to respond while profiling (albeit everything is done successfully without profiling). It does something, the CPU is loaded, but the only one way to finish the app is to kill it from the task maneger. Note, profiling can be still successful after several tries, e.g. after ten times, which is not very convenient. That's why I would like to solve it.
Could you tell me, how I can determine, if the processor model does not cause any problem?
Could you tell me, how I can check the Amplifier in general?
For example, people write about the sep/amplxe-runss commands.
Could it be helpful?
This is the output of Amplifier2021 you requested earlier:
"<mypath>\bin64\vtune" -collect-with runss -knob cpu-samples-mode=stack -knob sampling-interval=5 -knob enable-user-tasks=true -knob mrte-type=java,dotnet,python -app-working-dir <myworkingdir> -- <myexe>
The same options are applied in Amplifier2019 with the following output (the execution is only different):
"<mypath>\bin64\amplxe-cl" -collect-with runss -knob cpu-samples-mode=stack -knob sampling-interval=5 -knob enable-user-tasks=true -knob mrte-type=java,dotnet,python -app-working-dir <myworkingdir> -- <myexe>
Thank you for the details. The issue with hardware event based sampling might be due to the VTune version. Glad to know that the issue got resolved when you upgraded it to the latest version .It would be great if you could stick with the latest version for further debugging and profiling.
As mentioned earlier, we have self-check script to validate the performance of VTune Profiler.In windows “vtune-self-checker.bat” file will be there in the <vtune-installation-directory>/bin64 or <vtune-installation-directory>/bin32 folder if the installation is successfull. Try running that and check whether any errors are there in the logs.If the self check script gets executed successfully then the issue might be application specific.
We checked the command you provided and tried running the sample matrix application ( <vtune-installation-directory>/samples/en/C++/matrix/matrix.exe ).
vtune -collect-with runss -knob cpu-samples-mode=stack -knob sampling-interval=5 -knob enable-user-tasks=true -knob mrte-type=java,python -app-working-dir <myworkingdir> -- <myexe>
Version : VTune 2021.1
Processor : Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz 1.90 GHz
We couldn’t observe any issue while executing. In the earlier post you have mentioned that there is similar issue with matrix application also.
“Meanwhile, the second operation was found with a similar effect while performing profile: the operation does not complete and the application stops to respond after some step while profiling. And this is very strange and a kind of restriction, the operation is successfully performed without profiling”
Could you please verify that you profiled the sample matrix application available with VTune profiler. Also, try profiling it using same configuration in the latest version of VTune that you installed and let us know the updates. Meanwhile we will check with subject matter experts whether the issue is related to the processor you are using.
Thank you very much for your prompt reply and for the information.
There are some benefits of using the new version of Amplifier, Intel VTune 2021, but overall I still have the issues.
One notable thing is that I do not see the warning when I switch the hotspot analysis type from User-Mode Sampling to Hardware Event-Based Sampling. In fact, the results produced by Hardware Event-Based Sampling are strange and hard to be analysed. Thus, the User-Mode Sampling is mostly used, but even the new version of Amplifier still fails from time to time when I do hotspot analysis of some operations.
Another benefit of the new Amplifier version is that it contains the script vtune-self-checker.bat. I run the script, I get a log-file with some system information and I can send you the generated log-file.
Could you tell, how I can send you the file? It would nice to be able to send privately without posting on the forum...
Another thing I would like to clarify: I did not do profiling of the matrix application, because I have enough tasks for profiling, and I have results. But in general, I can do profiling of that application.
Could you explain,if it can help and what it can show?
I can briefly describe: there is a large application being developed, and it has multiple different functions. Only separate operations, which are modified, are checked by using Amplifier. Mostly profiling is successful, say, in 80-90% cases I get the result, but from time to time I cannot profile, because the program being profiled stops to respond. And I would like to know, why this happens.
We will connect with you through email and you can attach the log file in response. Basically, the self check script runs several analysis types on the matrix sample and we can infer from the logs whether there are any errors or not. If the sample application is profiled successfully then we can assume that the issue is application specific. I request you to share the self-check script logs via email for further understanding of the actual issue.
Thanks in advance,
Thank you for sharing the logs. From the logs everything looks good. We would assume that this is application specific and not a tool issue.
For debugging the issue, could you please use the analysis type "Anomaly detection" with “Processor Trace” and call stacks. Run the application and wait for it to hang-up. Stop profiling from VTune and the call stack should point to the region where it hung up. Let us know the updates.
Thank you very much for the valuable information.
I have tried to debug several examples using the settings you recommended, but it does not help so much, although I am not sure, I have done everything correct.
I am attaching the sceenshots of the settings I used, and the red shows those parameters which I changed, the red circle near "Use Intel Processor Trace to analyze transactional regions" I switch on and off.
Could you tell me, if these settings are correct and are the same what you recommended? And what would you recommend additionally?
The problem is that in the debug mode the profiling is successfull, the operation being profiled completes, but the profile is non-typical, it does not allow to go through the hierarchy of calls in Top-dows tree. I would call it "plain". And because it finishes successfully, I do not know how I can find the place where it stops. Maybe I am doing it not correct.
Thank you for waiting. We are checking some of the things internally that would help you to debug the problem. Give us some time and we will get back to you soon.
With latest 2021.1.2 vtune version, is your user based hotspot analysis successful?
The reason for earlier warning on the unidentified processor is because the 2019 VTune does not support the machine you have. With 2021.1.2 - VTune is able t detect machine for H/W event based analysis.
How much memory you have on the system? while observing on the task manager, do you see memory reaching the upper limit while profiling is on?