Error in Intel VTune Amplifier XE 2013

[danielsue]

Hi All,

I have build a parallel program and it works well. But when I run VTune hotspots Analysis, I got the following error

OMP: Error #136: Cannot create thread.
OMP: System error #1455: The paging file is too small for this operation to complete.
OMP: Error #178: Function GetExitCodeThread() failed:
OMP: System error #6: The handle is invalid.

OS: WIN7 X64 Pro

Compiler: Intel Parallel Studio XE2013 (Fortran)

CPU: Intel xeon E5

Memory: 16GB

I have checked automatically manage paging file size for all drives and the current allocated paging file is 16GB. Running my test only cost about 200MB, there should be enough space for both paging file and physical memory. Anything wrong with my setting?

Thanks and regards,

Daniel

[SergeyKostrov]

>>...I have checked automatically manage paging file size for all drives and the current allocated paging file is 16GB. Running >>my test only cost about 200MB, there should be enough space for both paging file and physical memory. Anything wrong >>with my setting? Try to use 'Custom Size' Virtual Memory ( VM ) instead of 'System managed size' VM. Also, I reported a similar problem about 2 months ago and I don't think that Intel software engineers considered it seriously. Also, similar problems were reported in the past by many software developers. Could you create a new thread with the description of the problem on Intel Visual Fortran compiler forum?

[Bernard]

Can you post full log file?

[SergeyKostrov]

>>...Can you post full log file? It is not clear what log file do you want to see and I'm not sure that Daniel has any log files ( text of errors provided in the initial post of the thread self-descriptive ).

[Peter_W_Intel]

Maybe your can try - (change sample interval from 1 to 10, don't collect stack info, short collection time : reduce extra-cost from VTune Amplifier XE)

"amplxe-cl -collect-with runss -knob sampling-interval=10 -knob cpu-samples-mode=nostack -duration 30 -- application.exe"

Is there same problem when using lightweight-hotspots?

Is it possible that you can attached test case that I can run on Windows* 7 on my side? It should be very helpful dig-out root-cause.

[danielsue]

Sergey Kostrov wrote:

>>...I have checked automatically manage paging file size for all drives and the current allocated paging file is 16GB. Running
>>my test only cost about 200MB, there should be enough space for both paging file and physical memory. Anything wrong
>>with my setting?

Try to use 'Custom Size' Virtual Memory ( VM ) instead of 'System managed size' VM.

Also, I reported a similar problem about 2 months ago and I don't think that Intel software engineers considered it seriously. Also, similar problems were reported in the past by many software developers.

Could you create a new thread with the description of the problem on Intel Visual Fortran compiler forum?

Hi Sergey,

Thanks for your quick response. I will try your suggestion later and get it back. Currently I can not do it as my computer is now running other task that I don't want to restart the system as the change of VM. The error log file is as follows:

OMP: Error #136: Cannot create thread.
OMP: System error #1455: The paging file is too small for this operation to complete.
OMP: Error #178: Function GetExitCodeThread() failed:
OMP: System error #6: The handle is invalid.

Thanks again,

Daniel

[danielsue]

Hi Peter,

The sample interval for my previous test is 10. I have also tried to use "without stacks", the error is the same, as follows:

OMP: Error #136: Cannot create thread.
OMP: System error #1455: The paging file is too small for this operation to complete.
OMP: Error #178: Function GetExitCodeThread() failed:
OMP: System error #6: The handle is invalid.

Thanks,

Daniel

Peter Wang (Intel) wrote:

Maybe your can try - (change sample interval from 1 to 10, don't collect stack info, short collection time : reduce extra-cost from VTune Amplifier XE)

"amplxe-cl -collect-with runss -knob sampling-interval=10 -knob cpu-samples-mode=nostack -duration 30 -- application.exe"

Is there same problem when using lightweight-hotspots?

Is it possible that you can attached test case that I can run on Windows* 7 on my side? It should be very helpful dig-out root-cause.

[Peter_W_Intel]

Is there same problem when using lightweight-hotspots?

[Bernard]

Strange error. Adi Levin's blog gives an short explanation about the CreatThread failure and this is related to 32-bit system with huge amount of threads or large memory consumption which is not your case.

[danielsue]

Hi Sergey,

It cannot work when use 'Custom Size' Virtual Memory ( VM ) instead of 'System managed size' VM.The test crashes with the following output:

The database has been cleared, elapsed time is 0.257 seconds.
Raw data has been loaded to the database, elapsed time is 0.142 seconds.
Data transformations have been finished, elapsed time is 0.000 seconds.
Warning: Cannot locate symbols for file `C:/Windows/system32/kernel32.dll'.
Warning: Cannot locate symbols for file `C:/Windows/SYSTEM32/ntdll.dll'.
Warning: Cannot locate symbols for file `C:/Windows/system32/KERNELBASE.dll'.
Warning: Cannot locate symbols for file `C:/Program Files (x86)/Intel/VTune Amplifier XE 2013/bin64/tpsstool.dll'.
Symbol resolution has been finished, elapsed time is 3.012 seconds.
Deferred data transformations have been finished, elapsed time is 0.031 seconds.
Precomputing frequently used data has been finished, elapsed time is 0.052 seconds.
Data model parameters have been set, elapsed time is 0.048 seconds.
Redundant overtime data has been discarded, elapsed time is 0.008 seconds.
Raw collector data has been discarded, elapsed time is 0.000 seconds.
Finalizing the result took 4.113 seconds.
Error: Unsupported item 'helpKeyword' for query 'vectorQuery(id='')[]'
Error: Unsupported item 'helpKeywordCalcPolicy' for query 'vectorQuery(id='')[]'
Error: Unsupported item 'helpKeyword' for query 'vectorQuery(id='')[]'
Error: Unsupported item 'helpKeywordCalcPolicy' for query 'vectorQuery(id='')[]'

Thanks,

Daniel

Sergey Kostrov wrote:

>>...I have checked automatically manage paging file size for all drives and the current allocated paging file is 16GB. Running
>>my test only cost about 200MB, there should be enough space for both paging file and physical memory. Anything wrong
>>with my setting?

Try to use 'Custom Size' Virtual Memory ( VM ) instead of 'System managed size' VM.

Also, I reported a similar problem about 2 months ago and I don't think that Intel software engineers considered it seriously. Also, similar problems were reported in the past by many software developers.

Could you create a new thread with the description of the problem on Intel Visual Fortran compiler forum?

[Bernard]

Hi danielsue,

there are available more advanced troubleshooting methods(like running VTune under debugger or putting a breakpoints on thread creation and memory allocation routines),but I think those methods should be employed by VTune developers to fix the problem.

[danielsue]

Hi iliyapolak,

I ran the VTune under debugger mode and got the above-mentioned errors. I guess the something is wrong in the setting or there is some bug in the tool. The testing cases range from a very small one (only 121 non-zero elements in matrix) to a very large one (1 billion non-zero elements in matrix), but they all stopped with the same error. Exactly there cases run well without VTune.

Thanks,

Daniel

iliyapolak wrote:

Hi danielsue,

there are available more advanced troubleshooting methods(like running VTune under debugger or putting a breakpoints on thread creation and memory allocation routines),but I think those methods should be employed by VTune developers to fix the problem.

[Bernard]

>>>I ran the VTune under debugger mode and got the above-mentioned errors>>>

Did you run VTune under windbg? It is not clear from your post.

[danielsue]

Yes, I ran VTune under debug mode in Visual Studio.Net. I also tried to run it in release mode, it produce the same error.

[SergeyKostrov]

>>...It cannot work when use 'Custom Size' Virtual Memory ( VM ) instead of 'System managed size' VM... I'd like to confirm that you've tried to change it. These four errors: >>... >>Error: Unsupported item 'helpKeyword' for query 'vectorQuery(id='')[]' >>Error: Unsupported item 'helpKeywordCalcPolicy' for query 'vectorQuery(id='')[]' >>Error: Unsupported item 'helpKeyword' for query 'vectorQuery(id='')[]' >>Error: Unsupported item 'helpKeywordCalcPolicy' for query 'vectorQuery(id='')[]' >>... are indication that something is wrong with internal SQLite database, or with SQL queries, which used by VTune. Did yiu try to install the most recent VTune Update?

[danielsue]

The VTune in my computer was not the latest, but it was updated two weeks ago. The most recent VTune, as can be seen from the intel software manager, was published on May-16-13. I will try to update it to see how it goes.

Sergey Kostrov wrote:

>>...It cannot work when use 'Custom Size' Virtual Memory ( VM ) instead of 'System managed size' VM...

I'd like to confirm that you've tried to change it.

These four errors:

>>...
>>Error: Unsupported item 'helpKeyword' for query 'vectorQuery(id='')[]'
>>Error: Unsupported item 'helpKeywordCalcPolicy' for query 'vectorQuery(id='')[]'
>>Error: Unsupported item 'helpKeyword' for query 'vectorQuery(id='')[]'
>>Error: Unsupported item 'helpKeywordCalcPolicy' for query 'vectorQuery(id='')[]'
>>...

are indication that something is wrong with internal SQLite database, or with SQL queries, which used by VTune. Did yiu try to install the most recent VTune Update?

[Peter_W_Intel]

I still don't know if there is problem of using lightweight-hotspots instead of hotspots, I posted last time.

Another consideration is to try VTune standalone (not in Visual Studio), it may save resource...

Was it possible that you ran 32bit application under 64bit OS? If it is the case, there is 2GB memory limitation. 

[danielsue]

After updating VTune to the latest version, the program still failed and I got the following information:

The database has been cleared, elapsed time is 0.294 seconds.
Raw data has been loaded to the database, elapsed time is 0.172 seconds.
Data transformations have been finished, elapsed time is 0.000 seconds.
Warning: Cannot locate symbols for file `C:/Windows/system32/kernel32.dll'.
Warning: Cannot locate symbols for file `C:/Windows/SYSTEM32/ntdll.dll'.
Warning: Cannot locate symbols for file `C:/Windows/system32/KERNELBASE.dll'.
Warning: Cannot locate symbols for file `C:/Program Files (x86)/Intel/VTune Amplifier XE 2013/bin64/tpsstool.dll'.
Symbol resolution has been finished, elapsed time is 3.111 seconds.
Deferred data transformations have been finished, elapsed time is 0.039 seconds.
Precomputing frequently used data has been finished, elapsed time is 0.059 seconds.
Data model parameters have been set, elapsed time is 0.047 seconds.
Redundant overtime data has been discarded, elapsed time is 0.008 seconds.
Raw collector data has been discarded, elapsed time is 0.000 seconds.
Finalizing the result took 4.303 seconds.

Sergey Kostrov wrote:

>>...It cannot work when use 'Custom Size' Virtual Memory ( VM ) instead of 'System managed size' VM...

I'd like to confirm that you've tried to change it.

These four errors:

>>...
>>Error: Unsupported item 'helpKeyword' for query 'vectorQuery(id='')[]'
>>Error: Unsupported item 'helpKeywordCalcPolicy' for query 'vectorQuery(id='')[]'
>>Error: Unsupported item 'helpKeyword' for query 'vectorQuery(id='')[]'
>>Error: Unsupported item 'helpKeywordCalcPolicy' for query 'vectorQuery(id='')[]'
>>...

are indication that something is wrong with internal SQLite database, or with SQL queries, which used by VTune. Did yiu try to install the most recent VTune Update?

[Bernard]

>>>Warning: Cannot locate symbols for file `C:/Windows/system32/kernel32.dll'.>>>

You can point VS debugger to MS debug symbols download server,but it will not solve the problem.Maybe if Intel devs made VTune symbols(all of them) public then it can improve an understanding of function calls.

[Bernard]

>>>Yes, I ran VTune under debug mode in Visual Studio.Net. I also tried to run it in release mode, it produce the same error.>>>

VS debug mode will not be helpful in locating the problem.Windbg could be more helpful,but as it was told in other discussion related to crash of fee.dll module such a task should be done by Intel developers who have the ultimate knowledge of the misbehaving software.

[SergeyKostrov]

>>>>...It cannot work when use 'Custom Size' Virtual Memory ( VM ) instead of 'System managed size' VM... >> >>I'd like to confirm that you've tried to change it. Daniel, could you provide some details on what happened when you've changed VM settings to use 'Custom Size'? Did you try it?