OMP: System error #1455: The paging file is too small for this operation to complete

[SergeyKostrov]

The title of the thread describes the problem with MKL and I'll provide additional technical details.

[Bernard]

Sergey

 Can you run VMmap tool and/or Xperf and investigate which library/function(s) is allocating a large amount of memory?

[jimdempseyatthecove]

>>

[ Test 14 ]

OMP_NUM_THREADS=80
KMP_NUM_THREADS=80
...
Calculated ( in seconds ): 328.1630
...
Number of CPUs used: 8 <***************
Number of Threads used: 80 <***********

[ Test 15 ]

OMP_NUM_THREADS=96
KMP_NUM_THREADS=96
...
OMP: Error #136: Cannot create thread. <******* ~2GB/thread
OMP: System error #1455: The paging file is too small for this operation to complete.
OMP: Error #178: Function GetExitCodeThread() failed:
OMP: System error #6: The handle is invalid.
...
Number of CPUs used: N/A
Number of Threads used: N/A
<<

Why are you oversubscribing the threads?
Oversubscription is generally counter-productive

Jim Dempsey

[SergeyKostrov]

>>... Can you run VMmap tool and/or Xperf and investigate which library/function(s) is allocating a large amount of memory? >>...Why are you oversubscribing the threads? Jim, Iliya, If you have free time you could spend as much as you can with the test peoject I've attached. I hope that Intel software engineers will investigate why excessive amount of memory is committed. There is nothing else I can do in that case.

[Bernard]

>>>If you have free time you could spend as much as you can with the test peoject I've attached>>>

I wish I could.I am still waiting for a new computer:)

[Bernard]

>>>Why are you oversubscribing the threads?
Oversubscription is generally counter-productive>>>

Btw a link to interested article about the thread oversubscription :http://blogs.msdn.com/b/visualizeparallel/archive/2009/12/01/oversubscription-a-classic-parallel-performance-problem.aspx?Redirected=true

[SergeyKostrov]

Oversubscription of OpenMP threads was not a concern in that case.

[Bernard]

I suppose that OpenMP library is pre-allocating or commiting memory as a function of total threads number hence the out of memory (pagefile.sys) situation.This error could be due to improper task partitioning in terms of allocated memory per thread.

[SergeyKostrov]

>>...I suppose that OpenMP library is pre-allocating or commiting memory... I know from the beginning that something is wrong with: scalable_malloc or scalable_aligned_malloc from tbbmalloc.dll or _MALLOC_POOL_INCR or KMP_MALLOC or kmp_malloc or kmpc_malloc from libiomp5md.dll. Should we continue that discussion? I don't think so because Intel software engineers will take a look at the problem and if they find a problem it will be fixed in the source codes.

[Bernard]

Agree with you.

[SergeyKostrov]

This is a short follow up because there was a typing error in my previous post and I did a correction: From libiompprof5md.dll To libiomp5md.dll Sorry about this.

[jimdempseyatthecove]

Sergey,

Are you using TBB's scalable malloc within a non-TBB app using OpenMP?

Nothing inherently wrong with this, assuming you know how a scalable malloc works. From your earlier posts, one can derive that each thread requires 2GB of RAM. Since this is occuring at thread startup, it is either requested thread stack, or Thread Local Storage. Once this thread is running, its first scalable malloc allocation is going to acquire a slab of RAM to be used in a manner as a private heap that the thread can allocate from _without_ using a critical section. Actually it is like n x private heaps for varying chunk sizes. Scallable malloc requires more memory than standard heap. Do what you can to reduce the 2GB. As your runtime stats show no improvements when you exceed the number of hardware threads, there is no driving reason for you to be requesting that many threads.

Jim Dempsey

[SergeyKostrov]

>>... there is no driving reason for you to be requesting that many threads... Jim, Do you understand a term Stress Testing of Software?

[jimdempseyatthecove]

Sergey,

Yes, you ran your stress test to the point of running out of available Virtual Memory (due to limitations on the page file). What do you expect to happen under these circumstances? The error message you reported early in this thread is self explanitory:

OMP: Error #136: Cannot create thread.
OMP: System error #1455: The paging file is too small for this operation to complete.

You specified how many threads to create, and due to your 2GB/thread setup, you ran out of page file. If you need to stress more threads - increase your page file (or tweak your program to use less than 2GB/thread).

Jim Dempsey

[SergeyKostrov]

Jim, Your comments Do Not change anything and it is not clear for me what are you going to prove? I also tried to use smaller sizes for OpenMP threads and it doesn't change matters as well. >>... >>...Smaller values for OMP_STACKSIZE and KMP_STACKSIZE do not resolve the issue >>...

[Bernard]

>>>Smaller values for OMP_STACKSIZE and KMP_STACKSIZE do not resolve the issue>>>

It yet needs to be confirmed which library is making those allocations.When my new pc will arrive I will check your programme with Xperf.

[SergeyKostrov]

I'd like to reiterate that I personally Do Not have any problems with that functionality ( a workaround works ) and the thread needs to be considered as the Report with lots of Technical Details that some issue with Intel Runtime library(ies), libiomp5md.dll or libimalloc.dll or tbbmalloc.dll, exists. Please, Do Not post if you're Not Intel software engineer and if you Do Not have access to source codes of libiomp5md.dll or libimalloc.dll libraries. Any comments from Intel software engineers? Thanks in advance.