error : unable to obtain mapped memory (see pch_diag.txt)

Ian_Mallett1 · ‎07-23-2016

Hi,

After updating to Version 2016.3.207 (Intel Compiler 16.0 Update 3), I am now getting errors like the following all over the place:

C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\vector(1320): error : unable to obtain mapped memory (see pch_diag.txt)
    		typename enable_if<_Is_iterator<_Iter>::value,
    		                   ^
            detected during instantiation of class "std::vector<_Ty, _Alloc> [with _Ty=glLib::Texturing::Texture2D *, _Alloc=std::allocator<glLib::Texturing::Texture2D *>]" at line 37 of "C:\dev\C++\Libraries\gllib\gllib\gllib/effects/imagespace/depth_peel.h"

The only related info comes from an ancient thread which proclaimed it fixed.

Thanks,

Melanie_B_Intel · ‎07-23-2016

The diagnostic you received is current. It is due to an implementation restriction in the Intel precompiled headers. When the Intel compiler restores precompiled headers (the /Yu option) it needs to obtain a big chunk of memory at a particular address. If that chunk of memory is being used by a different process then you will see that message. The solution is to find a chunk of memory that can be used, and provide that address when the precompiled headers are created. Alternatively, you could disable the use of precompiled headers.

There have been a couple posts about this recently. One of these posts provide more details including a description of how you can investigate the process memory usage.

See here: https://software.intel.com/en-us/forums/intel-c-compiler/topic/660191

And here: https://software.intel.com/en-us/forums/intel-c-compiler/topic/629164

Hope this helps. --Melanie

Ian_Mallett1 · ‎08-25-2016

Hi

If I'm reading this right (and based on information from Intel here), basically what's happening is the Intel compiler tries to load a large file (the PCH) into (virtual) address 0x0000'0000'3000'0000. But because of ASLR (an important security feature), the OS might have put part of ICC at that address first. Hence, the compiler gives up and terminates.

I'm sorry, but this seems really lame. Basically, whether or not the compiler works correctly is a random variable. There's a compiler option to change what address you're testing, but that doesn't change the fact that it's fundamentally unpredictable. (In fact, the only reason a compiler option is useful at all is because ASLR isn't fully random, so some addresses are less common than others though, experimentally, precisely which are less common changes, especially at reboots.)

I'm not sure exactly what reason one has for loading anything into a particular virtual address. The only case I know of where hardcoded pointers are used is for physical addresses when you're writing an operating system or doing embedded hardware IO--and that's only for basic setup before you configure the MMU. But then again, I'm not a compiler writer.

Regardless, even if you must use a fixed address for some reason, the existence of the compiler option to change the address suggests a fix: just pick the address automatically. Keep incrementing your fixed value by 256MiB until you get a big-enough space open. Respectfully, making the end user attempt to do this manually every time the OS changes its internal state is just wrong.

Thanks,
Ian

SergeyKostrov · ‎08-27-2016

>>... >>Intel compiler tries to load a large file (the PCH) into (virtual) address... How big is that Pch file? Is it bigger than Maximum Size ( MB ) of a Virtual Memory paging file used by a Windows OS? In order to verify it follow: My Computer -> System Properties -> Advanced -> Performance Settings -> Advanced -> take a look at Virtual Memory group box -> Press Change button. On all Windows OSs I use a value for Maximum Size ( MB ) of a paging file exceeds a value of Physical Memory installed on a computer at least in 3 - 4 times. I agree that the compilation error you've experienced is very strange but it looks like this is due to an Insufficient amount of virtual memory available on your computer system.

Ian_Mallett1 · ‎08-27-2016

The paging file doesn't have anything to do with this; the page file effectively extends the amount of physical memory by using the hard drive as a backing store for some pages. On modern OSes, physical memory is not a concept that is exposed to user-mode programs--such as ICC--and despite the misleading name, the page file has nothing to do with virtual memory (in the usual sense of address translation by MMU/TLB).

If I might ask a leading question: what technical reason does ICC have for loading the PCH into a fixed virtual memory address?

SergeyKostrov · ‎08-27-2016

There is another issue to mention. It is Not clear what Intel C++ compiler you're using, I mean a 32-bit or 64-bit? If you're using the 32-bit Intel C++ compiler, and a size of the Pch file is greater than 2GB, and Virtual Memory paging file is small, then that is why Win32 API memory mapped function failed. Take into account that for a regular 32-bit Windows application only 2GB of memory could be allocated on a 32-bit Windows OS ( if it doesn't support 3GB limit ).

SergeyKostrov · ‎08-27-2016

>>...what technical reason does ICC have for loading the PCH into a fixed virtual memory address?.. Intel won't answer that question but, is it difficult for you to verify and provide answer on what I've asked in my 2 posts?

Ian_Mallett1 · ‎08-27-2016

The computer is a 64-bit machine with matching 64-bit Win 7 OS. Hence, the compiler I installed was the 64-bit latest version (ver 16.0, I believe) of the Intel Compiler. Windows's documentation helpfully tells me that the virtual address space for user-mode programs on my machine is 8TiB--which is plenty for loading any saved file at all (considering that my HDD is only 1TiB).

As I said, the page file doesn't matter since physical and virtual memory are different. But just in case, it's 12 GiB--which is much, much larger than the PCH was. At the time the compiler crashes, 8GiB of physical DRAM was available too. That should be plenty to back a PCH with a few system headers in it.

---

Regarding my "leading question", it is not me that needs to know the answer. It's something your compiler writers should ask themselves. As I wrote, the only reason for hardcoded pointers is to write a driver or an OS (the Intel Compiler is neither of these things)--and those are physical addresses, not virtual addresses.

Yet, you also gave some hints; you're asking about the page file and also saying "then that is why Win32 API memory mapped function failed." From this, I infer that you are using "CreateFileMapping(...)" to load the PCH into the page file, then using "MapViewOfFileEx(...)" to memory-map it into ICC's virtual address space. And you're passing in the default or user-defined pointer into the "lpBaseAddress" argument. Then that call is failing because that address is already used.

Assuming these wild guesses are right (big "if", I'll grant), you could completely eliminate this entire problem by passing null and getting the correct address as that function's retval instead. You could alternately just call the function again with different addresses until you get something that succeeds. That parameter exists for historical reasons, and there's no reason to use it, precisely because you run into problems such as this.

Regardless, the point is that (1) AFAIK there's no good reason to ever hardcode a virtual address on a modern OS and (2) Users shouldn't have to care about such implementation details of ICC, regardless of whatever you decide to do.

SergeyKostrov · ‎08-27-2016

This is what I would suggest. >>...After updating to Version 2016.3.207 ( Intel Compiler 16.0 Update 3 ), If you didn't have that compilation problem with Update 2 then uninstalling Update 3 is the only workaround that could help you at the moment.

SergeyKostrov · ‎08-27-2016

>>...As I said, the page file doesn't matter since physical and virtual memory are different. But just in case, it's 12 GiB... One more thing. Before going back to Update 2 could you try to increase Virtual Memory paging file Maximum Size to 24GB, or 36GB? It is a fast verification that a limitation of Maximum Size of the paging file is Not the reason of the compilation problem. PS: On my primary development computer I have 32GB of Physical Memory and Maximum Size of the paging file is set to 128GB. It allows to do HPC of very big data set up-to 32 Giga Single-Precision elements ( more than 128GB of memory is used in that case ).

Ian_Mallett1 · ‎08-27-2016

Sigh. To humor you, I set the page file size to 131072 MiB (128 GiB), which is the largest I have space for. It still didn't work--because again, this is a virtual memory issue, not a page-file issue. Even if you're loading the PCH temporarily into the page file (which you do not have to do, BTW, and probably shouldn't since a direct mmap will be faster), the PCH it crashes while trying to include is ~48.6 MiB, and could fit 2,692 times over.

Now please listen to me.

This is a problem with ICC, not with its users. It is unacceptable to expect end users to try to perturb their operating system to work around the fact that ICC is doing blatantly incorrect things with its own virtual memory space. Even if it were, because of ASLR, the problem is theoretically impossible for an end user to reliably solve. You have plenty of RAM, swap, CPU, disk at your disposal--and I expect any program to function reliably under such unconstrained, extremely-normal, operating conditions. So again, even if I'm utterly wrong about any of the preceding or following--and apparently I cannot stress this enough--this is a bug, and it is not the user's responsibility to work around it by passing different compiler flags until ICC "just happens" to not get itself into a bogus state.
Tell your compiler people to stop memory mapping anything at a fixed virtual address. There is literally no reason to do this, and the Win32 system APIs that allow it are explicitly discouraged by Microsoft itself. Why does Microsoft tell you not to do this? Again, because it is wrong to use them, dangerous to use them, and there is no reason to use them. You should not be using them.
I strongly suspect the root cause is your passing in the fixed address when you mmap a loaded file (the PCH).
If so, the ready-made solution is to pass "null" instead. The OS will choose the address for you, which is what is supposed to happen in the first place when you allocate virtual memory. You won't crash, and I'll be happy.

This problem did occur in Update 2, just not as frequently. This probably means the code changes between updates 2 and 3 affected how the linker specified how ICC is to be laid out in RAM, and the change makes the problem more probable. It doesn't mean anything important. I don't know how long this bug existed, but it existed in Update 2 as well. It just only broke things some of the time--not all of the time. In any case, we can't switch back to update 2 due to several of its compiler bugs (that I've reported, even) that we depend on being fixed (ex: 1, 2, 3, etc.).

Until you fix this--and fix it correctly; not with crockish compile flags that delegate your technical problems to the user--we are forced to switch back to MSVC.

Judith_W_Intel · ‎09-01-2016

First, I am sorry for all the trouble. We would also consider this a bug/limitation of our implementation.I don't know why there would be any difference between Update 2 and Update 3 in terms of frequency.

But just so you understand that the fix is not simple. The reason for using the fixed address scheme is that the mapped address must be the same when you create the pch as when you use the pch. If we let the operating system decide then that would not necessarily be the case.

We could redesign our implementation to use a non fixed based address and then have indexed pointers into that base address. But that would be a major rewrite of our existing implementation.

JenniferJ · ‎09-13-2016

Compile time is important to everyone for productivity reason, in order to have a better compiler time for building the program, the fixed address for pch is chosen. Especially for Intel Compiler it has a lot of advanced optimizations and that requires time to accomplish.
That being said, we did file a feature request DPD200523124 for this issue. Because it's a very involved change, if the workarounds work for you, please use that for now.

One important note on this issue: we haven't got any testcase that could duplicate this issue internally. If you could help us in providing a testcase or isolating the issue, it'd be greatly appreciated. It might be something easier to fix.

Thanks,
Jennifer

Ian_Mallett1 · ‎09-13-2016

@Judith, @Jennifer: While the changes may be involved and compile times might be affected, the key point for me is that the code is wrong. E.g., the fact that sometimes it worked faster isn't especially relevant--since sometimes it doesn't work. I can also understand that the changes may be complex. Some bugs are like that. It's just too bad--they're still bugs; it shouldn't have been that way in the first place. So I'm sympathetic, but it doesn't really change anything.

As noted, for our development machines, the failures were frequent enough (and the workarounds didn't really help) that we had to switch. So currently, we're using MSVC.

@Jennifer: Unfortunately, by definition this bug depends on random aspects of the user's computer, such as ASLR. For the same reason that it is literally impossible for the user to reliably create a workaround (as explained above), it is impossible for me to generate a testcase I can guarantee will reproduce the problem.

Before we were forced to switch to MSVC, we could reproduce it pretty reliably. The main factor seemed to be large PCH size. Of course, as above, this is by far not guaranteed to cause the problem, but it might explain why a small test case is less likely to find it. But anyway, while no one has directly confirmed my inference, as derived above I'm pretty sure this problem comes from using that hardcoded virtual address--which is a CS 101 "no-no" and should be easy to find.

math · ‎10-30-2016

Good morning, since Intel® Parallel Studio XE 2017 no longer supports VS2010 I have to upgrade to VS2015. Unfortunately all the sudden the following problem occurs independent of any Win7 virtual page settings. All tests are done within Virtual Box 5.0.26 / Windows 7 64Bit a) VS2015 compiles and links all projects perfectly. b) VS2010 with parallel_studio_xe_2016_update3 works excellent. c) VS2015 with parallel_studio_xe_2016_update3 and/or Intel® Parallel Studio XE 2017 throws the following errors in various projects. 1>C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\atlmfc\include\atlhost.h(2214): error : unable to obtain mapped memory (see pch_diag.txt) 1> CComQIPtr spAdviseSink(GetControllingUnknown()); 1> ^ 1> detected during: 1> instantiation of class "ATL::CComPtrBase [with T=IAdviseSink]" at line 316 of "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\atlmfc\include\atlcomcli.h" 1> instantiation of class "ATL::CComPtr [with T=IAdviseSink]" at line 2214 1> implicit generation of "ATL::CComQIPtr::~CComQIPtr() [with T=IAdviseSink, piid=&__uuidof(IAdviseSink)]" at line 2214 1> instantiation of class "ATL::CComQIPtr [with T=IAdviseSink, piid=&__uuidof(IAdviseSink)]" at line 2214 1> 1> compilation aborted for stdafx.cpp (code 4) The errors are thrown independent of a project is compiled as 32bit or 64bit project 1> stdafx.cpp 1>C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\atlmfc\include\afxwin.h(2324): error : unable to obtain mapped memory (see pch_diag.txt) 1> void DrawMenuBar(); 1> ^ 1> 1> compilation aborted for stdafx.cpp (code 4) ========== Rebuild All: 0 succeeded, 1 failed, 0 skipped ========== 1> stdafx.cpp 1>C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\atlmfc\include\atlctl.h(3321): error : unable to obtain mapped memory (see pch_diag.txt) 1> STDMETHOD(InPlaceDeactivate)(void) 1> ^ 1> 1> compilation aborted for stdafx.cpp (code 4) ========== Rebuild All: 0 succeeded, 1 failed, 0 skipped ========== 1> stdafx.cpp 1>C:\Program Files (x86)\Windows Kits\8.1\Include\um\shlguid.h(254): error : unable to obtain mapped memory (see pch_diag.txt) 1> DEFINE_GUID(CLSID_ACLCustomMRU, 0x6935db93, 0x21e8, 0x4ccc, 0xbe, 0xb9, 0x9f, 0xe3, 0xc7, 0x7a, 0x29, 0x7a); 1> ^ 1> 1> compilation aborted for stdafx.cpp (code 4) ========== Rebuild All: 0 succeeded, 1 failed, 0 skipped ========== /Yu"stdafx.h" /GS /W3 /QxSSE4.2 /Gy /Zc:wchar_t /I"C:\Random_FileName1" /I"C:Random_FileName2" /Zi /O2 /Fd"Release\vc140.pdb" /D "WIN32" /D "_WINDOWS" /D "NDEBUG" /D "_USRDLL" /D "_WINDLL" /D "_UNICODE" /D "UNICODE" /D "_AFXDLL" /Qip /Zc:forScope /Gd /Oi /MD /QaxSSE4.2 /Fa"Release\" /EHsc /nologo /Fo"Release\" /Qprof-dir "Release\" /Fp"Release\Random_FileName3.pch" 1> stdafx.cpp 1>C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\atlmfc\include\afxwin1.inl(125): error : unable to obtain mapped memory (see pch_diag.txt) 1> _AFXWIN_INLINE CBrush* PASCAL CBrush::FromHandle(HBRUSH hBrush) 1> ^ 1> 1> compilation aborted for stdafx.cpp (code 4) Same project with other compiler commands: /GS /W3 /QxSSE4.2 /Gy /Zc:wchar_t /I"C:Random_FileName1" /I"C:Random_FileName2" /Zi /O2 /Fd"Release\vc140.pdb" /D "WIN32" /D "_WINDOWS" /D "NDEBUG" /D "_USRDLL" /D "_WINDLL" /D "_UNICODE" /D "UNICODE" /D "_AFXDLL" /Qip /Zc:forScope /Gd /Oi /MD /QaxSSE4.2 /Fa"Release\" /EHsc /nologo /Fo"Release\" /Qprof-dir "Release\" /Fp"Release\Random_FileName3.pch" 1> stdafx.cpp 1>C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\atlmfc\include\afxwin1.inl(125): error : unable to obtain mapped memory (see pch_diag.txt) 1> _AFXWIN_INLINE CBrush* PASCAL CBrush::FromHandle(HBRUSH hBrush) 1> ^ 1> 1> compilation aborted for stdafx.cpp (code 4) ========== Rebuild All: 0 succeeded, 1 failed, 0 skipped ========== Many thanks Martin

Mansion__Michael · ‎11-07-2016

Good morning

I have the same problem :

i just try to create an empty ATL project without adding any code inside.

i use vs 2015 update 3, on windows server 2008 R2 entreprise, with 768gb of ram memory and size of memory swap file to 1470000mb. (= arround 1.4Tb)

When i compile it with intel compiler 2017 update 1 : i have this error :

1>C:\Program Files (x86)\Windows Kits\8.1\Include\um\mshtml.h(53603): error : unable to obtain mapped memory (see pch_diag.txt)
1>        IHTMLNamespace : public IDispatch
1>        ^
1>
1> test_atl_07112016.cpp
1> compilation aborted for stdafx.cpp (code 4)
1>C:\Program Files (x86)\Windows Kits\8.1\Include\um\mshtml.h(53603): error : unable to obtain mapped memory (see pch_diag.txt)
1>        IHTMLNamespace : public IDispatch
1>        ^
1>
1> compilation aborted for test_atl_07112016.cpp (code 4)
========== Régénération globale : 0 a réussi, 1 a échoué, 0 a été ignoré ==========

I would like to use the intel compiler instead of microsoft one.

So How to compile just an empty ATL project with your compiler ?

What do you advise ?

i join my empty atl project as attachment

regards

Michael

jimdempseyatthecove · ‎11-07-2016

Jennifer (Intel),

>>Compile time is important to everyone for productivity reason, in order to have a better compiler time for building the program, the fixed address for pch is chosen.

This issue appears NOT to be related to the choice of a fixed address, rather it appears to be a matter of your choice of a fixed (maximum) size. I assume you should be able to increase this size (unless the size is a limitation of MS's devenv.exe).

FWIW - From a different post (Steve Lionel may remember), there is(was) an issue of redundant information placed into the object files inducing what appeared to be file bloat. See if Steve can point you to that thread. Then see if the cause of the file bloat is related to a .pdb bloat.

Jim Dempsey

Mansion__Michael · ‎11-07-2016

i have added an empty disk to increase the size of virtual memory in windows 2008 server R2 (x64)

now i have allocated actually : 1886402 mb (a little less than 2tb) (sorry i use a french version so maybe in the us version it is not the same words)

(i have tried also to use 2 disk to increase the virtual memory file to over 2.2tb with the same consequences)

i still have the problem with my empty atl project (see above for details of the atl empty project and the computer i use)

i have deactivated the precompiled header :

1>------ Début de la régénération globale : Projet : test_atl_07112016, Configuration : DIC x64 ------
1> 64 bit Processing .\test_atl_07112016.idl
1> test_atl_07112016.idl
1> 64 bit Processing C:\Program Files (x86)\Windows Kits\8.1\Include\um\oaidl.idl
1> oaidl.idl
1> 64 bit Processing C:\Program Files (x86)\Windows Kits\8.1\Include\um\objidl.idl
1> objidl.idl
1> 64 bit Processing C:\Program Files (x86)\Windows Kits\8.1\Include\um\unknwn.idl
1> unknwn.idl
1> 64 bit Processing C:\Program Files (x86)\Windows Kits\8.1\Include\shared\wtypes.idl
1> wtypes.idl
1> 64 bit Processing C:\Program Files (x86)\Windows Kits\8.1\Include\shared\wtypesbase.idl
1> wtypesbase.idl
1> 64 bit Processing C:\Program Files (x86)\Windows Kits\8.1\Include\shared\basetsd.h
1> basetsd.h
1> 64 bit Processing C:\Program Files (x86)\Windows Kits\8.1\Include\shared\guiddef.h
1> guiddef.h
1> 64 bit Processing C:\Program Files (x86)\Windows Kits\8.1\Include\um\ocidl.idl
1> ocidl.idl
1> 64 bit Processing C:\Program Files (x86)\Windows Kits\8.1\Include\um\oleidl.idl
1> oleidl.idl
1> 64 bit Processing C:\Program Files (x86)\Windows Kits\8.1\Include\um\servprov.idl
1> servprov.idl
1> 64 bit Processing C:\Program Files (x86)\Windows Kits\8.1\Include\um\urlmon.idl
1> urlmon.idl
1> 64 bit Processing C:\Program Files (x86)\Windows Kits\8.1\Include\um\msxml.idl
1> msxml.idl
1> 64 bit Processing C:\Program Files (x86)\Windows Kits\8.1\Include\um\oaidl.acf
1> oaidl.acf
1> 64 bit Processing C:\Program Files (x86)\Windows Kits\8.1\Include\um\ocidl.acf
1> ocidl.acf
1> stdafx.cpp
1>C:\Program Files (x86)\Windows Kits\8.1\Include\um\mshtml.h(49490): error : unable to obtain mapped memory (see pch_diag.txt)
1> IDOMParser : public IDispatch
1> ^
1>
1> compilation aborted for stdafx.cpp (code 4)
========== Régénération globale : 0 a réussi, 1 a échoué, 0 a été ignoré ==========

So there is an other solution ?

Should i change the operating system to a new one ? (maybe windows server 2012 or windows server 2016 can be better ?)

for an different post which point an other solution : you can put the link here

Thanks in advance

regards Michael

P.S. i have joined the project again so you can see pch_diag.txt inside corresponding to my configuration

Nikhil_D_1 · ‎09-27-2017

This ASLR has been a long standing issue (since 2013 for us) but haven't and haven't seen a resolution yet except for disabling precompiled headers which defeats the purpose. We upgraded to Parallel Studio XE2017 Update 3 and still have the same problem. OS is Win7/64bit. The longer compile times are quite an irritant and are impacting productivity. We cannot use any of the compiler features which was the reason we bought the product.

Give the high price point for this tool it is quite unacceptable to have a long standing issue like this, where an engineer is able to use the product/fail completely dependent on random chance. We have suffered through this since 2013 and the end result is we are gradually moving away from Intel tools. We gave it one more shot in 2017 but no luck.

Never seems to be an issue with the MSVS2015 compiler ? Any timeframe on a real fix ?

Jeff_H_ · ‎10-19-2017

After upgrading to XE2018 on 64-bit Windows 7, I've had to disable precompiled headers even in our most trivial projects due to this issue.

Will there be a fix coming anytime soon?

Wang__RuiQiang · ‎11-16-2017

In my case, even I choose "Not using procompiled header" , I still encounter this issue for one of my c++ projects of vc2017 that are just converted from visual studio 2010 projects.

What's strange is, after I recreate the project using same source code files, I am able to compile it using intel c++ compiler. Obviously certain project setting caused this issue.