Very large structs GCC ok, ICC seg fault, simple example.

jkwi · ‎02-08-2013

We have a very large structure ~10GB. GCC does not seem to have a problem with it but ICC is seg faulting. I've boiled it down to simple assignment statements. Any ideas? Files attached. If we don't allocate but declare the structure it works fine. Any problems with Intel allocating large amounts of memory? We've tried moving things around in the structure definition but it continues to seg fault.

icc -mcmodel=medium -shared-intel -g test.c -o test
gcc -mcmodel=medium -g test.c -o test

icc: Version 12.1.5.339 Build 20120612
gcc: gcc (GCC) 4.4.6 20120305 (Red Hat 4.4.6-4)

GCC output:
Allocate the grid struct
sizeof of grids 9.393871GB
Address of grids->G2.rain.count 0x7f5d05143190
Address of grids->G2.d0.count 0x7f5d61d35190
Address of grids->G2.Nt.count 0x7f5d96d2d190
Address of grids->G2.dBZm.count 0x7f5f079a0190
location: 0
G2.dBZ.stdev[ii][jj][kk][ll][mm] = 2.001000
Address of grids->G2.dBZm.count 0x7f5f079a0190
assigned G2.dBZm.count
assigend G2.dBZm.mean
assigned G2.dBZm.stdev
Address of G2.d0.count 0x7f5d61d35190
Going to assign d0.count
assigned G2.d0.count
Going to assign d0.mean
assigned G2.d0.mean
Going to assign d0.stdev
assigned G2.d0.stdev
Successful write

Intel ICC:
Allocate the grid struct
sizeof of grids 9.393871GB
Address of grids->G2.rain.count 0x7fc9e070f190
Address of grids->G2.d0.count 0x7fca3d301190
Address of grids->G2.Nt.count 0x7fca722f9190
Address of grids->G2.dBZm.count 0x7fcbe2f6c190
location: 0
G2.dBZ.stdev[ii][jj][kk][ll][mm] = 2.001000
Address of grids->G2.dBZm.count 0x7fcbe2f6c190
assigned G2.dBZm.count
assigend G2.dBZm.mean
assigned G2.dBZm.stdev
Address of G2.d0.count 0x7fca3d301190
Going to assign d0.count
Segmentation fault

TimP · ‎02-08-2013

Did you set maximum stack limit?

If you want to avoid generated code bloat, you may require -O1 or -Os.

SergeyKostrov · ‎02-08-2013

Hi everybody, >>... If we don't allocate but declare the structure it works fine... I understood the problem as follows: - On a 64-bit system with Linux OS and at least 16GB of physical memory the application ( compiled with ICC ) crashes if ~10GB memory block is dynamically allocated with a CRT-function malloc? - On the same system the application works as expected if the memory block is allocated from the stack. Please confirm that you're on a 64-bit system. Note: Just in case please check Virtual Memory settings. For example, two values are usually used, that is, Minimal Size and Maximum Size. So, in your case the Minimal Size has to be greater than 10GB and a recommended value is 16GB. Then, set Maximum Size to 24GB. But, I'm not sure that it will fix the problem with Intel C++ compiler. Anyway, it makes sense to try.

SergeyKostrov · ‎02-08-2013

There is a bug in the codes. The processing continues even if a memory for the structure is not allocated and grids pointer is NULL: ... printf( "Allocate the grid struct\n" ); grids = ( L3DPR_GRIDS * )malloc( sizeof( L3DPR_GRIDS ) ); if( grids == NULL ) { printf( "Error return from malloc\n" ); return ( int )-1; // Added by SergeyK: Exit if the memory is Not allocated } ... printf( "Address of grids->G2.rain.count %p\n",grids->G2.rain.count ); ... Note: I could verify your codes with Intel C++ compiler for Windows. Please confirm if you're interested in that.

jimdempseyatthecove · ‎02-09-2013

Have you run a simple test to verify if sizeof returns 32-bit or 64-bit?

printf("%d\n", sizeof(sizeof(L3DPR_GRIDS ) ));

Jim Dempsey

jkwi · ‎02-11-2013

Thanks for the responses. I did add the exit if malloc failed but it always did succeed as the printf was never executed.
I also added the printf of the sizeof pointer and it is 8 bytes. Still same situation, GCC works, ICC fails. New code and output attached. As far as I can tell this is pretty basic and no clue why Intel compiler is doing this. I also added output from an older 11.1 version of ICC (Version 11.1 Build 20100806 Package ID: l_cproc_p_11.1.073) check out the addresses of some of the elements of the structure (they are the same address!!) This makes no sense to me.

Info on machine:
cputime      unlimited
filesize     unlimited
datasize     unlimited
stacksize    unlimited
coredumpsize unlimited
memoryuse    unlimited
vmemoryuse   unlimited
descriptors 1024
memorylocked unlimited
maxproc      266240

Linux 2.6.32-220.17.1.el6.621g0000.x86_64 #1 SMP Wed May 16 19:27:42 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

GCC output:
Allocate the grid struct
sizeof of grids 9.393871GB
sizeof pointer to grids 8
Address of grids->G2.rain.count 0x2b8154cad190
Address of grids->G2.d0.count 0x2b81b189f190
Address of grids->G2.Nt.count 0x2b81e6897190
Address of grids->G2.dBZm.count 0x2b835750a190
location: 0
G2.dBZ.stdev[ii][jj][kk][ll][mm] = 2.001000
Address of grids->G2.dBZm.count 0x2b835750a190
assigned G2.dBZm.count
assigend G2.dBZm.mean
assigned G2.dBZm.stdev
Address of G2.d0.count 0x2b81b189f190
Going to assign d0.count
assigned G2.d0.count
Going to assign d0.mean
assigned G2.d0.mean
Going to assign d0.stdev
assigned G2.d0.stdev
Successful write

ICC output: Version 12.1.5.339 Build 20120612
Allocate the grid struct
sizeof of grids 9.393871GB
sizeof pointer to grids 8
Address of grids->G2.rain.count 0x2b3a9a790190
Address of grids->G2.d0.count 0x2b3a6d022010
Address of grids->G2.Nt.count 0x2b3a6d022010
Address of grids->G2.dBZm.count 0x2b3a6d022010
location: 0
G2.dBZ.stdev[ii][jj][kk][ll][mm] = 2.001000
Address of grids->G2.dBZm.count 0x2b3a6d022010
assigned G2.dBZm.count
assigend G2.dBZm.mean
assigned G2.dBZm.stdev
Address of G2.d0.count 0x2b3a6d022010
Going to assign d0.count
Segmentation fault (core dumped)

Version 11.1 Build 20100806 Package ID: l_cproc_p_11.1.073
Allocate the grid struct
sizeof of grids 9.393871GB
sizeof pointer to grids 8
Address of grids->G2.rain.count 0x7fd3e3e2b190
Address of grids->G2.d0.count 0x7fd3b66bd010
Address of grids->G2.Nt.count 0x7fd3b66bd010
Address of grids->G2.dBZm.count 0x7fd3b66bd010
location: 0
G2.dBZ.stdev[ii][jj][kk][ll][mm] = 2.001000
Address of grids->G2.dBZm.count 0x7fd3b66bd010
assigned G2.dBZm.count
assigend G2.dBZm.mean
assigned G2.dBZm.stdev
Address of G2.d0.count 0x7fd3b66bd010
Going to assign d0.count
Segmentation fault

SergeyKostrov · ‎02-11-2013

>>... I did add the exit if malloc failed but it always did succeed as the printf was never executed... I will do verification on a 64-bit WIndows 7 Professional and I hope that Intel software engineers will be able to verify the test case on a 64-bit Linux. I'll post my results as soon as they are ready.

jimdempseyatthecove · ‎02-11-2013

>>I also added the printf of the sizeof pointer and it is 8 bytes.

Its not size of pointer "sizeo(void*)". its the size of the return of sizeof "sizeof(sizeof(void*))"

sizeof has a return type of size_t. I've occasionally seen size_t defined/typedefed with "unsigned int" as opposed to "unsigned intptr_t" and on x64 platform where default "int" is 32-bit.

Jim Dempsey

ama1 · ‎02-11-2013

Program was run with stack limit set to unlimited with no effect. My machine has MemTotal: 132089148 kB and I get the same result (I work with jkwi). Thanks!

jkwi · ‎02-11-2013

Added:
printf("sizeof sizeof L3DPR_GRIDS %d\n", (int)sizeof(sizeof(L3DPR_GRIDS)));
printf("sizeof pointer to grids %d\n",(int)sizeof(grids));

Seems upload of files is changing case of filenames to all lower letters.

ICC Version 12.1.5.339 Build 20120612
Allocate the grid struct
sizeof of grids 9.393871GB
sizeof sizeof L3DPR_GRIDS 8
sizeof pointer to grids 8
Address of grids->G2.rain.count 0x7fa223056190
Address of grids->G2.d0.count 0x7fa27fc48190
Address of grids->G2.Nt.count 0x7fa2b4c40190
Address of grids->G2.dBZm.count 0x7fa4258b3190
location: 0
G2.dBZ.stdev[ii][jj][kk][ll][mm] = 2.001000
Address of grids->G2.dBZm.count 0x7fa4258b3190
assigned G2.dBZm.count
assigend G2.dBZm.mean
assigned G2.dBZm.stdev
Address of G2.d0.count 0x7fa27fc48190
Going to assign d0.count
Segmentation fault

SergeyKostrov · ‎02-11-2013

Hi everybody, Thanks for the test-case and it really allows to see what different C++ compilers can do! >>... >>Address of G2.d0.count 0x7fa27fc48190 >>Going to assign d0.count >>Segmentation fault >>... 1. I reproduced the same problem when there is attempt to access to ...G2.d0.count struct member on a Windows 7 Professional ( 64-bit / 16GB physical memory / 64GB virtual memory ) with Intel Parallel Studio XE 2013 ( Initial Release ). 2. I will post all my results and a test project for Visual Studio 2008 Professional Edition. 3. CRT-function malloc worked well and allocated more than 9GB of memory ( Release Configuration of the test application ). >>... >>...I did add the exit if malloc failed but it always did succeed as the printf was never executed. >>... 4. The pointer wasn't initialized to NULL when it was declared. Please remember that CRT-functions, like malloc or calloc, don't set it to NULL if they fail to allocate a block of memory. Here are two examples: [ Without initialization to NULL ( a possible bug ) ] ... int *piData; piData = ( int * )malloc( 8192 * sizeof( int ) ); if( piData == NULL ) return; ... [ With initialization to NULL ] ... int *piData = NULL; piData = ( int * )malloc( 8192 * sizeof( int ) ); if( piData == NULL ) return; ...

SergeyKostrov · ‎02-11-2013

Here are compilation results for Intel C++ compiler: [ Intel C++ compiler - x64 - DEBUG ] ------ Build started: Project: MemTestApp, Configuration: Debug x64 ------ Compiling with Intel(R) C++ Compiler XE 13.0.0.089 [Intel(R) 64]... (Intel C++ Environment) Stdafx.cpp Compiling with Intel(R) C++ Compiler XE 13.0.0.089 [Intel(R) 64]... (Intel C++ Environment) MemTestApp.cpp Compiling manifest to resources... (Microsoft VC++ Environment) Microsoft (R) Windows (R) Resource Compiler Version 6.1.6723.1 Copyright (C) Microsoft Corporation. All rights reserved. Linking... (Intel C++ Environment) xilink: executing 'link' Embedding manifest... (Microsoft VC++ Environment) Microsoft (R) Windows (R) Resource Compiler Version 6.1.6723.1 Copyright (C) Microsoft Corporation. All rights reserved. xilink: executing 'link' MemTestApp - 0 error(s), 0 warning(s), 0 remark(s) [ Intel C++ compiler - x64 - RELEASE ] ------ Build started: Project: MemTestApp, Configuration: Release x64 ------ Compiling with Intel(R) C++ Compiler XE 13.0.0.089 [Intel(R) 64]... (Intel C++ Environment) Stdafx.cpp Compiling with Intel(R) C++ Compiler XE 13.0.0.089 [Intel(R) 64]... (Intel C++ Environment) MemTestApp.cpp Linking... (Intel C++ Environment) xilink: executing 'link' Embedding manifest... (Microsoft VC++ Environment) MemTestApp - 0 error(s), 0 warning(s), 0 remark(s) [ Intel C++ compiler - Win32 - DEBUG ] ------ Build started: Project: MemTestApp, Configuration: Debug Win32 ------ Compiling with Intel(R) C++ Compiler XE 13.0.0.089 [IA-32]... (Intel C++ Environment) Stdafx.cpp Compiling with Intel(R) C++ Compiler XE 13.0.0.089 [IA-32]... (Intel C++ Environment) MemTestApp.cpp C:\WuTemp\MemTestApp\MemTestApp.cpp(56) (col. 2): internal error: 04010002_1071 compilation aborted for .\MemTestApp.cpp (code 4) MemTestApp - 1 error(s), 0 warning(s), 0 remark(s) [ Intel C++ compiler - Win32 - RELEASE ] ------ Build started: Project: MemTestApp, Configuration: Release Win32 ------ Compiling with Intel(R) C++ Compiler XE 13.0.0.089 [IA-32]... (Intel C++ Environment) Stdafx.cpp Compiling with Intel(R) C++ Compiler XE 13.0.0.089 [IA-32]... (Intel C++ Environment) MemTestApp.cpp C:\WuTemp\MemTestApp\MemTestApp.cpp(56) (col. 2): internal error: 04010002_1071 compilation aborted for .\MemTestApp.cpp (code 4) MemTestApp - 1 error(s), 0 warning(s), 0 remark(s)

SergeyKostrov · ‎02-11-2013

Here are compilation results for Microsoft C++ compiler ( Unfortunately, it couldn't compile sources for all configurations ): [ Microsoft C++ compiler - x64 - DEBUG ] ------ Build started: Project: MemTestApp, Configuration: Release x64 ------ Compiling... cl : Command line warning D9035 : option 'Wp64' has been deprecated and will be removed in a future release Stdafx.cpp c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(484) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(485) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(486) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(487) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(488) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(489) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(490) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(491) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(492) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(493) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(494) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(495) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(496) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(497) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(498) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(542) : error C2089: '' : 'struct' too large MemTestApp - 16 error(s), 1 warning(s) [ Microsoft C++ compiler - x64 - RELEASE ] ------ Build started: Project: MemTestApp, Configuration: Debug x64 ------ Compiling... cl : Command line warning D9035 : option 'Wp64' has been deprecated and will be removed in a future release Stdafx.cpp c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(484) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(485) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(486) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(487) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(488) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(489) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(490) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(491) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(492) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(493) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(494) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(495) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(496) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(497) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(498) : error C2089: '' : 'struct' too large c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(542) : error C2089: '' : 'struct' too large MemTestApp - 16 error(s), 1 warning(s) Note: The same errors are displayed for Win32 configurations.

SergeyKostrov · ‎02-11-2013

Here is Output of the test application: 64-bit Windows platform Allocate the grid struct sizeof of grids 9.393871GB Address of grids->G2.rain.count 000000016C9EE1C0 Address of grids->G2.d0.count 00000001C95E01C0 Address of grids->G2.Nt.count 00000001FE5D81C0 Address of grids->G2.dBZm.count 000000036F24B1C0 location: 0 G2.dBZ.stdev[ii][jj][kk][ll][mm] = 2.001000 Address of grids->G2.dBZm.count 000000036F24B1C0 assigned G2.dBZm.count assigend G2.dBZm.mean assigned G2.dBZm.stdev Address of G2.d0.count 00000001C95E01C0 Successful write Press ESC to Exit... Note: I commented out the following blocks because something goes wrong with accessing d0 struct members: ... // printf( "Going to assign d0.count\n" ); // grids->G2.d0.count[ii][jj][kk][ll][mm] = ( __int32 )6; // Access Violation if uncommented // printf( "assigned G2.d0.count\n" ); // printf( "Going to assign d0.mean\n" ); // grids->G2.d0.mean[ii][jj][kk][ll][mm] = five + plusVal; // Access Violation if uncommented // printf( "assigned G2.d0.mean\n" ); // printf( "Going to assign d0.stdev\n" ); // grids->G2.d0.stdev[ii][jj][kk][ll][mm] = six + plusVal; // Access Violation if uncommented // printf( "assigned G2.d0.stdev\n" ); ...

SergeyKostrov · ‎02-11-2013

Here is a test project for Visual Studio 2008 Professional Edition ( Intel C++ compiler is set ). Please let me know if you have any questions. PS: This is a really good test case.

Mark_S_Intel1 · ‎02-11-2013

Thank you for the problem report and all your comments. I will investigate this issue and get back to you.

--mark

jkwi · ‎02-11-2013

Thanks for verifying this and all the additional info Sergey. Hopefully the project info will help Intel.

We will be waiting for a response from Intel.

Thanks again!!

bernaske · ‎02-12-2013

Hi Sergey,

i have the code samples compile with Intel C++ 13.1 and gcc-4.6.3, then i have debug with gdb, idb, and the Allinea DDT debuggers

also i analyse with inspector XE 2013, advisor XE 2013, the tools report memory leak at line 62, i think you are right its a codeing error not

a compiler error

best regards

Franz Bernasek

SergeyKostrov · ‎02-12-2013

>>... i think you are right its a codeing error not a compiler error... I think this is a compiler error and Intel engineers should confirm if it is not. I don't see any problems with the test case.

SergeyKostrov · ‎02-12-2013

Hi Franz, >>...i analyse with inspector XE 2013, advisor XE 2013, the tools report memory leak at line 62... Thank you very much for the note. Please find attached updated test case with a small fix. I added a call to free CRT-function at the end of main function: ... if( grids != NULL ) { free( grids ); grids = NULL; } ... Best regards, Sergey

SergeyKostrov · ‎02-12-2013

This is a short follow up regarding the problem with Microsoft C++ compiler... >>... >>------ Build started: Project: MemTestApp, Configuration: Release x64 ------ >>Compiling... >>cl : Command line warning D9035 : option 'Wp64' has been deprecated and will be removed in a future release >>Stdafx.cpp >>c:\wutemp\memtestapp\Tk-3dpr-hdf5.h(484) : error C2089: '' : 'struct' too large >>... I reported the problem to Microsoft and this is a response: ... ...Visual C++ does not handle structures larger than 2GB. This appears to be a known issue... ...