C++ - 2018 Update1/Update2 broken with CentOS7 devtoolset-6/devtoolset-7

[Scott_S_1]

We have the following test case. It compiles and runs normally on CentOS7 with Intel PSXE 2018 and devtoolset-6/devtoolset-7. In PSXE 2018 Update1 or Update 2, the same code compiles, but is broken. If, in Update1 or Update2, the code is compiled with -O1, it no longer fails:

$ cat test.cpp 
#include <string>

std::string a() {
return "aaa";
}
std::string b() {
return std::string("bbbbbbbb") + std::string("c");
/* return std::string("bbbbbbb") + std::string("c"); */
}
int main( int argc, char *argv[] ) {
std::string s = a() + b();
return 0;
}

[ssilverman@scottc7virt icctest]$ icpc -v; icpc -o test.gcc485 test.cpp
icpc version 18.0.2 (gcc version 4.8.5 compatibility)
[ssilverman@scottc7virt icctest]$ scl enable devtoolset-6 "icpc -v; icpc -o test.gcc631 test.cpp"
icpc version 18.0.2 (gcc version 6.3.1 compatibility)

[ssilverman@scottc7virt icctest]$ scl enable devtoolset-7 "icpc -v; icpc -o test.gcc700 test.cpp"
icpc version 18.0.2 (gcc version 7.0.0 compatibility)

[ssilverman@scottc7virt icctest]$ ./test.gcc485 
[ssilverman@scottc7virt icctest]$ ./test.gcc631
*** Error in `./test.gcc631': double free or corruption (fasttop): 0x00000000018260a0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7c619)[0x7f9dbe6af619]
./test.gcc631[0x400f7d]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f9dbe654c05]
./test.gcc631[0x400be9]
======= Memory map: ========
00400000-00404000 r-xp 00000000 fd:02 16728111                           /home/ssilverman/icctest/test.gcc631
00603000-00604000 r--p 00003000 fd:02 16728111                           /home/ssilverman/icctest/test.gcc631
00604000-00605000 rw-p 00004000 fd:02 16728111                           /home/ssilverman/icctest/test.gcc631
01826000-01847000 rw-p 00000000 00:00 0                                  [heap]
7f9db8000000-7f9db8021000 rw-p 00000000 00:00 0 
7f9db8021000-7f9dbc000000 ---p 00000000 00:00 0 
7f9dbe42f000-7f9dbe431000 r-xp 00000000 fd:00 33780127                   /usr/lib64/libdl-2.17.so
7f9dbe431000-7f9dbe631000 ---p 00002000 fd:00 33780127                   /usr/lib64/libdl-2.17.so
7f9dbe631000-7f9dbe632000 r--p 00002000 fd:00 33780127                   /usr/lib64/libdl-2.17.so
7f9dbe632000-7f9dbe633000 rw-p 00003000 fd:00 33780127                   /usr/lib64/libdl-2.17.so
7f9dbe633000-7f9dbe7eb000 r-xp 00000000 fd:00 33709255                   /usr/lib64/libc-2.17.so
7f9dbe7eb000-7f9dbe9eb000 ---p 001b8000 fd:00 33709255                   /usr/lib64/libc-2.17.so
7f9dbe9eb000-7f9dbe9ef000 r--p 001b8000 fd:00 33709255                   /usr/lib64/libc-2.17.so
7f9dbe9ef000-7f9dbe9f1000 rw-p 001bc000 fd:00 33709255                   /usr/lib64/libc-2.17.so
7f9dbe9f1000-7f9dbe9f6000 rw-p 00000000 00:00 0 
7f9dbe9f6000-7f9dbea0b000 r-xp 00000000 fd:00 37251501                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f9dbea0b000-7f9dbec0a000 ---p 00015000 fd:00 37251501                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f9dbec0a000-7f9dbec0b000 r--p 00014000 fd:00 37251501                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f9dbec0b000-7f9dbec0c000 rw-p 00015000 fd:00 37251501                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f9dbec0c000-7f9dbed0d000 r-xp 00000000 fd:00 33780129                   /usr/lib64/libm-2.17.so
7f9dbed0d000-7f9dbef0c000 ---p 00101000 fd:00 33780129                   /usr/lib64/libm-2.17.so
7f9dbef0c000-7f9dbef0d000 r--p 00100000 fd:00 33780129                   /usr/lib64/libm-2.17.so
7f9dbef0d000-7f9dbef0e000 rw-p 00101000 fd:00 33780129                   /usr/lib64/libm-2.17.so
7f9dbef0e000-7f9dbeff7000 r-xp 00000000 fd:00 33608209                   /usr/lib64/libstdc++.so.6.0.19
7f9dbeff7000-7f9dbf1f7000 ---p 000e9000 fd:00 33608209                   /usr/lib64/libstdc++.so.6.0.19
7f9dbf1f7000-7f9dbf1ff000 r--p 000e9000 fd:00 33608209                   /usr/lib64/libstdc++.so.6.0.19
7f9dbf1ff000-7f9dbf201000 rw-p 000f1000 fd:00 33608209                   /usr/lib64/libstdc++.so.6.0.19
7f9dbf201000-7f9dbf216000 rw-p 00000000 00:00 0 
7f9dbf216000-7f9dbf237000 r-xp 00000000 fd:00 33699482                   /usr/lib64/ld-2.17.so
7f9dbf404000-7f9dbf40a000 rw-p 00000000 00:00 0 
7f9dbf435000-7f9dbf437000 rw-p 00000000 00:00 0 
7f9dbf437000-7f9dbf438000 r--p 00021000 fd:00 33699482                   /usr/lib64/ld-2.17.so
7f9dbf438000-7f9dbf439000 rw-p 00022000 fd:00 33699482                   /usr/lib64/ld-2.17.so
7f9dbf439000-7f9dbf43a000 rw-p 00000000 00:00 0 
7ffe3ce5b000-7ffe3ce7d000 rw-p 00000000 00:00 0                          [stack]
7ffe3cf10000-7ffe3cf12000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted (core dumped)

[ssilverman@scottc7virt icctest]$ ./test.gcc700 
*** Error in `./test.gcc700': double free or corruption (fasttop): 0x000000000166a0a0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7c619)[0x7fec8a577619]
./test.gcc700[0x400f5d]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fec8a51cc05]
./test.gcc700[0x400be9]
======= Memory map: ========
00400000-00404000 r-xp 00000000 fd:02 16728113                           /home/ssilverman/icctest/test.gcc700
00603000-00604000 r--p 00003000 fd:02 16728113                           /home/ssilverman/icctest/test.gcc700
00604000-00605000 rw-p 00004000 fd:02 16728113                           /home/ssilverman/icctest/test.gcc700
0166a000-0168b000 rw-p 00000000 00:00 0                                  [heap]
7fec84000000-7fec84021000 rw-p 00000000 00:00 0 
7fec84021000-7fec88000000 ---p 00000000 00:00 0 
7fec8a2f7000-7fec8a2f9000 r-xp 00000000 fd:00 33780127                   /usr/lib64/libdl-2.17.so
7fec8a2f9000-7fec8a4f9000 ---p 00002000 fd:00 33780127                   /usr/lib64/libdl-2.17.so
7fec8a4f9000-7fec8a4fa000 r--p 00002000 fd:00 33780127                   /usr/lib64/libdl-2.17.so
7fec8a4fa000-7fec8a4fb000 rw-p 00003000 fd:00 33780127                   /usr/lib64/libdl-2.17.so
7fec8a4fb000-7fec8a6b3000 r-xp 00000000 fd:00 33709255                   /usr/lib64/libc-2.17.so
7fec8a6b3000-7fec8a8b3000 ---p 001b8000 fd:00 33709255                   /usr/lib64/libc-2.17.so
7fec8a8b3000-7fec8a8b7000 r--p 001b8000 fd:00 33709255                   /usr/lib64/libc-2.17.so
7fec8a8b7000-7fec8a8b9000 rw-p 001bc000 fd:00 33709255                   /usr/lib64/libc-2.17.so
7fec8a8b9000-7fec8a8be000 rw-p 00000000 00:00 0 
7fec8a8be000-7fec8a8d3000 r-xp 00000000 fd:00 37251501                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fec8a8d3000-7fec8aad2000 ---p 00015000 fd:00 37251501                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fec8aad2000-7fec8aad3000 r--p 00014000 fd:00 37251501                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fec8aad3000-7fec8aad4000 rw-p 00015000 fd:00 37251501                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fec8aad4000-7fec8abd5000 r-xp 00000000 fd:00 33780129                   /usr/lib64/libm-2.17.so
7fec8abd5000-7fec8add4000 ---p 00101000 fd:00 33780129                   /usr/lib64/libm-2.17.so
7fec8add4000-7fec8add5000 r--p 00100000 fd:00 33780129                   /usr/lib64/libm-2.17.so
7fec8add5000-7fec8add6000 rw-p 00101000 fd:00 33780129                   /usr/lib64/libm-2.17.so
7fec8add6000-7fec8aebf000 r-xp 00000000 fd:00 33608209                   /usr/lib64/libstdc++.so.6.0.19
7fec8aebf000-7fec8b0bf000 ---p 000e9000 fd:00 33608209                   /usr/lib64/libstdc++.so.6.0.19
7fec8b0bf000-7fec8b0c7000 r--p 000e9000 fd:00 33608209                   /usr/lib64/libstdc++.so.6.0.19
7fec8b0c7000-7fec8b0c9000 rw-p 000f1000 fd:00 33608209                   /usr/lib64/libstdc++.so.6.0.19
7fec8b0c9000-7fec8b0de000 rw-p 00000000 00:00 0 
7fec8b0de000-7fec8b0ff000 r-xp 00000000 fd:00 33699482                   /usr/lib64/ld-2.17.so
7fec8b2cc000-7fec8b2d2000 rw-p 00000000 00:00 0 
7fec8b2fd000-7fec8b2ff000 rw-p 00000000 00:00 0 
7fec8b2ff000-7fec8b300000 r--p 00021000 fd:00 33699482                   /usr/lib64/ld-2.17.so
7fec8b300000-7fec8b301000 rw-p 00022000 fd:00 33699482                   /usr/lib64/ld-2.17.so
7fec8b301000-7fec8b302000 rw-p 00000000 00:00 0 
7ffec8c7f000-7ffec8ca1000 rw-p 00000000 00:00 0                          [stack]
7ffec8d11000-7ffec8d13000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted (core dumped)

[Viet_H_Intel]

Hi,

Not sure what happened in your ENV, but I couldn't be able to reproduce it.

vahoang@orcsle150:/tmp$ cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

vahoang@orcsle150:/tmp$ gcc -v

... ... ...

gcc version 6.3.0 (GCC)

vahoang@orcsle150:/tmp$ rm a.out ;icpc test.cpp -O2 ; ./a.out
vahoang@orcsle150:/tmp$

vahoang@orcsle150:/tmp$ gcc -v
... ... ...
gcc version 7.1.0 (GCC)

vahoang@orcsle150:/tmp$ rm a.out ;icpc test.cpp -O2 ; ./a.out
vahoang@orcsle150:/tmp$

vahoang@orcsle150:/tmp$ icpc -V
Intel(R) C++ Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 18.0.1.163 Build 20171018
Copyright (C) 1985-2017 Intel Corporation.  All rights reserved.

ahoang@orcsle150:/tmp$ cat test.cp

#include <string>

std::string a() {
return "aaa";
}
std::string b() {
return std::string("bbbbbbbb") + std::string("c");
/* return std::string("bbbbbbb") + std::string("c"); */
}
int main( int argc, char *argv[] ) {
std::string s = a() + b();
return 0;
}

[Scott_S_1]

Are you using devtoolset-6 / devtoolset-7 as described?

yum install centos-release-scl

yum install devtoolset-6

scl enable devtooolset-6 "bash" or "icpc test.cpp; ./a.out"

[Viet_H_Intel]

Hi Scott,

I might not have the same system as yours. Can you please attached the problematic binary so that I can try to run on my system?

Thanks,

Viet

[Scott_S_1]

Sure, three attachments:

test.gcc485 (Compiled with default CentOS libraries)

test.gcc631 (Compiled under devtoolset-6)

test.gcc700 (Compiled under devtoolset-7)

[Viet_H_Intel]

Hi Scott,

I saw something in your binaries like:

 400f60:       41 8b 55 00             mov    0x0(%r13),%edx
 400f64:       8d 42 ff                    lea    -0x1(%rdx),%eax
 400f67:       41 89 45 00             mov    %eax,0x0(%r13)

These 0x0 offsets should have been resolved at final link. Not sure why they weren't 

I need to ask my lab Technician to set up an identical system to investigate it further.

Thanks,

Viet 

[Scott_S_1]

Great, glad you were able to find something. As I said earlier, it should be pretty simple to set up an identical system. From a base CentOS 7 install, you just need "yum install centos-release-scl; yum install devtoolset-6; yum install devtoolset-7".

If there's anything else I can do to help troubleshoot, don't hesitate to ask. I have many users eager to see this resolved.

[Viet_H_Intel]

I was able to reproduce the issue after my team-mate setup the system and filed a bug to our developer. The internal bug number is CMPLRS-49462. I will keep you posted.

Thanks,

Viet

[Scott_S_1]

Viet,

It has been about a month since your last update, confirming you could reproduce the issue. Has any progress been made in resolving it?

[Viet_H_Intel]

Hi Scott,

As far as I see, no progress have been made yet. 

Regards,

Viet 

[Scott_S_1]

Is it possible to escalate the issue somehow? I have several fully paid licenses that are supposed to receive "priority support" (unfortunately, I have an open ticket there which has not received much attention).

[Viet_H_Intel]

I will try to escalate it. Can I have the other ticket number to see what is going there as well?

Thanks,

Viet 

[Scott_S_1]

• Request03157744
• Created ByScott
• StatusOpen
• Created12/11/2017 8:38 AM
• Products or ServicesIntel® C++ Compiler for Linux*

[Viet_H_Intel]

Thanks Scott,

I've escalated your issue. I will update you once I hear from our developer.

Regards,

Viet

[Scott_S_1]

It has been another 10 days, anything to report? I'm starting to have some serious problems as a result of being stuck on 2018 pre-update1.

[Viet_H_Intel]

Hi Scott,

We've identified a bug in our compiler and it's being fixed.

Thanks,

Viet  

[Scott_S_1]

I've just installed 2018 update 3, and it seems this bug is still present. Is there an ETA on the fix for this? It has been nearly 6 months since my original ticket was opened.

[Viet_H_Intel]

Hi Scott,

The fix didn't make it in time for 18.0 Update 3 code cut off, but it will be fix in the next 18.0 update.

Regards,

Viet