Performance problem involving mixed gcc/icc compilation

[cfspc]

Greetings,

I have been trying to use ICC on a large application and have come
across some strange performance problems. I am using ICC 11.081,
gcc 4.1.2p2 andLinux Red Hat Enterprise 5.

Iam compiling some of the application modules with ICC and others
with GCC. I also have a reference version of the application that
is compiled only with GCC.

The application works in several "steps" and steps are closely matched
to modules. Some of the modules I compile with ICC are faster than
the modules compiled with GCC. However, I am seeing something very
surprising/disturbing: some of the modules that are compiled with GCC
on both builds are slower in the hybrid ICC/GCC build than in the
reference GCC only build. The performance difference ranges from 5%
to 8%.

Since the modules that are exhibiting performance degradation are
compiled in the same way (using GCC) and they don't make significant
use of any code compiled with ICC I am wondering if this performance
problem can be due to something that happens at link time, maybe
something in the intel libs that is slower than in the gcc version? Any
thoughts/suggestions?

Thanks in advance,

Carlos

[TimP]

You omit a lot of important information. If you call a gcc function from an icc function on 32-bit OS, it doesn't receive a 64-bit aligned stack, as it would when called from a gcc function (unless you set -Os or -mpreferred-stack-boundary).
Typically, Intel math libraries run significantly faster than glibc, at least when an Intel CPU is recognized.

[cfspc]

Quoting - tim18

You omit a lot of important information. If you call a gcc function from an icc function on 32-bit OS, it doesn't receive a 64-bit aligned stack, as it would when called from a gcc function (unless you set -Os or -mpreferred-stack-boundary).
Typically, Intel math libraries run significantly faster than glibc, at least when an Intel CPU is recognized.

Hi Tim,

I am working on a 64 bit OS. In any case we need to build this on 32 bit as well. I will try the
-mpreferred-stack-boundary (Thanks!). Would the stack aligment hurt if I was calling icc
functions from gcc?

That being said, the modules that are being affected are mostly isolated i.e. the whole module
takes a while to run and calls only functions compiled with GCC which is why I was wondering
if by linking with icc/icpc I am somehow replacing some low level lib functions with intel
versions.

Intel math libraries are really much faster than glibc and we see good performance improvements on
the modules compiled with ICC.

Thanks again,

Carlos

[TimP]

32-bit gcc passes an aligned stack to called functions by default, while 32-bit icc does not. Both 32- and 64-bit gcc reduce the stack alignment automatically when -Os is set, in order to save space on stack at the expense of compatibility with auto-vectorization. gcc stack alignment options won't help in interfacing with icc. I pointed this out because both data and code alignment variations may affect performance.
In that connection. ITLB locality might have an effect, and you might have affected it accidentally, most likely if your link order changed. Large database applications, in particular, sometimes benefit from optimizing link order to localize frequently called functions.