How to link hugetlbfs to icc

[haitaoxiaojun]

I am doing some performance test for Intel 64bit x86 system. We have Itel compiler 'icc', but when I tried to link icc with hugetlbfs, such as:
icc -o latency -m64 latency.o walk.o -L../tools -l_tools -lnuma -B /usr/local/share/libhugetlbfs -Wl,-hugetlbfs-align
ipo: warning #11015: Warning unknown option --hugetlbfs-align
/usr/bin/ld: latency: section .ctors vma 0xfffffffffffffdb8 overlaps previous sections
/usr/bin/ld: latency: section .dtors vma 0xfffffffffffffdc8 overlaps previous sections
/usr/bin/ld: latency: section .jcr vma 0xfffffffffffffdd8 overlaps previous sections
/usr/bin/ld: latency: section .dynamic vma 0xfffffffffffffde0 overlaps previous sections
/usr/bin/ld: latency: section .got vma 0xffffffffffffffd0 overlaps previous sections
/usr/bin/ld: latency: section .got.plt vma 0xffffffffffffffe8 overlaps previous sections
/usr/bin/ld: latency: section `.ctors' can't be allocated in segment 3
LOAD: .interp .note.ABI-tag .note.SuSE .hash .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt .init .plt .text .fini .rodata .eh_frame_hdr .eh_frame .ctors .dtors .jc
r .dynamic .got .got.plt
/usr/bin/ld: final link failed: Bad value
make: *** [latency] Error 1

..

The problem is -hugetlbfs-align. hugetlbfs is installed and it is worked with 'cc'. How to fix it?

Thanks!
Haitao

[TimP]

Some time ago, we started on a project to evaluate hugetlbfs with Intel compilers. This was considered feasible for OpenSuSE but not for RH. The malloc() part of glibc was rebuilt against hugetlbfs, so that any use of new[] or malloc() took [dis]advantage of hugetlbfs. That works the same with icc or gcc. It's conceivable you could find some applications which work well in that mode, but in practice it didn't work well when a huge page had to be swapped, in spite of demonstrating more than expected reduction on tlb miss. The decision makers were so disappointed that it wasn't a no-brainer fix that the project was dropped. Current CPUs have improved tlb behavior so that it shouldn't be so critical whether you use a stock malloc() or a hugetlb one.
cc often is a symlink to gcc. Maybe you mean some other proprietary compiler.
Did you shut off the ipo option for icc? I suppose you didn't use the analogous LTO option for gcc. The reference to ipo is worrisome, but it may simply mean that icc is searching for ipo objects when building the ld script. You might want to check what is going in the ld script (by setting -#).
You could try using gcc to link your non-ipo icc .o files, by specifically including the necessary icc library references (static or dynamic choice preferably same as you made before).

[Judith_W_Intel]

I think -Wl,-hugetlbfs-align should be -Wl,-lhugetlbfs-align

(note the missing l before the library name).

Otherwise -hugetbfs-align is the name of the option told instead of -l.


Judy

[haitaoxiaojun]

-Wl,--hugetlbfs-align (double dash) NOT -Wl,-hugetlbfs-align were what I use for 'cc'. It does not work for 'icc'. '-Wl,-lhugetlbfs-align' did not work either. -Wl,-hugetlbfs-align (one dash) did work for both 'cc' and 'icc' but I am not sure whether it is what I want.

Thanks!
Haitao

[Kittur_G_Intel]

Hi Haitao,
Just letting you know I'll also follow up with our product developementteam folks on this offline and will update you accordingly.
-regards,
Kittur

[Kittur_G_Intel]

Hi Haito,

The -Wl,-hugetlbfs-align is what's recommended and that's working for now. This is the update I have so far, just FYI....

-regards,
Kittur

[arthas_kang]

you must disable ipo option.  icc can't pass --hugetlbfs-align option to libhugetlbfs ld when use object files.

Example:

icc -ggdb -gdwarf-3 -c RegExp.c

icc RegExp.o -o RegExp.static -no-ipo -ggdb -gdwarf-3  -B/usr/share/libhugetlbfs/ -Wl,--hugetlbfs-align

Single source file example:

icc RegExp.c -o RegExp -ggdb -gdwarf-3  -B/usr/share/libhugetlbfs/ -Wl,--hugetlbfs-align -lippch

[Kittur_G_Intel]

Thanks, that's correct since icc will search for intermediate objects which won't be found. I'll pass on your feedback to the product team accordingly.

_kittur

[Mark_D_9]

Is there now a version of icc which allows compiling/linking with '-B/usr/share/libhugetlbfs/ -Wl,--hugetlbfs-align' w/o requiring that I disable inter-procedural optimization with '-no-ipo'?

[TimP]

For several years, linux distributions have implemented transparent huge pages, which often is superior to working with hugetlbfs, as well as happening automatically.  On an application I worked on recently, this produces a 25% performance improvement on a Nehalem test platform.  The next-page hardware prefetch introduced in Haswell CPUs should reduce the importance of hugetlb.