TBB cross compilation on ARM architecture

[sammmas]

Recently, I want to use TBB on a development board with an ARM multi-core processor. Although TBB is not officially announced to be ported on ARM architecture, I got a patch (written by Mr. Raf Schietekat) that seems to port TBB on ARM architecture.

My cross compilation environment is as follows:
Host: OpenSUSE Linux 11 64bit (IA64)
Target:Embedded Linux (ARM)
Cross Compiler:Sourcery G++ Lite 2009q1-203 for ARM GNU/Linux(http://www.codesourcery.com/sgpp/lite/arm/portal/release858)
TBB version: tbb21_20080825osstbb21_20081019osstbb21_20090511oss
TBB patch version: atomic.Raf_Schietekat.20090402

My cross compilation steps are as follows:
1. In Line 42 of linux.gcc.inc, I modified "CPLUS = g++" into "CPLUS = /home/sammmas/ARMCrossCompiler/bin/arm-none-linux-gnueabi-c++"
2. type "make arch=arm" (I know "arm" is not defined in the makefile)

However, when I cross-compiled the TBB source files (patched successfully), I got the following error messages.

[cpp]cd "./build/linux_arm_gcc_cc4.3.2_libc2.9_kernel2.6.27.5_release" && sh ../../build/generate_tbbvars.sh linux_arm_gcc_cc4.3.2_libc2.9_kernel2.6.27.5_release
cd "./build/linux_arm_gcc_cc4.3.2_libc2.9_kernel2.6.27.5_debug" && sh ../../build/generate_tbbvars.sh linux_arm_gcc_cc4.3.2_libc2.9_kernel2.6.27.5_debug
make -C "./build/linux_arm_gcc_cc4.3.2_libc2.9_kernel2.6.27.5_debug"  -r -f ../../build/Makefile.tbb cfg=debug tbb_root=../..
make[1]: Entering directory `/home/creteken/tbb/tbb21_20080605oss/build/linux_arm_gcc_cc4.3.2_libc2.9_kernel2.6.27.5_debug'
../../build/Makefile.tbb:39: CONFIG: cfg=debug arch=arm compiler=gcc os=linux runtime=cc4.3.2_libc2.9_kernel2.6.27.5
/home/creteken/CodeSourcery/Sourcery_G++_Lite/bin/arm-none-linux-gnueabi-g++ -c -MMD -DTBB_DO_ASSERT -DDO_ITT_NOTIFY -g -O0 -DUSE_PTHREAD -fPIC -D__TBB_BUILD=1 -Wall -Wno-parentheses  -I../../src -I../../include ../../src/tbb/concurrent_hash_map.cpp
In file included from ../../include/tbb/tbb_machine.h:699,
                 from ../../include/tbb/atomic.h:102,
                 from ../../include/tbb/spin_rw_mutex.h:33,
                 from ../../include/tbb/concurrent_hash_map.h:40,
                 from ../../src/tbb/concurrent_hash_map.cpp:29:
../../include/tbb/machine/gcc_arm.h: In static member function 'static bool __TBB_CompareStore<1u, signed char, M>::op(volatile void*, int8_t&, int8_t)':
../../include/tbb/machine/gcc_arm.h:135: error: 'atomic_word' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:135: error: '::unsignedword' has not been declared
../../include/tbb/machine/gcc_arm.h:135: error: expected `;' before 'ucm'
../../include/tbb/machine/gcc_arm.h:135: error: 'indended' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:135: error: 'ucm' was not declared in this scope
../../include/tbb/machine/gcc_arm.h: In static member function 'static int8_t __TBB_CompareAndSwap<1u, signed char, M>::op(volatile void*, int8_t, int8_t)':
../../include/tbb/machine/gcc_arm.h:135: error: 'atomic_word' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:135: error: '::unsignedword' has not been declared
../../include/tbb/machine/gcc_arm.h:135: error: expected `;' before 'ucm'
../../include/tbb/machine/gcc_arm.h:135: error: 'ucm' was not declared in this scope
../../include/tbb/machine/gcc_arm.h: In static member function 'static bool __TBB_CompareStore<2u, short int, M>::op(volatile void*, int16_t&, int16_t)':
../../include/tbb/machine/gcc_arm.h:136: error: 'atomic_word' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:136: error: '::unsignedword' has not been declared
../../include/tbb/machine/gcc_arm.h:136: error: expected `;' before 'ucm'
../../include/tbb/machine/gcc_arm.h:136: error: 'indended' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:136: error: 'ucm' was not declared in this scope
../../include/tbb/machine/gcc_arm.h: In static member function 'static int16_t __TBB_CompareAndSwap<2u, short int, M>::op(volatile void*, int16_t, int16_t)':
../../include/tbb/machine/gcc_arm.h:136: error: 'atomic_word' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:136: error: '::unsignedword' has not been declared
../../include/tbb/machine/gcc_arm.h:136: error: expected `;' before 'ucm'
../../include/tbb/machine/gcc_arm.h:136: error: 'ucm' was not declared in this scope
../../include/tbb/machine/gcc_arm.h: In static member function 'static bool __TBB_CompareStore<4u, int, M>::op(volatile void*, int32_t&, int32_t)':
../../include/tbb/machine/gcc_arm.h:137: error: 'atomic_word' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:137: error: '::unsignedword' has not been declared
../../include/tbb/machine/gcc_arm.h:137: error: expected `;' before 'ucm'
../../include/tbb/machine/gcc_arm.h:137: error: 'indended' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:137: error: 'ucm' was not declared in this scope
../../include/tbb/machine/gcc_arm.h: In static member function 'static int32_t __TBB_CompareAndSwap<4u, int, M>::op(volatile void*, int32_t, int32_t)':
../../include/tbb/machine/gcc_arm.h:137: error: 'atomic_word' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:137: error: '::unsignedword' has not been declared
../../include/tbb/machine/gcc_arm.h:137: error: expected `;' before 'ucm'
../../include/tbb/machine/gcc_arm.h:137: error: 'ucm' was not declared in this scope
make[1]: *** [concurrent_hash_map.o] Error 1
make[1]: Leaving directory `/home/creteken/tbb/tbb21_20080605oss/build/linux_arm_gcc_cc4.3.2_libc2.9_kernel2.6.27.5_debug'
make: *** [tbb] Error 2
[/cpp]

The problem seems to be that the definitions are in wrong scopes.
Where should I put the definitions? In namespace tbb? Or in the global namespace?
I tried to move the definitions to the global namespace and the above error messages disappear, but other scope problems come!!
Can someone tell me how to get rid of the above error messages?
Thank you very much!!

[RafSchietekat]

I don't know why these link errors occur, but I see nothing platform-specific in these error messages, so a lot of people should be able to help here.

[clhjoe]

Hi,

I asked the previous in the other thread(http://software.intel.com/en-us/forums/showthread.php?t=70663) and fixed the problem. But when I execute the program which include TBB library will causes the "segmentationfault". And Mr. Alexey said that it might does not work porting to ARM11MPCORE. Is there any ways to fix the problem? Thank you for your so many helps.

[RafSchietekat]

Do you have a core dump file to analyse in a debugger like gdb, or can you run the program inside a debugger and deduce something from the contents of the stack at the time of the segmentation fault? I see no reason why TBB would not be portable to an architecture like ARM. It is conceivable that some example programs or even TBB itself erroneously rely on possibly stronger memory semantics for Intel Architecture, but at this time I would not be looking there yet, and I don't even remember whether that would apply to ARM.

My advice: try out a few simple examples, to see whether at least some things work, and then make them more challenging. When you've got a failing program that works on x86, use a debugger to find out what's happening.

(Added) It should be safe to comment out the following sections in include/tbb/machine/gcc_arm.h: __TBB_FetchOp, __TBB_Op(__TBB_store), __TBB_Load. The implementation will fall back to something more expensive but also more conservative. If this makes a difference, that would be a significant piece of information.

[clhjoe]

Sorry for so late to reply. I was thinking how to monitor the informations during the program executing. I used strace to see what just happened. I executed

$ strace -f -o log test

and then here is the end of the log msg.

[bash]534   set_robust_list(0x40020700, 0xc)  = -1 ENOSYS (Function not implemented)
534   futex(0xbebfde24, FUTEX_WAKE_PRIVATE, 1) = 0
534   futex(0xbebfde24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 40351000) = -1 ENOSYS (Function not implemented)
534   rt_sigaction(SIGRTMIN, {0x403382d0, [], SA_SIGINFO|0x4000000}, NULL, 8) = 0
534   rt_sigaction(SIGRT_1, {0x40338168, [], SA_RESTART|SA_SIGINFO|0x4000000}, NULL, 8) = 0
534   rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
534   getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
534   futex(0x401219f0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
534   futex(0x40121a0c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
534   futex(0x4011bb80, FUTEX_WAKE_PRIVATE, 2147483647) = 0
534   fstat64(1, {st_mode=S_IFCHR|0777, st_rdev=makedev(5, 1), ...}) = 0
534   ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
534   mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40022000
534   futex(0x403300c4, FUTEX_WAKE_PRIVATE, 2147483647) = 0
534   brk(0)                            = 0x12000
534   brk(0x33000)                      = 0x33000
534   open("/lib/tls/v6l/fast-mult/half/libtbbmalloc.so.2", O_RDONLY) = 3
534   read(3, "177ELF111���������3�(�1���334t��004���"..., 512) = 512
534   fstat64(3, {st_mode=S_IFREG|0755, st_size=23837, ...}) = 0
534   mmap2(NULL, 47212, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40364000
534   mprotect(0x40368000, 28672, PROT_NONE) = 0
534   mmap2(0x4036f000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3) = 0x4036f000
534   close(3)                          = 0
534   --- SIGSEGV (Segmentation fault) @ 0 (0) ---
534   +++ killed by SIGSEGV +++
[/bash]

The msg shows "(Function not implemented)" two times. And does it let the program crash? Or is there any other reasons? Thank you.

[RafSchietekat]

strace is not where I would be looking for information; see #23.

[martinbogo]

Raf,

I'm working on building libtbb2 on ARM as well, since it's a dependency of FaceBook's HipHop PHP compiler.

I also looked for your patch on the Developer site and did not locate it for download. Would you be so kind as to repost the patch, or a new patch based on current work that I can try?

I'm building natively, on Ubuntu Karmic (9.10) on the tegra2 ARM Cortex-A9 platform.

-Martin

[RafSchietekat]

The latest though dated version is mentioned in #144 of thread "Additions to atomic", but it's based on a version that is no longer downloadable (it was restored at some point, but is now gone again). I'm not sure whether I should upload the full patched distribution.

Serious sponsorship for an update is welcome, but it's a nontrivial effort (Intel have stopped trying). and there are many competing sources of fun and/or learning...

[vladvic]

Hi,
Please, can I get somewhere the mentioned TBB version and the ARM patch?..

[Alexey-Kukanov]

I re-enabled tbb21_20080925ossfor download again. The link to the patch (which is much more than just ARM port) is in the previous post by Raf S.

[Denis_Bolshakov]

Maybe it's reasonable to locate ARM port to http://www.threadingbuildingblocks.org/file.php?fid=86 (contribution Archive)?

[clhjoe]

I still can't get the core dump info. however, without "-O2" parameter when compiling the library and tbb app , it works.