TBB cross compilation on ARM architecture

[sammmas]

Recently, I want to use TBB on a development board with an ARM multi-core processor. Although TBB is not officially announced to be ported on ARM architecture, I got a patch (written by Mr. Raf Schietekat) that seems to port TBB on ARM architecture.

My cross compilation environment is as follows:
Host: OpenSUSE Linux 11 64bit (IA64)
Target:Embedded Linux (ARM)
Cross Compiler:Sourcery G++ Lite 2009q1-203 for ARM GNU/Linux(http://www.codesourcery.com/sgpp/lite/arm/portal/release858)
TBB version: tbb21_20080825osstbb21_20081019osstbb21_20090511oss
TBB patch version: atomic.Raf_Schietekat.20090402

My cross compilation steps are as follows:
1. In Line 42 of linux.gcc.inc, I modified "CPLUS = g++" into "CPLUS = /home/sammmas/ARMCrossCompiler/bin/arm-none-linux-gnueabi-c++"
2. type "make arch=arm" (I know "arm" is not defined in the makefile)

However, when I cross-compiled the TBB source files (patched successfully), I got the following error messages.

[cpp]cd "./build/linux_arm_gcc_cc4.3.2_libc2.9_kernel2.6.27.5_release" && sh ../../build/generate_tbbvars.sh linux_arm_gcc_cc4.3.2_libc2.9_kernel2.6.27.5_release
cd "./build/linux_arm_gcc_cc4.3.2_libc2.9_kernel2.6.27.5_debug" && sh ../../build/generate_tbbvars.sh linux_arm_gcc_cc4.3.2_libc2.9_kernel2.6.27.5_debug
make -C "./build/linux_arm_gcc_cc4.3.2_libc2.9_kernel2.6.27.5_debug"  -r -f ../../build/Makefile.tbb cfg=debug tbb_root=../..
make[1]: Entering directory `/home/creteken/tbb/tbb21_20080605oss/build/linux_arm_gcc_cc4.3.2_libc2.9_kernel2.6.27.5_debug'
../../build/Makefile.tbb:39: CONFIG: cfg=debug arch=arm compiler=gcc os=linux runtime=cc4.3.2_libc2.9_kernel2.6.27.5
/home/creteken/CodeSourcery/Sourcery_G++_Lite/bin/arm-none-linux-gnueabi-g++ -c -MMD -DTBB_DO_ASSERT -DDO_ITT_NOTIFY -g -O0 -DUSE_PTHREAD -fPIC -D__TBB_BUILD=1 -Wall -Wno-parentheses  -I../../src -I../../include ../../src/tbb/concurrent_hash_map.cpp
In file included from ../../include/tbb/tbb_machine.h:699,
                 from ../../include/tbb/atomic.h:102,
                 from ../../include/tbb/spin_rw_mutex.h:33,
                 from ../../include/tbb/concurrent_hash_map.h:40,
                 from ../../src/tbb/concurrent_hash_map.cpp:29:
../../include/tbb/machine/gcc_arm.h: In static member function 'static bool __TBB_CompareStore<1u, signed char, M>::op(volatile void*, int8_t&, int8_t)':
../../include/tbb/machine/gcc_arm.h:135: error: 'atomic_word' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:135: error: '::unsignedword' has not been declared
../../include/tbb/machine/gcc_arm.h:135: error: expected `;' before 'ucm'
../../include/tbb/machine/gcc_arm.h:135: error: 'indended' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:135: error: 'ucm' was not declared in this scope
../../include/tbb/machine/gcc_arm.h: In static member function 'static int8_t __TBB_CompareAndSwap<1u, signed char, M>::op(volatile void*, int8_t, int8_t)':
../../include/tbb/machine/gcc_arm.h:135: error: 'atomic_word' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:135: error: '::unsignedword' has not been declared
../../include/tbb/machine/gcc_arm.h:135: error: expected `;' before 'ucm'
../../include/tbb/machine/gcc_arm.h:135: error: 'ucm' was not declared in this scope
../../include/tbb/machine/gcc_arm.h: In static member function 'static bool __TBB_CompareStore<2u, short int, M>::op(volatile void*, int16_t&, int16_t)':
../../include/tbb/machine/gcc_arm.h:136: error: 'atomic_word' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:136: error: '::unsignedword' has not been declared
../../include/tbb/machine/gcc_arm.h:136: error: expected `;' before 'ucm'
../../include/tbb/machine/gcc_arm.h:136: error: 'indended' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:136: error: 'ucm' was not declared in this scope
../../include/tbb/machine/gcc_arm.h: In static member function 'static int16_t __TBB_CompareAndSwap<2u, short int, M>::op(volatile void*, int16_t, int16_t)':
../../include/tbb/machine/gcc_arm.h:136: error: 'atomic_word' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:136: error: '::unsignedword' has not been declared
../../include/tbb/machine/gcc_arm.h:136: error: expected `;' before 'ucm'
../../include/tbb/machine/gcc_arm.h:136: error: 'ucm' was not declared in this scope
../../include/tbb/machine/gcc_arm.h: In static member function 'static bool __TBB_CompareStore<4u, int, M>::op(volatile void*, int32_t&, int32_t)':
../../include/tbb/machine/gcc_arm.h:137: error: 'atomic_word' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:137: error: '::unsignedword' has not been declared
../../include/tbb/machine/gcc_arm.h:137: error: expected `;' before 'ucm'
../../include/tbb/machine/gcc_arm.h:137: error: 'indended' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:137: error: 'ucm' was not declared in this scope
../../include/tbb/machine/gcc_arm.h: In static member function 'static int32_t __TBB_CompareAndSwap<4u, int, M>::op(volatile void*, int32_t, int32_t)':
../../include/tbb/machine/gcc_arm.h:137: error: 'atomic_word' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:137: error: '::unsignedword' has not been declared
../../include/tbb/machine/gcc_arm.h:137: error: expected `;' before 'ucm'
../../include/tbb/machine/gcc_arm.h:137: error: 'ucm' was not declared in this scope
make[1]: *** [concurrent_hash_map.o] Error 1
make[1]: Leaving directory `/home/creteken/tbb/tbb21_20080605oss/build/linux_arm_gcc_cc4.3.2_libc2.9_kernel2.6.27.5_debug'
make: *** [tbb] Error 2
[/cpp]

The problem seems to be that the definitions are in wrong scopes.
Where should I put the definitions? In namespace tbb? Or in the global namespace?
I tried to move the definitions to the global namespace and the above error messages disappear, but other scope problems come!!
Can someone tell me how to get rid of the above error messages?
Thank you very much!!

[RafSchietekat]

atomic.Raf_Schietekat.20090402 is meant to be used with tbb21_20080925oss, but you only mention 3 other versions.

Please apply the following corrections in gcc_arm.h: atomic_word->tbb::internal::atomic_word, indended->intended, and try again.

[sammmas]

Quoting - Raf Schietekat

atomic.Raf_Schietekat.20090402 is meant to be used with tbb21_20080925oss, but you only mention 3 other versions.

Please apply the following corrections in gcc_arm.h: atomic_word->tbb::internal::atomic_word, indended->intended, and try again.

Dear Mr. Raf Schietekat:

Thank you for your reply so soon.
I cannot find TBB version tbb21_20080925 in the TBB official website.
Where can I download tbb21_20080925?
I have one more question.
In addtion to my modification of linux.gcc.inc, is there anything else I should modify if I want to cross-compile TBB on ARM architecture?
Thank you for your reply again.

[RafSchietekat]

I could post an already merged version by tomorrow (bigger file, though, and impossible to see what has changed without the original).

[Alexey-Kukanov · Accepted solution]

Quoting - sammmas

I cannot find TBB version tbb21_20080925 in the TBB official website.
Where can I download tbb21_20080925?

I re-enabled tbb21_20080925 for download. Check the Development Release section.

[sammmas]

Quoting - Raf Schietekat

I could post an already merged version by tomorrow (bigger file, though, and impossible to see what has changed without the original).

Dear Mr. Raf Schietekat:

Thank you for your kindly help.
I will try again with your merged version.

sammmas

[sammmas]

Quoting - Alexey Kukanov (Intel)

I re-enabled tbb21_20080925 for download. Check the Development Release section.

Dear Mr. Alexey Kukanov:

I will download tbb21_20080925 and patch it.
Thank you very much!

sammmas

[sammmas]

Quoting - Raf Schietekat

atomic.Raf_Schietekat.20090402 is meant to be used with tbb21_20080925oss, but you only mention 3 other versions.

Please apply the following corrections in gcc_arm.h: atomic_word->tbb::internal::atomic_word, indended->intended, and try again.

Dear Raf Schietekat:

I downloaded tbb21_20080925oss_src from the TBB official websit and patch it successfully.
I also applied the corrections you mentioned in gcc_arm.h: atomic_word --> tbb::internal::atomic_word, indended --> indended.
I compiled the patched and modified tbb21_20080925oss_src and got the following error messages:

[cpp]cd "./build/linux_arm_gcc_cc4.3.1_libc2.8_kernel2.6.25.20_release" && sh ../../build/generate_tbbvars.sh linux_arm_gcc_cc4.3.1_libc2.8_kernel2.6.25.20_release
cd "./build/linux_arm_gcc_cc4.3.1_libc2.8_kernel2.6.25.20_debug" && sh ../../build/generate_tbbvars.sh linux_arm_gcc_cc4.3.1_libc2.8_kernel2.6.25.20_debug
make -C "./build/linux_arm_gcc_cc4.3.1_libc2.8_kernel2.6.25.20_debug"  -r -f ../../build/Makefile.tbb cfg=debug tbb_root=../..
make[1]: Entering directory `/home/sammmas/TBB/tbb21_20080925oss/build/linux_arm_gcc_cc4.3.1_libc2.8_kernel2.6.25.20_debug'
../../build/Makefile.tbb:39: CONFIG: cfg=debug arch=arm compiler=gcc os=linux runtime=cc4.3.1_libc2.8_kernel2.6.25.20
/home/sammmas/ARMCrossCompiler/bin/arm-none-linux-gnueabi-g++ -c -MMD -DTBB_DO_ASSERT -DDO_ITT_NOTIFY -g -O0 -DUSE_PTHREAD -fPIC -D__TBB_BUILD=1 -Wall -Wno-parentheses  -I../../src -I../../include ../../src/tbb/concurrent_hash_map.cpp
In file included from ../../include/tbb/tbb_machine.h:699,
                 from ../../include/tbb/atomic.h:102,
                 from ../../include/tbb/spin_rw_mutex.h:33,
                 from ../../include/tbb/concurrent_hash_map.h:40,
                 from ../../src/tbb/concurrent_hash_map.cpp:29:
../../include/tbb/machine/gcc_arm.h:104: error: 'S' was not declared in this scope
../../include/tbb/machine/gcc_arm.h:104: error: template argument 1 is invalid
../../include/tbb/machine/gcc_arm.h:104: error: expected initializer before 'ucm'
../../include/tbb/machine/gcc_arm.h:106: error: expected `)' before 'X'
../../include/tbb/machine/gcc_arm.h:114: error: expected unqualified-id before 'if'
../../include/tbb/machine/gcc_arm.h:115: error: expected unqualified-id before 'return'
../../include/tbb/machine/gcc_arm.h:116: error: expected declaration before '}' token
make[1]: *** [concurrent_hash_map.o] Error 1
make[1]: Leaving directory `/home/sammmas/TBB/tbb21_20080925oss/build/linux_arm_gcc_cc4.3.1_libc2.8_kernel2.6.25.20_debug'
make: *** [tbb] Error 2
[/cpp]

This time, the error seems to be the problem of template argument deduction.
Does it mean that the style of template argument deduction is not supported by the cross-compiler I use?
How can I get rid of the above error message?
Thank you very much for your kindly help!

[sammmas]

Quoting - Raf Schietekat

atomic.Raf_Schietekat.20090402 is meant to be used with tbb21_20080925oss, but you only mention 3 other versions.

Please apply the following corrections in gcc_arm.h: atomic_word->tbb::internal::atomic_word, indended->intended, and try again.

Dear Raf Schietekat:

I downloaded tbb21_20080925oss_src from the TBB official websit and patched it successfully. I also applied the corrections you mentioned in gcc_arm.h, and the error messages I mentioned last time do disappear. However, there is a problem that should be clarified, I think. In the makefile, there exists a variable arch, which might cause a cross compilatioin problem. I traced the makefile and found that there does not exist any piece of codes corresponding to ARM architecture. In order not to use the wrong value of arch (I don't want to use the value of the host platform; I want to use the value of the target platform, i.e., ARM), I specified the value of arch explicitly to "arm". When I compiled the source code, I typed "make arch=arm" in the command line and got the following error messages:

[cpp]cd "./build/linux_arm_gcc_cc4.3.1_libc2.8_kernel2.6.25.20_release" && sh ../../build/generate_tbbvars.sh linux_arm_gcc_cc4.3.1_libc2.8_kernel2.6.25.20_release
cd "./build/linux_arm_gcc_cc4.3.1_libc2.8_kernel2.6.25.20_debug" && sh ../../build/generate_tbbvars.sh linux_arm_gcc_cc4.3.1_libc2.8_kernel2.6.25.20_debug
make -C "./build/linux_arm_gcc_cc4.3.1_libc2.8_kernel2.6.25.20_debug"  -r -f ../../build/Makefile.tbb cfg=debug tbb_root=../..
make[1]: Entering directory `/home/sammmas/TBB/tbb21_20080925oss/build/linux_arm_gcc_cc4.3.1_libc2.8_kernel2.6.25.20_debug'
../../build/Makefile.tbb:39: CONFIG: cfg=debug arch=arm compiler=gcc os=linux runtime=cc4.3.1_libc2.8_kernel2.6.25.20
/home/sammmas/ARMCrossCompiler/bin/arm-none-linux-gnueabi-g++ -c -MMD -DTBB_DO_ASSERT -DDO_ITT_NOTIFY -g -O0 -DUSE_PTHREAD -fPIC -D__TBB_BUILD=1 -Wall -Wno-parentheses  -I../../src -I../../include ../../src/tbb/concurrent_queue.cpp
/tmp/ccfvZlBP.s: Assembler messages:
/tmp/ccfvZlBP.s:1932: Error: selected processor does not support `ldrex ip,[r0]'
/tmp/ccfvZlBP.s:1935: Error: selected processor does not support `strex lr,r2,[r0]'
/tmp/ccfvZlBP.s:2534: Error: selected processor does not support `ldrex r0,[r2]'
/tmp/ccfvZlBP.s:2536: Error: selected processor does not support `strex ip,ip,[r2]'
make[1]: *** [concurrent_queue.o] Error 1
make[1]: Leaving directory `/home/sammmas/TBB/tbb21_20080925oss/build/linux_arm_gcc_cc4.3.1_libc2.8_kernel2.6.25.20_debug'
make: *** [tbb] Error 2
[/cpp]

Does it mean that I should modify the makefile?
If yes, where should I modify in the makefile?
Thank you very much!

[RafSchietekat]

I haven't looked into setting things up for clean cross-compilation yet, but let's first worry about actually compiling the code: maybe you have to use -mcpu or -march or somesuch for your compiler to know that it has a V6 or better to work with?

(Added) Compare with SunOS.gcc.inc, where I specified -mcpu=v9 for SPARC.

[clhjoe]

Quoting - Raf Schietekat

I haven't looked into setting things up for clean cross-compilation yet, but let's first worry about actually compiling the code: maybe you have to use -mcpu or -march or somesuch for your compiler to know that it has a V6 or better to work with?

(Added) Compare with SunOS.gcc.inc, where I specified -mcpu=v9 for SPARC.

HI,I got "/tmp/ccEY8K6G.s:7389: Error: registers may not be the same -- `strex ip,ip,[r2]'" message after I specified -mcpu=mpcore for ARM11MPCORE. I'm not familiar with assembly, so what should I modify "strex" X " %[itd],%[itd],[%[ptr]]nt" /* store conditionally */ to or others? Thank you.

[RafSchietekat]

Could youspecify -save-temps for a bit more context?

(Correction) Not a double but a single dash at the start of the compiler option.

[clhjoe]

Quoting - Raf Schietekat

Could youspecify -save-temps for a bit more context?

(Correction) Not a double but a single dash at the start of the compiler option.

Dear Raf Schietekat,
I got the error meesage when compile concurrent_queue.cpp. So I add -save-temps to compile it again and got three files concurrent_queue.ii ,concurrent_queue.s and concurrent_queue.d. please check with attachments. Thank you for your help.

[RafSchietekat]

Hmm, those files do not go together with the error message you provided earlier. But I'll have a look anyway.

(Added) Maybe I didn't really really need those files, but they did provide some inspiration and made it easier to spot and confirm the problem. I didn't see this restriction (that the registers have to be different) in the specification document, and still don't, but I'll post a new proposal shortly. Meanwhile, if you don't want to wait for that, you can probably comment out the whole __TBB_FetchOp section (lines 172-224 or so) in include/tbb/machine/gcc_arm.h; just don't trust any benchmarks you may make as final yet.

[clhjoe]

Quoting - Raf Schietekat

Hmm, those files do not go together with the error message you provided earlier. But I'll have a look anyway.

(Added) Maybe I didn't really really need those files, but they did provide some inspiration and made it easier to spot and confirm the problem. I didn't see this restriction (that the registers have to be different) in the specification document, and still don't, but I'll post a new proposal shortly. Meanwhile, if you don't want to wait for that, you can probably comment out the whole __TBB_FetchOp section (lines 172-224 or so) in include/tbb/machine/gcc_arm.h; just don't trust any benchmarks you may make as final yet.

Hi,
I have looked into http://www.keil.com/support/man/docs/armasm/armasm_cihbghef.htm . And it says "For STREX, Rd must not be the same register as Rt, Rt2, or Rn." I will follow your suggestion. Thank you very much.

[RafSchietekat]

"I have looked into http://www.keil.com/support/man/docs/armasm/armasm_cihbghef.htm . And it says "For STREX, Rd must not be the same register as Rt, Rt2, or Rn.""
I still don't find that where I was looking, but no matter.

Please try the attached consolidated include/tbb/machine/gcc_arm.h on top of the 2009-04-02 patch.

(Added 2009-12-07) Note that 64-bit atomics, currently relying on default delegation to __TBB_CompareAndSwap (at an unknown performance penalty relative todirect implementations), will probably require a substitution "r8"->"r8,r9" (without the quotes) at gcc_arm.h:165 and"r12"->"r12,r13" (same remark) at gcc_arm.h:170, based on the apparently more reliable specification referenced earlier, which is kinda relevant because my implementation is only based on specification and the feedback I get here from anyone actually building from the code. But we'll get there in the end! :-)

[clhjoe]

Quoting - Raf Schietekat

"I have looked into http://www.keil.com/support/man/docs/armasm/armasm_cihbghef.htm . And it says "For STREX, Rd must not be the same register as Rt, Rt2, or Rn.""
I still don't find that where I was looking, but no matter.

Please try the attached consolidated include/tbb/machine/gcc_arm.h on top of the 2009-04-02 patch.

(Added 2009-12-07) Note that 64-bit atomics, currently relying on default delegation to __TBB_CompareAndSwap (at an unknown performance penalty relative todirect implementations), will probably require a substitution "r8"->"r8,r9" (without the quotes) at gcc_arm.h:165 and"r12"->"r12,r13" (same remark) at gcc_arm.h:170, based on the apparently more reliable specification referenced earlier, which is kinda relevant because my implementation is only based on specification and the feedback I get here from anyone actually building from the code. But we'll get there in the end! :-)

I have compile it successfully. Thank you for you kindly help.

[RafSchietekat]

"I have compile it successfully."
With or without the late-breaking suggestion for 64-bit support? Have you also successfully run the tests ("make all" would at some point run test_atomic)? Please also report back if anything fails later on, because tests (can) only do so much (test_atomic does not actually test atomics across threads).

[clhjoe]

Quoting - Raf Schietekat

"I have compile it successfully."
With or without the late-breaking suggestion for 64-bit support? Have you also successfully run the tests ("make all" would at some point run test_atomic)? Please also report back if anything fails later on, because tests (can) only do so much (test_atomic does not actually test atomics across threads).

I'm sorry for so late to report. I can successfully do "make" with and without the late-breaking suggestion for 64-bit support . However, when I type "make all",every examples are failed. I useexamples/test_all/fibonacci as sample, I directly execute "arm-unknown-linux-gnueabi-g++ -mcpu=mpcore -O2 -DNDEBUG -o fibonacci Fibonacci.cpp -ltbb -I /opt/tbb-port/basedir/include -L /opt/tbb-port/basedir/build/linux_mpcore_arm-unknown-linux-gnueabi-gcc_cc4.2.4_libc2.10.1_kernel2.6.31_release" and it shows in the attachments. Thank you.

[RafSchietekat]

Could it be that tbb_error.txt is empty?

[clhjoe]

Quoting - Raf Schietekat

Could it be that tbb_error.txt is empty?

I'm so sorry. I update the file again. Thank you.