Compiling TBB for Android

[gfour]

I am trying to run TBB on a Tegra 3 (quad-core) tablet running Android 4.

I made some modifications to make the libray compile:
- Android doesn't provide cpu_set_t, so I used the sched.h from http://source-android.frandroid.com/bionic/libc/include/sched.h
- sched_setaffinity/sched_getaffinity are not exposed in header files (although bionic contains them), called them with syscall
- src/tbb/dynamic_link.cpp contains functionality that I do not know how to make it work on Android, so it's empty stubs right now
- the clz instructions (include/tbb/machine/gcc_generic.h) were made 'static inline'
- Android doesn't have posix_memalign, so it was commented out. Can the TBB run without it? Should a simple memalign suffice?

I also commented out the following line in build/linux.gcc.inc (since Android already contains pthreads/rt in bionic):

#LIBS += -lpthread -lrt

and set the compilers of the Android toolchain:

CPLUS = ~/custom/android-toolchain/bin/arm-linux-androideabi-g++
CONLY = ~/custom/android-toolchain/bin/arm-linux-androideabi-gcc

The library seems to compile+link OK for Android 4/Tegra 3, with the following command-line:
make arch=arm compiler=gcc CXXFLAGS="-DTBB_USE_GCC_BUILTINS -march=armv7-a -mfloat-abi=softfp -mfpu=neon -ldl"

Some tests seem to compile OK too, with the following command-line:
make test arch=arm compiler=gcc CXXFLAGS="-DTBB_USE_GCC_BUILTINS -march=armv7-a -mfloat-abi=softfp -mfpu=neon -ldl"

PROBLEMS:

(A) Most tests fail on loading, only 4 load and run OK. The problem seems to be linking. The following output is from an ssh session with the tablet.

./test_ScalableAllocator.exe
reloc_library[1285]: 6308 cannot locate '__cxa_begin_catch'...
CANNOT LINK EXECUTABLE

./test_ScalableAllocator_STL.exe
reloc_library[1285]: 6314 cannot locate '_ZNSt15_List_node_base4hookEPS_'...
CANNOT LINK EXECUTABLE

./test_aligned_space.exe
link_image[1936]: 6325 could not load needed library 'libtbb.so.2' for './test_aligned_space.exe' (reloc_library[1285]: 6325 cannot locate '_ZNSt13runtime_errorC1ERKSs'...
)CANNOT LINK EXECUTABLE

./test_assembly.exe
link_image[1936]: 6326 could not load needed library 'libtbb.so.2' for './test_assembly.exe' (reloc_library[1285]: 6326 cannot locate '_ZNSt13runtime_errorC1ERKSs'...
)CANNOT LINK EXECUTABLE

./test_fast_random.exe
reloc_library[1285]: 6337 cannot locate '_ZNSt13runtime_errorC1ERKSs'...
CANNOT LINK EXECUTABLE

/test_malloc_atexit.exe
link_image[1936]: 6342 could not load needed library 'libtbbmalloc_proxy.so.2' for './test_malloc_atexit.exe' (reloc_library[1285]: 6342 cannot locate '__cxa_call_unexpected'...
)CANNOT LINK EXECUTABLE

/test_malloc_compliance.exe
reloc_library[1285]: 6388 cannot locate '_ZSt20__throw_length_errorPKc'...
CANNOT LINK EXECUTABLE

./test_malloc_init_shutdown.exe
done

./test_malloc_lib_unload.exe
link_image[1936]: 6404 could not load needed library 'test_malloc_lib_unload_dll.so' for './test_malloc_lib_unload.exe' (load_library[1091]: Library 'test_malloc_lib_unload_dll.so' not found)CANNOT LINK EXECUTABLE

/test_malloc_overload.exe
libtbbmalloc not found
fail

(NOTE: the directory contains libtbb.so/libtbbmalloc.so/libtbbmalloc_proxy.so/libtbb.so.2/libtbbmalloc.so.2/libtbbmalloc_proxy.so.2 and LD_LIBRARY_PATH=. , maybe the problem is elsewhere)

./test_malloc_overload_proxy.exe
link_image[1936]: 6450 could not load needed library 'libtbbmalloc_proxy.so.2' for './test_malloc_overload_proxy.exe' (reloc_library[1285]: 6450 cannot locate '__cxa_call_unexpected'...
)CANNOT LINK EXECUTABLE

/test_malloc_pools.exe
reloc_library[1285]: 6462 cannot locate '__gxx_personality_v0'...
CANNOT LINK EXECUTABLE

./test_malloc_pure_c.exe
done

./test_malloc_regression.exe
done

./test_malloc_whitebox.exe
done

./test_task_assertions.exe
reloc_library[1285]: 7055 cannot locate '_ZNSt13runtime_errorC1ERKSs'...
CANNOT LINK EXECUTABLE

./test_task_leaks.exe
reloc_library[1285]: 7058 cannot locate '_ZNSt13runtime_errorC1ERKSs'...
CANNOT LINK EXECUTABLE

(B) The gcc of the Android toolchain seems to have problems with some atomic built-ins and cannot link some of the tests, e.g.:

~/custom/android-toolchain/bin/arm-linux-androideabi-g++ -o test_atomic.exe -DDO_ITT_NOTIFY -O2 -DUSE_PTHREAD -Wall -Wshadow -Wcast-qual -Woverloaded-virtual -Wnon-virtual-dtor -Wextra -DTBB_USE_GCC_BUILTINS -march=armv7-a -mfloat-abi=softfp -mfpu=neon -ldl test_atomic.o libtbb.so -Wl,-rpath-link=.
test_atomic.o: In function `_Z12TestParallelIxEvPKc.clone.6':
test_atomic.cpp:(.text+0x2d60): undefined reference to `__sync_val_compare_and_swap_8'
test_atomic.cpp:(.text+0x3074): undefined reference to `__sync_val_compare_and_swap_8'
test_atomic.cpp:(.text+0x31cc): undefined reference to `__sync_val_compare_and_swap_8'
test_atomic.cpp:(.text+0x34e0): undefined reference to `__sync_val_compare_and_swap_8'
test_atomic.cpp:(.text+0x3638): undefined reference to `__sync_val_compare_and_swap_8'
test_atomic.o:test_atomic.cpp:(.text+0x3950): more undefined references to `__sync_val_compare_and_swap_8' follow
test_atomic.o: In function `void TestFetchAndAdd(unsigned long long)':
test_atomic.cpp:(.text._Z15TestFetchAndAddIyEvT_[void TestFetchAndAdd(unsigned long long)]+0x9c): undefined reference to `__sync_fetch_and_add_8'
test_atomic.cpp:(.text._Z15TestFetchAndAddIyEvT_[void TestFetchAndAdd(unsigned long long)]+0xe8): undefined reference to `__sync_fetch_and_add_8'
test_atomic.cpp:(.text._Z15TestFetchAndAddIyEvT_[void TestFetchAndAdd(unsigned long long)]+0x120): undefined reference to `__sync_fetch_and_add_8'
test_atomic.cpp:(.text._Z15TestFetchAndAddIyEvT_[void TestFetchAndAdd(unsigned long long)]+0x168): undefined reference to `__sync_fetch_and_add_8'
test_atomic.cpp:(.text._Z15TestFetchAndAddIyEvT_[void TestFetchAndAdd(unsigned long long)]+0x1d8): undefined reference to `__sync_fetch_and_add_8'
test_atomic.o:test_atomic.cpp:(.text._Z15TestFetchAndAddIyEvT_[void TestFetchAndAdd(unsigned long long)]+0x250): more undefined references to `__sync_fetch_and_add_8' follow
test_atomic.o: In function `void TestFetchAndAdd(unsigned long long)':
test_atomic.cpp:(.text._Z15TestFetchAndAddIyEvT_[void TestFetchAndAdd(unsigned long long)]+0x2b8): undefined reference to `__sync_val_compare_and_swap_8'
test_atomic.cpp:(.text._Z15TestFetchAndAddIyEvT_[void TestFetchAndAdd(unsigned long long)]+0x2f0): undefined reference to `__sync_fetch_and_add_8'
...
(long list of similar errors)
...

gcc information:
$ ~/custom/android-toolchain/bin/arm-linux-androideabi-g++ -v
Using built-in specs.
Target: arm-linux-androideabi
Configured with: /tmp/ndk-digit/src/build/../gcc/gcc-4.4.3/configure --prefix=/opt/digit/repo/opensource/ndk/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86 --target=arm-linux-androideabi --host=x86_64-linux-gnu --build=x86_64-linux-gnu --with-gnu-as --with-gnu-ld --enable-languages=c,c++ --with-gmp=/tmp/ndk-digit/build/toolchain/temp-install --with-mpfr=/tmp/ndk-digit/build/toolchain/temp-install --disable-libssp --enable-threads --disable-nls --disable-libmudflap --disable-libgomp --disable-libstdc__-v3 --disable-sjlj-exceptions --disable-shared --disable-tls --with-float=soft --with-fpu=vfp --with-arch=armv5te --enable-target-optspace --enable-initfini-array --disable-nls --prefix=/opt/digit/repo/opensource/ndk/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86 --with-sysroot=/opt/digit/repo/opensource/ndk/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/sysroot --with-binutils-version=2.19 --with-mpfr-version=2.4.1 --with-gmp-version=4.2.4 --with-gcc-version=4.4.3 --with-gdb-version=6.6 --with-arch=armv5te --program-transform-name='s,^,arm-linux-androideabi-,'
Thread model: posix
gcc version 4.4.3 (GCC)



Regarding (A), am I doing something wrong when linking the tests?

And what about (B), does it look like a problem of GCC support for the Tegra?

[Anton_P_Intel]

Hi gfour,
As arch=arm is not directly know to TBB make system try to specify either arch=arm32 or arch=arm64 in the make command, depending on what your system really is.

[Grant_H_Intel]

Regarding A)
Youmust use pre-built toolchain and link with libgnustl_shared.so, libstdc++.so will NOT work. Also, you must pushlibgnustl_shared.so to your Android device and set up LD_LIBRARY_PATH to find it.

Regarding B)
Try setting -D__TBB_64BIT_ATOMICS=0 on your command line for a temporary workaround, since GCC for ARM from NDK r7bseems to not support 64-bit atomics.

[gfour]

@Grant Haab: Thank you for the response. For exceptions, I eventually did it in a similar way to yours: I used the modified libstdc++.so.6 that comes with Android since r5b and supports exceptions (and I preload the library with LD_PRELOAD).

About the atomics, yes, it seems that the latest version of the Android NDK (r7c) does not support them. If I turn off 64-bit atomics, what do I lose? Performance? I compile for the Tegra, which is a 32-bit platform. I didn't try your solution yet, I will report back when I do. Are the atomics generated related to the ARM atomics of the Atomic TBB Contribution (http://threadingbuildingblocks.org/ver.php?fid=120)?

@Anton Potapov: Thank you for the help; unfortunately, arch=arm32 doesn't seem to improve the situation.

[RafSchietekat]

"About the atomics, yes, it seems that the latest version of the Android NDK (r7c) does not support them. If I turn off 64-bit atomics, what do I lose? Performance? I compile for the Tegra, which is a 32-bit platform. I didn't try your solution yet, I will report back when I do."
TBB does not use 64-bit atomics on a 32-bit platform, so there would not be an adverse impact on performance.

"Are the atomics generated related to the ARM atomics of the Atomic TBB Contribution (http://threadingbuildingblocks.org/ver.php?fid=120)?"
I would suppose that the atomics are the generic implementation provided by the g++ compiler, which are somewhat suboptimal. I have investigated a native implementation earlier, and if there's interest I could refactor it for the current code base, let you test it, and then submit it for integration.

"@Anton Potapov: Thank you for the help; unfortunately, arch=arm32 doesn't seem to improve the situation."
I was quite surprised when I read this suggestion, since arm32 does not occur in the source either.

[gfour]

@Raf Shietekat: Yes, I would be very interested in testing your implementation of the ARM atomics.

[Anton_P_Intel]

"@Anton Potapov: Thank you for the help; unfortunately, arch=arm32 doesn't seem to improve the situation."
""I was quite surprised when I read this suggestion, since arm32 does not occur in the source either""

Yes you right the sources do not mention arm at whole. The trick is that in order to get know correct symbol names to link with, makefiles try to find either "32" or "64" literal constants in the architecture name, if that architecture is not known to the build system.

[Anton_P_Intel]

"About the atomics, yes, it seems that the latest version of the Android NDK (r7c) does not support them. If I turn off 64-bit atomics, what do I lose? Performance? I compile for the Tegra, which is a 32-bit platform. I didn't try your solution yet, I will report back when I do."
"TBB does not use 64-bit atomics on a 32-bit platform, so there would not be an adverse impact on performance."

Raf is right. TBB does not use 64 bit on a 32-bit platform.
Disabling of double word atomics (via setting -D__TBB_64BIT_ATOMICS=0) during building of test cases, as Grant suggested seems to be right way. I.e. to run tests:

make test arch=arm compiler=gcc CXXFLAGS="-DTBB_USE_GCC_BUILTINS -D__TBB_64BIT_ATOMICS=0 -march=armv7-a -mfloat-abi=softfp -mfpu=neon -ldl"

Obviously, the only thing that should be cared of - is to take care that user code does _not_ use double word atomics. However you will get the same link error if your code does use them :)

[RafSchietekat]

"Yes, I would be very interested in testing your implementation of the ARM atomics."
I'll look into it over the weekend (refactoring required).

(2012-04-21 Added) Sorry, perhaps after it's been tested with the generic implementation. Hopefully unsuccessfully. :-)

[gfour]

It seems that the 64-bit generics should not be enabled: the generic implementation+tests now compile fine on both armv6 and armv7-a.

About half of the test programs work; however some important ones don't (such as test_atomic.exe). I will check to see that the problem lies with the ARM generics and not with any modifications of mine.

[tsug]

Hi,

I would be really interested how you made TBB compile with the NDK. Is there any chance that you will provide a tutorial, or even better, that it makes into TBB ?

It's been a day i'm trying and didn't pass the "cpu_set_t" step ...

Thanks a lot,

BQ.

[tokiroto]

I vote too for the Tutorial. Because, someone does something but no one able to explain completely or give a simple tutorial about the procces.