How to minimize size of a shared library

Bo_Jensen · ‎03-13-2012

I am building a large shared library, which uses a lot of boost functionality.

The library files are being compiled with :

[bash]-fPIC -fvisibility=hidden -fvisibility-inlines-hidden -ffunction-sections -fdata-sections [/bash] and linked with :

[bash]-static-libgcc -ldl -Wl,--gc-sections[/bash] The problem is the library size blow up to a amazing 17MB, without the above settings it's 28MB.. If I do a strip on the library afterwards, we get down in about 13-14MB, which is still too much. Looking at the output of nm it seems the hidding does not work properly, since it's exporting a large number of functions still.

With exact the same settings I can get 5.5MB after strip with gcc and 9MB before strip.

Even if I compile with intel and link with gcc directly, then the problem persists.

What settings do I need to set for intel to remove all the redudancy ?

xunxun · ‎03-13-2012

Try -O1 or -Os?

ipo sometimes can reduce the size, and sometimes may increase the size.

Bo_Jensen · ‎03-14-2012

Thanks for the suggestion, I had already tried it. With -Os the library gets down in 15MB unstripped, which is still far from gcc size and important optimization is now disabled, since it's a libary doing numerical calculations I don't want to lower the optimization level.

The problem is that it exports a huge number of functions, which is not marked to be exported.

SergeyKostrov · ‎03-14-2012

Quoting Bo Jensen

...The problem is that it exports a huge number of functions, which is not marked to be exported...

I think you need to review a couple of cases when some functions, declared as non-exported, are included in
a binary file of the shared library. I would consider cases:

1. If some 'FunctionA' is declared as inline, has implementation in a header file, and called at least
once in your codes onlythenithas to be included in a binary file ( this is expected result )

2.If some 'FunctionB' is declared as inline, has implementation in a header file, and not called at all in
any codes then it doesn't have to be included in a binary file ( this is expected result ).If it is included
then it looks like there is a problem with a C/C++ compiler.

The Boost is a huge template library andasimpleC++ templates-based test-case that uses atest shared
library, with some set offunctions declared differently,could helpto understand the problem.

xunxun · ‎03-14-2012

Can you ldd your library?
May consider that make some other library dynamic link.

ps: in general, the size icc generated is larger than gcc.

Bo_Jensen · ‎03-27-2012

Have not had time to look at this until now.

Try to use this as an example :

[bash]#include #include #include template class __attribute__ ((visibility ("hidden"))) MyDummyClass { private: std::map<:STRING> freq; public: void MyDummyClassFunc() { std::string word; while (std::cin >> word) { freq[word]++; } std::map<:STRING>::const_iterator iter; for (iter=freq.begin(); iter != freq.end(); ++iter) { std::cout << iter->second << " " << iter->first << std::endl; } } }; int Dummy() { MyDummyClass test; test.MyDummyClassFunc(); return 0; }
[/bash] Compiled with intel :

[bash]icpc -shared test.cpp -fvisibility=hidden -fvisibility-inlines-hidden -ffunction-sections -fdata-sections -Wl,--gc-sections -o test.so [/bash] and similar for g++.

On the intel created binary I get :

[bash]00002140 T .text._ZN12MyDummyClassIiED1Ev 000020d0 T .text._ZNSt4pairIKSsiED1Ev 00001e70 T .text._ZNSt8_Rb_treeISsSt4pairIKSsiESt10_Select1stIS2_ESt4lessISsESaIS2_EE11_M_put_nodeEPSt13_Rb_tree_nodeIS2_E 00001e80 T .text._ZNSt8_Rb_treeISsSt4pairIKSsiESt10_Select1stIS2_ESt4lessISsESaIS2_EE16_M_insert_uniqueERKS2_ 000020f0 T .text._ZNSt8_Rb_treeISsSt4pairIKSsiESt10_Select1stIS2_ESt4lessISsESaIS2_EE8_M_eraseEPSt13_Rb_tree_nodeIS2_E 00003c30 T __intel_f2int 00002500 T __intel_new_proc_init 00002600 T __intel_new_proc_init.A 00002540 T __intel_new_proc_init.H 00003210 T __intel_new_proc_init_B 000034d0 T __intel_new_proc_init_B.A 00003410 T __intel_new_proc_init_B.J 00003340 T __intel_new_proc_init_B.K 00003270 T __intel_new_proc_init_B.L 000027e0 T __intel_new_proc_init_G 00002880 T __intel_new_proc_init_G.A 00002820 T __intel_new_proc_init_G.R 000028d0 T __intel_new_proc_init_H 00002970 T __intel_new_proc_init_H.A 00002910 T __intel_new_proc_init_H.P 000026f0 T __intel_new_proc_init_I 00002790 T __intel_new_proc_init_I.A 00002730 T __intel_new_proc_init_I.S 000029f0 T __intel_new_proc_init_L 00002b00 T __intel_new_proc_init_L.A 00002a30 T __intel_new_proc_init_L.O 00003720 T __intel_new_proc_init_N 000039e0 T __intel_new_proc_init_N.A 00003920 T __intel_new_proc_init_N.J 00003850 T __intel_new_proc_init_N.K 00003780 T __intel_new_proc_init_N.L 00002f90 T __intel_new_proc_init_P 000030a0 T __intel_new_proc_init_P.A 00002fd0 T __intel_new_proc_init_P.L 00002b80 T __intel_new_proc_init_S 00002c90 T __intel_new_proc_init_S.A 00002bc0 T __intel_new_proc_init_S.N 00002d10 T __intel_new_proc_init_T 00002e20 T __intel_new_proc_init_T.A 00002d50 T __intel_new_proc_init_T.M 00002610 T __intel_proc_init 000026e0 T __intel_proc_init.A 00002650 T __intel_proc_init.H 00003550 T __intel_proc_init_B 000036a0 T __intel_proc_init_B.A 00003610 T __intel_proc_init_B.J 000035e0 T __intel_proc_init_B.K 000035b0 T __intel_proc_init_B.L 00003a60 T __intel_proc_init_N 00003bb0 T __intel_proc_init_N.A 00003b20 T __intel_proc_init_N.J 00003af0 T __intel_proc_init_N.K 00003ac0 T __intel_proc_init_N.L 00003120 T __intel_proc_init_P 00003190 T __intel_proc_init_P.A 00003160 T __intel_proc_init_P.L 00002ea0 T __intel_proc_init_T 00002f10 T __intel_proc_init_T.A 00002ee0 T __intel_proc_init_T.M 000024b0 T __intel_set_fpx_mask 00002490 T __intel_set_fpx_mask.A 00002460 T __intel_set_fpx_mask.H 000040bc T _fini 00001c9c T _init [/bash] While on the g++ generated I get the expected :

[bash]00003fdc T _fini 00002e0c T _init [/bash] So notice that the map stl functions gets exported ! I realize there are some issues with exporting STL functions, but gcc seem to manage that fine.

Now from the example it seems it's only related to STL, but it's not, I see some really strange exports, which I have a hard time reproducing in a small example.

I should say how I am building my library : I have module sources some with template functions and some with out. In some cases the templates are explicit instantiated, so I can reduce compile time and build in a modular fashion. All my classes have a hidding attribute and the explicit instantiations too. The problem is if a class get's implicit instantiated before explicit instantiation, then the attribute hiding is not controlled and it get's exported.

The final build is a c library, that is build with a wrapper class.

So my question is how can I control which functions get's exported, especially for external template classes like boost or STL ?

JenniferJ · ‎03-29-2012

I thought that I should be able to duplicate your issue with the testcase attached. But no. I tried 12.1 (Version 12.1.3.293 Build 20120212)in IA32 Ubuntu 11.04and SUSE Linux Server 11 x64, get the following.

32053 2012-03-29 09:57 t_gcc_ia32.so

30388 2012-03-29 09:56 t_icc_ia32.so

43273 2012-03-29 09:51 t_gcc_x64.so

15513 2012-03-29 09:50 t_icc_x64.so

cmd used is from your posting with "-fPIC": icpc -fPIC -shared t.cpp -fvisibility=hidden -fvisibility-inlines-hidden -ffunction-sections -fdata-sections -Wl,--gc-sections -o t.so

So I might have missed something. Please let me know if the above is the what you have. We'd like to improve the code size issue if possible.

Thanks,
Jennifer

Bo_Jensen · ‎03-30-2012

Thanks for looking into this.

I might not have explained it clear enough, but above example is just a small one illustrating that exports end up in the library, which is not intended and most certainly different from gcc. So I was not argumenting on library size on this example only the exports. I think what goes on for my large library is that soo many functions get's exported, which makes the linker command --gc-section useless, since it can not tell which ones are not needed. Since my code is huge, I thought starting with having similar exported function like gcc was a good starting point. Try to run nm test.so | grep " T " on your two libraries.

JenniferJ · ‎03-30-2012

I see.
With the "grep" cmd, I did see the big difference in the symbols being exported.

Let me file a bug to the compiler about this issue, but I'm not sure how likely it can be addressed in the current release though.

thanks,
Jennifer

Bo_Jensen · ‎03-30-2012

I have now tried everything I could come up with, but the only way I can get the expected exported functions is by using a Version Script.

JenniferJ · ‎07-06-2012

Hello,
This issue has been fixed in the latest update 11.

You can download the update 11 from Intel Registration Center.

Jennifer