running a program with shared object and setting LD_LIBRARY_PATH

[Greg_T_]

I'm working on porting our software from Windows to Linux, and am learning about the differences in using Fortran on the two operating systems.  I'd like to clarify the need for the LD_LIBRARY_PATH environment variable and the -L compiler option.

To run a program that calls a subroutine in a shared object (*.so) library file, it seems that the *.so library file needs to be "installed" for Linux to find it.  To compile a test shared object that contains two Fortran source files I used:

>ifort -shared -fPIC random_integer.f90 ran3.for -o random_number.so -static-intel

I copied the random_number.so file to the directory where the main executable is located.  To compile the executable I used:

>ifort -o main.a main.f90 -L. random_number.so -static-intel

From the Fortran user's guide, it seems that -L informs the compiler where to look for libraries, so I used the current directory "." (period), followed by the name of the .so library.  Is there a preferred approach to tell the executable what shared object libraries to use?  Perhaps a syntax within the Fortran source code instead of a compiler option?

Is there a naming convention for executable files on Linux?  I've seen ".a" used a few places for an executable and for a library. 

It seems that on Linux the main.a program does not automatically find the .so library in the same directory.  So before running main.a I needed to "install" the .so library using the LD_LIBRARY_PATH environment variable; I used these commands:

>LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH

>export LD_LIBRARY_PATH

>./main.a

Where the ".:$" syntax gives the current directory ".", the colon ":" separator for other paths, and "$" dollar sign to evaluate the current path and add it.  It sounds like an installed program package may put library files in /usr/lib (root permission needed), but for test cases it will probably be easier to put the *.so file(s) in the same directory with the executable file.  Is the LD_LIBRARY_PATH environment variable needed?  Is there another approach?

This approach worked to compile the library and executable, and then run it successfully, but it seems a little cumbersome.  I was able to run this test program on an Ubuntu Linux computer too, which is encouraging.  Is there a preferred approach to run a program using *.so shared object library files on Linux?  Should I put the LD_LIBRARY_PATH initialization in a bash script to start the program?  Or is it better to put *.so files in /usr/lib or other "approved" directory, even for use in testing?

I'm using Intel Parallel Studio XE 2016 for the Fortran compiler on both Linux (Red Hat 6.8) and Windows.  The Linux install instructions in the sticky article at the top of the forum helped me a bunch with the Fortran install.  Thanks for your advice.

Regards,
Greg

[mecej4]

The normal use for file extension ".a" in Linux/Unix is for static libraries. Therefore, by using ".a" as the suffix for executables, you are out to make trouble for yourself. In fact, if you specify the use of a library, e.g., by specifying -lm for the math library, you will cause the linker ld to look for the library file libm.a; if that file is an executable (a.out format) or something else, the linker will probably stutter a lot. Better to stick to the standard conventions.

It is misleading to talk about "installing" user-supplied shared libraries by specifying LD_LIBRARY_PATH. LD_LIBRARY_PATH is an environment variable that the linker knows about. Just as you may modify %PATH% in Windows to be able to run some EXE or BAT files, you set LD_LIBRARY_PATH in Linux. In Windows, %PATH% is used to locate all types of runnable files (BAT, EXE, COM, DLL). In Linux, LD_LIBRARY_PATH and PATH are distinct, with good reasons.

You may find something useful in the article https://en.wikipedia.org/wiki/Rpath .

[Greg_T_]

Thank you for pointing me to the -rpath option.  I'll read about it and give it a try to see if I can get the program and shared object library to run without needing to set the LD_LIBRARY_PATH environment variable, which would certainly be more convenient.  My hope is to have the .so shared object library file in the same directory as the executable so that it is easy to find during run-time (and a convenient location for testing).  Is it expected that the -rpath option when compiling the executable will allow that?

Thanks for the advice on the .a file extension.  Instead of the .a file extension for an executable, is .out the preferred default?  Or would an extension such as .run or even .exe (my Windows experience showing here) be another acceptable choice?  It seems that file name extensions are not as crucial in Linux as compared to Windows, so could the executable file name be given without any file extension?

Thanks for the information and help.

Regards,

Greg

[mecej4]

Just as the .a suffix can be (mis-) used for executables and static libraries, as you discovered, .a could be used even for PDF files, if someone wanted to be really bohemian about it, Linux does not have hard rules about how to name and give suffixes to executable files. A simple convention to adopt is to name the result of compiling and linking Fortran source in xyz.f as xyz. If there is more than one source file and some libraries, you can choose the name in any convenient way. For example: ifort x1.f x2.f -L. -lsubs -o myprog .

In the nascent days of Unix programs were written in assembler (C was not yet born), and "a.out" stood for "assembler output". However, that is just a convention. The operating system itself relies on the x-bit of the file attribute to decide whether the file is executable or not. The Unix file program is another useful utility to find out the nature of a file based on its contents, ignoring the name conventions. (These are my perceptions, I make no claims of historical accuracy!)

When you start a compilation session, you usually run ("source") a shell script (e.g., ifortvars.sh) to set up the environment for your compiler, Abaqus, etc. (unless you are using gcc and gfortran). You can easily incorporate a custom setting of LD_LIBRARY_PATH in the same shell script. In fact, ifortvars.sh sets up LD_LIBRARY_PATH to access the MKL shared libraries (and other such "toolboxes":  IPP, DAAL,...). You can treat the shared libraries that you want in the same way.

[Greg_T_]

After reading about the -rpath option I've been able to use it when compiling and linking the main executable.  Thanks again for pointing me to that option.  Using -rpath avoids needing to set the LD_LIBRARY_PATH environment variable before running the executable, which is much more convenient.  I found I can use the -Wl (upper case W and lower case l "ell") option to set the local directory.  The updated compiler command I'm using for the main executable is:

>ifort -o main.run main.f90 -fPIC -L. -static-intel -Wl,-rpath,. random_number.so

It looks to me that the -Wl option uses commas to separate each part of the "-rpath ." option.  I've put the shared object library "random_number.so" in the same directory as main.f90, so I'd like the compiler and linker to look for it there.  I'm using the "." (period) to indicate the current directory for -rpath.  Likewise in the -L. option I'm using "." for the compiler to look in the current directory for the shared object library file.

As a side note, the name "main.run" is displayed with an executable program icon in the Red Hat Linux desktop display, so it seems Red Hat is happy with ".run".  Does anyone else use the ".run" extension for executables, or is there another file extension preference?

I can then run the main.run executable from the Linux prompt using just the program name:

>./main.run

I found that now I don't need to set LD_LIBRARY_PATH, so this is an improvement and more convenient for me.  I've also copied this test program to another computer, and it runs there too.  After copying main.run and random_number.so to the other computer I find that I need to reset the executable status of main.run using chmod:

>chmod u+x main.run

Is it common to need to update the executable status when copying program files to other computers?

Thanks for your help.

Regards, Greg