Software Archive
Read-only legacy content

What is the correct way to load the Library Path?

CStac
Beginner
1,750 Views

Greetings,

I have some code which I compiled like this on my host:
$ ifort -openmp -mmic -o test.phi test.f90 -O4

I copied it up to the mic and tried to run


mic0$ ./test.phi
./test.phi: error while loading shared libraries: libiomp5.so: cannot open shared object file: No such file or directory

Oh! I read about this in the docuentation, the library path is missing. Simple fix, right? I NFS mount the /opt/intel up to the mic so it should go smoothly.
mic0$ source /opt/intel/composer_xe_2015.2.164/bin/compilervars.sh intel64

Except...
mic0$ env |grep LD_LIB
MIC_LD_LIBRARY_PATH=/opt/intel/composer_xe_2015.2.164/compiler/lib/mic:/opt/intel/composer_xe_2015.2.164/mpirt/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.2.164/ipp/lib/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.2.164/compiler/lib/mic:/opt/intel/composer_xe_2015.2.164/mkl/lib/mic:/opt/intel/composer_xe_2015.2.164/tbb/lib/mic

Huh...it doesn't set LD_LIBRARY_PATH, it sets MIC_LD_LIBRARY_PATH. Odd. But does the code run?
mic0$ ./test.phi
./test.phi: error while loading shared libraries: libiomp5.so: cannot open shared object file: No such file or directory

Nope! Well, I will set the path manually.
mic0$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/composer_xe_2015.2.164/compiler/lib/mic
mic0$ ./test.phi
[snip output, but it runs!]

So I went looking and I don't see anything that sets the LD_LIBRARY_PATH from the Intel tools on the Phi.

What am I missing? What file should I be sourcing for this to work?

Thanks!

0 Kudos
8 Replies
TimP
Honored Contributor III
1,750 Views

As you have built a MIC native application which you copy over to mic, you must do the same with required .so libraries, if you haven't mounted your host compiler/lib/mic directory.  If you have full privilege on the mic, you can copy to /usr/lib64/, otherwise you will need to set LD_LIBRARY_PATH.  MIC_LD_LIBRARY_PATH applies to offload applications (and only on the host side), but (at least for OpenMP library) would be set for you when you set up the compiler environment. 

You take risks when you set unsupported options such as -O4, particularly when specified out of sequence, although that doesn't appear to be central to your question.

If you could focus your interests, you might find a succinct explanation if you would open up your search engine. 

0 Kudos
CStac
Beginner
1,750 Views
Greetings,
Tim Prince wrote:

if you haven't mounted your host compiler/lib/mic directory.

I mentioned that I have already done that via the NFS mount.
Tim Prince wrote:
otherwise you will need to set LD_LIBRARY_PATH.
I did that too. However, I had to do it manually. The Intel compilervars.sh didn't do that for me which I expected it to.
Tim Prince wrote:
MIC_LD_LIBRARY_PATH applies to offload applications (and only on the host side), but (at least for OpenMP library) would be set for you when you set up the compiler environment. 

Thanks for the info on MIC_LD_LIBRARY_PATH. I was curious how that was used. So why doesn't the the compilervars.sh also set LD_LIBRARY_PATH on the Phi?
Tim Prince wrote:

You take risks when you set unsupported options such as -O4, particularly when specified out of sequence, although that doesn't appear to be central to your question.

For this particular application the performance boost is better and the risks are very minimal, so it works. :-)
Tim Prince wrote:

If you could focus your interests, you might find a succinct explanation if you would open up your search engine. 

I have opened my search engine, and I can't find anything on the best methods for having LD_LIBRARY_PATH auto-set for me. Everything related to the searches for the missing so file say I need to set LD_LIBRARY_PATH. Duh. I figured that out long before I started searching. Everything related to the LIBRARY_PATH says to run compilervars.sh which clearly doesn't work. I have tried various combinations and I am stuck in trying to figure out the best method for having the LD_LIBRARY_PATH loaded. What I can't figure out is why compilervars.sh doesn't set LD_LIBRARY_PATH on the Phi. I have tried a number of the environment scripts as provided by the Intel tools, and none of them set LD_LIBRARY_PATH on the Phi. They only set MIC_LD_LIBRARY_PATH which doesn't seem to work (which, thanks to your explanation, I understand why it doesn't work now). Sure, I can create my own script in my Intel directory and call it but I find it hard to believe that Intel hasn't already provided a script to solve this exact problem. Also, when creating my own script, I take the risk that I miss or add directories to the LD_LIBRARY_PATH that I shouldn't have. So, what is the correct way to load the LD_LIBRARY_PATH on the Phi? Thanks!
0 Kudos
TimP
Honored Contributor III
1,750 Views

If your mount is working and you can see the libraries (e.g. with ls) on the mic side, then making LD_LIBRARY_PATH consistent should work.  You might be able to extract a script from compilervars.sh to set LD_LIBRARY_PATH on the mic.

I don't know the pros and cons of why there is no extension of compilervars.sh to extend the setup to mic, other than that the big data centers each want to do their own thing.

In a somewhat analogous situation, there was a script provided with Intel MPI in the past to set ssh permissions but it seemed not to have been maintained for mpss upgrades.  Maybe if things ever settle down some of the things can be standardized.

0 Kudos
CStac
Beginner
1,750 Views
Greetings, So I pinged an acquaintance that is also going through the learning steps of using the Phi. He responded "Why are you trying to run natively? I use micnativeloadex..." So I tried 'micnativeloadex test.phi' and it works! No need to set the LD_LIBRARY_PATH. Hooray! That is a good step forward and if that is the recommended method for running natively then it would explain why the compilervars.sh doesn't set LD_LIBRARY_PATH.. So I tried my more advanced test, but it failed because it couldn't read the input file. Well, that makes sense because the file is still on the host. So I SCP'd the file up to the mic, but it still couldn't find it in my home folder. I am not entirely sure where micnativeloadex runs from on the Phi. My acquaintance doesn't have this problem so he wasn't sure how to solve it. I thought of three possible ways to solve it. 1) Rewrite my code to look at the mic home directory. Simple, but not very portable. 2) Rewrite my code to take another argument to point to the new location. 3) Rewrite my code to take another argument _and_ mount /home as NFS to eliminate the SCP step. Since every solution I can think of involves me re-writing my code, I thought I would ask if there is something I am missing. I don't see anything in the documentation for micnativeloadex on where to copy files or anything. Is it possible to just copy up my file with SCP to the directory that micnativeloadex runs my code from? Where is that directory? It seems some people NFS mount /home and others don't. Is there a best practise for this sort of thing? Is it recommended to mount /home? Thanks!
0 Kudos
CStac
Beginner
1,750 Views
Partial answering my own question. It looks like when I run the code and it trys to find my input file it is looking here: /var/volatile/tmp/coi_procs/1/5664/inputrules.dat However, that 5664 is the process ID. So pre-copying to the same directory isn't a great idea (or really feasible). I will do the rewrite of my code to take it in as an argument and hopefully that will work. I will mess with NFS mounting later. Thanks!
0 Kudos
CStac
Beginner
1,750 Views
Greetings, My code requires a input file to run. I re-wrote my code to take the filename as an argument. $ micnativeloadex blarg.phi -a "/home/user/inputrules.dat" The program dies when I do a check to read that file. I copied the input file into mic0:/tmp and that worked just fine. I guess the micnativeloadex can't read /home directories? If so, then there really isn't a reason to NFS mount the /home. Or am I still missing something basic here? I feel like I am missing something basic... Thanks!
0 Kudos
Frances_R_Intel
Employee
1,750 Views

By default, the process that micnativeloadex starts on the coprocessor runs as user micuser - hence the inability to read a file out of your home directory. In general, NFS mounting your home directory is a good thing but in this case it doesn't help you. Using micuser allows the coprocessor to be configured without enabling user login capability.

micnativeloadex does not need to set LD_LIBRARY_PATH on the coprocessor. It finds all the libraries it needs on the host using MIC_LD_LIBRARY_PATH, then copies those libraries and your program up to a tmp directory on the coprocessor - the same directory where it was looking for your input file. It does not count on anything on the coprocessor being located in a particular location.

The Intel compilers are designed to run only on the host, not the coprocessor; consequently the scripts in the composerxe directory are intended to run only on the host. We might be able to provide a script similar to compilervars.sh which could be used if the administrator chooses to NFS mount /opt/intel from the host over to the coprocessor. However, as Tim pointed out, many administrators of larger systems have particular ideas about where they would like things installed - often for very good reasons - so we don't want to get in their way. In those cases, it would be easier for the administrators to write their own scripts.

It would be nice if there was an option on micnativeloadex that allowed you to list your input files and have them copied over the working directory automatically. If you are feeling ambitious, you can find the source code for micnativeloadex in mpss-<version>/src/mpss-coi-<version>.tar.bz and add such an option yourself. Or you can submit a feature request ticket at https://premier.intel.com. The thing with doing the change yourself is that you can have the capability more quickly. The thing with submitting a feature request is that it will take longer but you won't need to worry about rebuilding micnativeloadex each time you upgrade the MPSS.

So, that's the story and I hope this helps.  

0 Kudos
CStac
Beginner
1,750 Views
Greetings Frances! Thank you very much for the details. That helps a lot.
Frances Roth (Intel) wrote:
...many administrators of larger systems have particular ideas about where they would like things installed - often for very good reasons - so we don't want to get in their way. In those cases, it would be easier for the administrators to write their own scripts.

I understand that admins might want to install software packages differently, but I am not sure why it matters. My version of compilervars.sh has the $PROD_DIR set in it. I could easily change it to /my/special/location/or/whatever , install the Intel tools there, and it would work. Right? There is a disconnect in my brain between why the location matters and why the compilervars do not set the path on the Phi. So if I am missing something obvious here, please let me know. :-D
Frances Roth (Intel) wrote:
The Intel compilers are designed to run only on the host, not the coprocessor; consequently the scripts in the composerxe directory are intended to run only on the host.
I think that is the best explanation I have heard for why the compilervars.sh doesn't work on the Phi. However, if I have a OpenMP program that I want to run on the Phi and I can't run it manually because LD_LIBRARY_PATH isn't set and I can't run it through micnativeloadex because it won't read in files from my /home directory...then how am I supposed to run my OpenMP program?? On a semi-related note, my work sponsors us to have access to a digital library. I found a PDF for Colfax's Parallel Programming and Optimization with Intel Xeon Phi Coprocessors. With just a few minutes of skim reading a select few sections I have found answers to several questions that I have or would have solved a lot faster if I had read this book before I high-dived into the Phi. I think this will be my weekend reading and once I work my way through some of the more relevant sections to my code, I will take another stab at it. Maybe it has the answer for how I am supposed to run my OpenMP code on the Phi. :-) Thanks again Tim and Frances!
0 Kudos
Reply