Python Numeric Dynamic Linker Problem with MKL 10.x

[balexander]

I'm having a problem with the new 10.x MKL: specifically, while importing the old Numeric-24.2 module into python (built against the new MKL 10.x) that leads to the following error message:

ImportError: /a-long-path/libmkl.so: invalid ELF header

The Numeric module in question was built and linked against -lmkl in the exact same manner we used to do it for the 9.x MKL (we've been doing this for a couple years or so - works great for pythonic MKL). I know there's a long list of new linker directives for the 10.x arch, but it hasn't been necessary to retool our legacy code yet. Numeric-24.2 is as legacy as it gets, and the code in there still works like a champ (yes, I know about numpy, but no, I don't have time to port our code right now).

The file libmkl.so is actually one short line of ASCII text along these lines:

"GROUP (libone.so, libtwo.so, libthree.so)"

with one, two, three being some default lower level libraries needed to comply with the new linker architecture. Seems like a redirection instruction to me. Problem is, I can't find an API anywhere that looks like this.

The only place I can see anyone on the Internet, my man pages, the gcc docs, the Intel docs, and so forth mentioning a GROUP directive as part of an ELF formatted library is in the NASM manual, and it doesn't look like what I want; that mention looks like some kind of weird Microsoftian issue, and I'm on Linux.

The Intel compilers seem to be able to figure out this directive, since the binaries compile just fine with no complaints or warnings (indicating that the required objects were properly resolved). However, it appears that perhaps the Python interpreter cannot.

Is there someone out here in Intel-land that can explain this GROUP directive? Specifically, is this intended to be a compilation failure point to force me to update to the new link mechanism (not backwards compatible), or was it intended to permit me to use legacy code with the new 10.x linking mechanism (highly desirable, and you should get high praise if this was the intent). If the latter, is this a bug I need to file against Python's dynamic loader module?

Thanks in advance to the group,

- Bill

[TimP]

Does it work if you simply link against those 3 libraries? GROUP is a gnu ld script command, described in 'info ld'. If your linker mechanism can handle circular references, you won't need it. Otherwise, repeating the list of 3 libraries in the right order 3 times should be sufficient.

[balexander]

Tim - thanks for the catch on ld. I simply forgot to look there (sloppy).

Yes, I could link against the base libraries directly, but the problem is that I'm building on a LAN with many machines and have to support in-place legacy builds. If I fork off a version for 10.x, all my install scripts break. But maybe I do have to go this route. I suppose some conditional logic test on the value of the MKLROOT variable might trap this, but it seems like the ld GROUP macro should work.

So the fact that Python can't parse the GROUP macro in ld sounds like a Python problem. Agree?

- Bill

[Tyler_T_Intel]

Bill,

Indeed, the "dummy" library was meant to provide backward compatibility so that our new linking model would not force a change to existing linking (while also allowing more flexibility/capability for those who do choose to move to the new model).

If the "dummy" library doesn't work for your situation, you could replace the "dummy" library with the specific libraries needed (this should only entail modifying your link line, but not otherwise "porting" your legacy code).

You can find more information on the new (10.0)linking model here: http://www.intel.com/software/products/mkl/docs/linux/WebHelp/userguide.htm

[balexander]

Thanks for the explanation - once the ld script was explained, I figured that the script had to have been put in place to support the lazy folks like me. :) I've read the data on the new linking scheme, but I honestly don't have time to port our code over right now. Problem is, we have a customer who wants the 10.x compilers and MKL in order to take advantage of the superior optimizations. We are starting our port, but it takes time. And I don't mean to disagree too strongly, but our large collection of makefiles do comprise a significant part of the software. Changing linker directives on a piece of code that is expected to build cross-platform and cross-architecture (e.g. 9.x vs. 10.x) is a major ordeal, even if it's just a one-liner from your perspective. It means I have to task a developer with building the conditionals and tests to get it right - after, of course, they have diagnosed what the issue is in the first place, and then testing and validation - I have to pay money for this to happen. This is why backwards compatibility is always so important - the money. But I admit that at some point the new product is attractive enough (or incompatible enough) that we will pay it and be happy. That day may not be today.

I think right now my plan is to create a .so that links the libraries I need with ar, and then use that to replace libmkl.so. If this fails, then I will have to link directly to the subcomponents, but that means I'm contributing new development to Numeric - a software package that has been deprecated for years (not smart).

On another note, as a long-time MKL user I do have to say that the linker directives are getting ridiculous. I've put up with -lguide and friends for openmp, which we don't even use. Now we get even more directives. What I really want is a fast lapack. I understand that all these fine-grained controls are needed for the super-optimizations required for truly fast code, but it would be nice if a rapid-prototyping option also existed. I just want the code to build and work - it will be fast enough for a first-time build. I used to know of a tool vendor who used the motto "minus-O and go" to describe how easy optimization should be for a rapid prototyper. Once the rapid-prototype is done, we will tune it ad-nauseum with special switches for compilers, OSen, whatever. But we have to know it's going to work first. Whoops - sorry for the rant - I need to watch that.

In any case, you are correct - direct linking should work. But my problem is a legacy Python application - I was hoping to find a painless way to just build the code, since our innovation is occuring in another area right now. But many thanks for the tip.