Dll-Hell with 3rd-party code

[Tobias_Loew]

Hi,

I'm developing on Windows. In my code I use IF 2013 but I've got 3rd-party code which is linked against the IF 2011 dlls. As all the IF dll have the same name in the different versions it is nearly impossible to ensure the correct dlls are loaded. It gets even worse when I'm trying to load my dlls into other processes that already use IF-dlls (e.g. Matlab).

Have you ever though about tagging your dlls with a version-number like msvc or mfc dlls (I don't mean an SxS mechanism, just a main dll versionnumber would be very helpful.)

regards

Tobias

[Lorri_M_Intel]

We do include a Windows "version resource" in the Fortran runtime libraries.   You can reference this programmatically (I don't have example code handy, but I'm pretty sure I've got something somewhere) or you can tell via the Windows Explorer.

We aim for backwards compatibility, so a Composer XE 2013 set of RTLs should work fine with code compiled with Composer XE 2011. 
The reverse is not always true; sometimes there are new interfaces created and used by the compiler, but it is a strong goal to keep compatible with the older object files.

                 --Lorri

[Tobias_Loew]

I just stumbled over another dll-version-conflict: I've got two version of libmmd.dll (both File-version 13.0.0.2) that are binary different. One is in the compiler-install-folder/redist/ia32/compiler, the other is at Common Files\Intel\Shared Libraries\redist\ia32\compiler.

Which one is the right one to load?

[Tobias_Loew]

Hi Lorri,

thanks for your fast reply. Backward compatibility is a fine thing, but it doesn't help if I'm loading my dll (that relies on dll-entry-points from IF 2013) into a Matlab-process that has already loaded a libmmd from 2011 (or even older). Any suggestions how to handle those cases?

regards

Tobias

[Lorri_M_Intel]

I'm sorry, I don't know off the top of my head; I'm not as familiar with the libraries in the Common Files\... directory.

You should be safe to use the one with the latest date, I would think.

I'll see what I can find out.

               --Lorri

[Bernard]

Tobias,

loader snaps functionality of windbg can be very helpful in order to resolve which dll's are actually loaded by the application at load time.

[SergeyKostrov]

>>...Have you ever though about tagging your dlls with a version-number like msvc or mfc dlls (I don't mean an SxS mechanism, >>just a main dll versionnumber would be very helpful.) Here are a couple of very simple steps to understand what DLLs are actually loaded wihen your application is working: - Compile a Debug version of your application and start debugging in Visual Studio - Start Debugging ( press F10 ) - Do something with your application and exit - Take a look at Output Window for a list of loaded DLLs and find all Intel DLLs ( take into account that full paths are displayed ) Also, I don't think Intel will ever change naming conventions for DLLs because too many software products dependent on it.

[SergeyKostrov]

One more comment: >>...- Take a look at Output Window for a list of loaded DLLs and find all Intel DLLs ( take into account that full paths are displayed ) The same applies to 3rd party DLLs your application uses. Please posts results of your verification.

[Bernard]

Few words about the searching and loading order of the dll's.NtLoader first will check local subdirectory next it will check executable directory next system32 directory and c:\windows directory.This can be seen with process monitor including also load image event if successful.

[IanH]

Before checking the path the loader checks is whether there is a DLL of the same name already loaded.  This is what is tripping up the OP - Matlab has loaded the Fortran runtime DLL's that it was built with, Fortran DLL's loaded subsequent to that will use those DLL's.  The set of search directories is irrelevant by that stage.

The solution is to (carefully) delete the Fortran runtime DLL's in the Matlab directory (or whatever program has done a local runtime install) that shadow the central "latest runtime installation" DLL's under common files.  It's annoying and fiddly (so for those out there that write programs that load extensions DLL's that still do local installs of the runtime - that sharp pain that you are now feeling in your side is due to a pin I'm sticking into a small doll here), but in the absence of something like version specific DLL filenames or use of SxS this is what you have to do.

Current "best practice" in my view is for all programs to use the Intel provided runtime msi or merge module, but there are a few recalcitrant vendors out there who haven't got their act together yet.  (If you don't ever plan to load extension DLL's written by others, and you don't mess with the PATH, then perhaps a local install is tolerable, but having shared libraries that aren't then shared seems a bit silly...)

[Bernard]

I have forgotten to add what IanH mentioned at the beginning of his answer.

[SergeyKostrov]

>>...As all the IF dll have the same name in the different versions it is nearly impossible to ensure the correct dlls are loaded... It is clear that you have some problems with search paths defined in PATH environment variable. As I've already mentioned a Debug version of your application when debugged in Visual Studio will load all 3rd party DLLs used by the application and you will be able to review the list of these loaded DLLs ( with full paths ) in Output Window of the Visual Studio.

[IanH]

Sergey Kostrov wrote:

>>...As all the IF dll have the same name in the different versions it is nearly impossible to ensure the correct dlls are loaded...

It is clear that you have some problems with search paths defined in PATH environment variable. As I've already mentioned a Debug version of your application when debugged in Visual Studio will load all 3rd party DLLs used by the application and you will be able to review the list of these loaded DLLs ( with full paths ) in Output Window of the Visual Studio.

Old runtime DLL's can still be loaded even if the ifort central installation with the latest and greatest runtime is first in the PATH. 

Note that the OP already knows that the wrong DLL is (attempting to be) loaded.  That's probably because if the operating system tries to load an EXE or DLL compiled with a later version of the compiler in conjunction with an old runtime DLL, the load of the EXE or DLL (nearly?) always fails due to missing entry points.  In that common case there's nothing to debug. 

The problem is well defined.  It is relatively easy to diagnose.  Finding a solution to the problem is the issue.

[Bernard]

Interesting why Intel does not use SxS like mechanism.

[SergeyKostrov]

>>...why Intel does not use SxS like mechanism... What do you mean by '...SxS like mechanism...'? Please privide more details.

[Bernard]

It is used by Microsoft and it stands for Side by Side assemblies.It was invented to reduce so called "dll hell".SxS works primarily by embedding extra data(dependency) in form of manifest file. Here is link where you can read more detailed description://en.wikipedia.org/wiki/Side-by-side_assembly

[SergeyKostrov]

>>...Old runtime DLL's can still be loaded even if the ifort central installation with the latest and greatest runtime is first in the PATH... Please explain how is it possible? I don't consider it as a problem with 3rd party DLLs and I see that there is a problem with project organization, like, settings, paths, etc, and that is why Tobias asked for help. Look, thousands ( if not more ) developers who use Ifort don't have any problems. Isn't that true?

[IanH]

A number of directories are searched prior to the ones listed in PATH.  If you are writing DLL's for use in another application a very important one is the application executable directory.  C:\Windows\system32 and friends also get looked at. I can easily construct somewhat more contrived examples if you want!

It is a problem I have had to deal with (and solve for users - which on rights restricted corporate workstations can be real fun) several times.  There is another active thread on the forum dealing with this problem, so while it isn't a common issue, it is far from rare.

That said - I don't have a problem with Intel's approach to the runtime, now that they offer a standard central install location.  The issue is that runtime installation habits were far from ideal prior to them effectively setting that standard central location - those "bad" habits (necessary previous, but not for some years now!) are what cause problems today.

(To the OP - that's why you have two sets of DLLs of the same version on your machine - the set under C:\Program files\Intel is notionally provided for use by developers writing and running programs on the same machine (multiple sets of DLL's, associated with each version of the compiler installed will be found under that tree - the developer may want to switch between them, none of the relevant folders may necessarily be on the system path), the set under C:\Program files\Common files\Intel other is provided for applications (notionally) compiled elsewhere - it only contains the latest and greatest set of runtime DLL's presented to the machine via an installation package (for example - that central location could be updated by the installation of a third party program built with a more recent release of the compiler than what the developer is using) and it would be sensible to have the relevant directory in the system path.)

[SergeyKostrov]

>>...those "bad" habits (necessary previous, but not for some years now!) are what cause problems today... Ian, It would nice if you provide more information about these problems and more details are needed. I think everybody, including Intel software engineers, will be interested to review your cases. Thanks in advance.

[Bernard]

Windows Internals has good explanation about dll search,mapping and resolving algorithms including process's dll database.

[IanH]

I'm sure the Intel folk are well aware of the current situation.  A few years back they provided the solution going forward!  But that doesn't change past implementations or force laggards to update.

A few years back Intel started releasing a merge module that put the runtime DLL's into a "standard" central location under Common files\Intel (there might have been an msi that previously did this as well).  Now, if you are deploying an application that relies on the Fortran runtime, the easiest and simplest approach is to bundle the merge module into your application's msi, and you are done - just like you do for the MS VC++ runtime.  Biggest downside to this approach is an increase in msi size, but (depending on your application and circumstances) that's bearable in the main.  New compiler runtime comes out?  Not a problem - the runtime DLL's get updated, they are backward compatible bar blatant errors (I've never seen a report of a backwards compatibility failure on this forum) so old stuff keeps working and new program gets its new DLLs.  Install a program built with old DLL's?  Not a problem - the merge module logic means that the runtime will not be updated in that case, backwards compatibility again means that the program built against the old DLL's continues to work with the pre-existing newer runtime installation.

Prior to that it was up to each application where the DLL's went - typically that was either the application's program files folder or system32 (evil in many ways - see other thread) or similar.  (You can still do that today if you want... but it takes more effort, so why bother?) The problem with this approach is that you can end up with multiple copies of the runtime installed in multiple places, and that can cause issues if your install mucks with PATH or your program allows for third party DLL's - hence this thread.