ifort: error #10001: could not find directory in which g++ resides

[necktweaker]

We have ifort installed on our cluster and have been using it for over a year. Occasionally, we have a problem on the compute nodes where we are unable to compile using ifort.

Logged into the compute nodes as root, we can compile the executable without error. However, when we attempt to compile on a user account we get the following error:

ifort: error #10001: could not find directory in which g++ resides

$ifort -V
Intel Fortran Intel 64 Compiler Professional for applications running on Intel 64, Version
11.1 Build 20100806 Package ID: l_cprof_p_11.1.073
Copyright (C) 1985-2010 Intel Corporation. All rights reserved.

$gcc --version
gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-46)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$g++ --version
g++ (GCC) 4.1.2 20080704 (Red Hat 4.1.2-46)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I have checked the rest of the forums on this site (and other sites). Some people suggested that 'cxxlib-gcc' could be the problem, however we are not using this compile flag. Others suggested that this error was due to /tmp being full. This is also not the cause for us.

I have compared the env variables for both root and the user accounts and I did not see any differences in theLD_LIBRARY_PATH,LIBRARY_PATH, orINTEL_LICENSE_FILE variables.

Does anybody have any suggestion on how we can fix this problem?

Thanks

[TimP]

Are gcc/g++ installed locally on each node, or on a shared file system which doesn't always respond immediately? It's common on a cluster to set up specific nodes for compilation, perhaps not on the same shared file systems used for cluster execution.
In some linux distributions, the PATH order (/usr/local/bin or /usr/bin first) varies according to who sets up the shell.

[necktweaker]

gcc and g++ are installed locally on each node. This problem only seems to occur on single nodes at a time. Ex: Today I got the "ifort: error #10001: could not find directory in which g++ resides" on compute-0-29, while at the same time I could compile without error on compute-0-12.

A complete reinstall of the node seems to fix the problem, however it is not very efficient to need to reinstall a node each time we experience this problem.