Intermittent "#fatal: can't fopen file" with #include

[Brian_Francis]

I'mrunning Visual Studio 2008 SP1 with Fortran 10.1.021 on a Windows Vista SP1 64-bit Intel Core i7 machine with parallel project builds enabled.

Occasionally I get a bogus "#fatal: can't fopen file: ..." on a Fortran #include directive. I am building 4 times (win32/x64, debug/release) and this error only pops up on some of the builds.

I also occasionally get a problem creating a Fortran library. I get a fatal error LNK1181: cannot open input file '...lib'. As above, it will only fail on oneof the builds, and not the same library every time.

Ideas?

[Steven_L_Intel1 · Accepted solution]

I have seen numerous problems reported with parallel builds in VS, and it can be especially problematic for Fortran as it does not know about module dependencies. I would recommend setting the maximum number of builds to 1.

[TimP]

Quoting - Brian Francis

Occasionally I get a bogus "#fatal: can't fopen file: ..." on a Fortran #include directive. I am building 4 times (win32/x64, debug/release) and this error only pops up on some of the builds.

I also occasionally get a problem creating a Fortran library. I get a fatal error LNK1181: cannot open input file '...lib'. As above, it will only fail on oneof the builds, and not the same library every time.

We have seen this effect apparently caused by a temporary loss of access to Windows PATH environment variables (SET may show an empty environment). In that case, it might be worked around by arranging a dummy read of the environment variable prior to a 2nd read for actual use. For example, if building from Makefile, you would echo the include or lib path to the build log prior to starting ifort; then you would either find a mistake in your setup, or the problem would seldom break your build. I guess you could set up a pre-build SET event, if that is relevant. I got the impression (a long time ago) that Microsoft was aware of this problem, but felt that it was most often exposed by build procedures they didn't want to support. If you could reproduce the problem with a build procedure which Microsoft supports, it would be good if an important customer would submit it as a bug.

[Brian_Francis]

Quoting - tim18

Quoting - Brian Francis

Occasionally I get a bogus "#fatal: can't fopen file: ..." on a Fortran #include directive. I am building 4 times (win32/x64, debug/release) and this error only pops up on some of the builds.

I also occasionally get a problem creating a Fortran library. I get a fatal error LNK1181: cannot open input file '...lib'. As above, it will only fail on oneof the builds, and not the same library every time.

We have seen this effect apparently caused by a temporary loss of access to Windows PATH environment variables (SET may show an empty environment). In that case, it might be worked around by arranging a dummy read of the environment variable prior to a 2nd read for actual use. For example, if building from Makefile, you would echo the include or lib path to the build log prior to starting ifort; then you would either find a mistake in your setup, or the problem would seldom break your build. I guess you could set up a pre-build SET event, if that is relevant. I got the impression (a long time ago) that Microsoft was aware of this problem, but felt that it was most often exposed by build procedures they didn't want to support. If you could reproduce the problem with a build procedure which Microsoft supports, it would be good if an important customer would submit it as a bug.

Thanks Tim.

I'm building from VS solution files. The #include issue does not appear to be path related since the error reports a fully qualified DOS path (e.g., Z;blahblahfile.fnc). I was wondering if perhaps Fortran was trying to open the file exclusively, or if perhaps there is an isue with unclosed file handles? I'm compiling tens of thousands of files (a mixture of C, C++ and Fortran) over about 20 solutions. The same include files keep cropping up, but only on some of the builds. To me it looks like Fortran grabs an exclusive lock on the file anddoesn't release it, causing subsequent builds to fail.

I've got no ideas about the LINK issue. I'll go pour myself a cup of tea and check the leaves for clues.

Brian.

[Steven_L_Intel1]

If you enable the preprocessor, a separate program called fpp is run before each compilation. When this exits, all file locks, if any, are released. I would not expect that it opens input files for exclusive access.

[Brian_Francis]

Quoting - Steve Lionel (Intel)

I have seen numerous problems reported with parallel builds in VS, and it can be especially problematic for Fortran as it does not know about module dependencies. I would recommend setting the maximum number of builds to 1.

You can imagine my complete frustration about having to turn off parallel builds onmy brand newquad-core hyperthreaded i7 processor. I upgraded the entire team with these babies and now it appears I'd havebeen better off buying reallyfast single core processors.

[Steven_L_Intel1]

Well, if you aren't using modules, then the parallel builds should work. I don't know what the underlying problem is.

[TimP]

Quoting - Brian Francis

You can imagine my complete frustration about having to turn off parallel builds onmy brand newquad-core hyperthreaded i7 processor. I upgraded the entire team with these babies and now it appears I'd havebeen better off buying reallyfast single core processors.

I use parallel builds based on gnu make. Many of them split the pre-processing between a standalone preprocessor and the ifort fpp.It looks like youhave a reasonable expectation for the Microsoft parallel build. I wonder if a test could be made to see whether multiple copies of fpp are sharing include files properly.

[Brian_Francis]

Quoting - Steve Lionel (Intel)

Well, if you aren't using modules, then the parallel builds should work. I don't know what the underlying problem is.

Steve. I don't know what you mean by"modules"?

I am using VS2008 .sln files consisting of multiple .vcproj and .vfproj project files. Is there something different other than Gnu make (thanks Tim, but we've been down that road and got off of it a few years ago) that I should be doing to build my software?

[Steven_L_Intel1]

I mean Fortran MODULEs. I may be mistaken, but I did not think that the VS parallel build feature knew about module dependencies - I may be wrong and it ties in to the Fortran build system. I just know I have seen build issues with parallel VS builds in the past, and MSVC users have seen them too.

[Brian_Francis]

Quoting - Steve Lionel (Intel)

I mean Fortran MODULEs. I may be mistaken, but I did not think that the VS parallel build feature knew about module dependencies - I may be wrong and it ties in to the Fortran build system. I just know I have seen build issues with parallel VS builds in the past, and MSVC users have seen them too.

Thanks Steve. Let me say first of all that in no way does my frustration extend to you and the team who provide such a great service answering questions and helping us solve mostly self inflicted problems. Bravo!

I'm going to try and summarize what we've seen and done to this point in case anyone out there is facing the same issues:

----

We noticed problems with parallel builds on VS2005 with C++ and Fortran. Symptoms are easily noticeable as it typically involves a crash of VS itself. We loaded the latest VS2005 patch that was supposed to include a fix the parallel build problem (KB950882), butwe saw very little improvement. We still had crashes onVC8-only projects so I knew this wasn't the solution.

Migrating from VC6 to VC8 had been a real pain. It took months to get all the wrinkles out of moving from .dsp/dsw to .sln/.v[cf]proj. So when I talked to the team about moving on up to VS2008 I just about had to put down a mutiny.Pressing ahead, we picked one station and began the process of upgrading to see how things would go. It was done in a day and went very smoothly. I recommend you first checkout all solution and projects from your SCM (we're using Perforce), and thenopen each solution in turn and VS2008 will upgrade itself.

This upgrade solved all of our parallel build problems with VC projects, but we continue to have intermittent failures with the Fortran pre-processor #include directive. Although we compile many thousands of files, we only get the odd failure, so your mileage might vary.

We have decided to upgrade to VS2008 because of its greater stability, and becasue we can now take advantage of parallel builds on VC9 projects. The "Batch Build" facility does not use parallel builds so we can use this to interactively build Fortran projects. As long as we're not intensivelycompiling Fortran projects, we can use parallel builds on mixed VC/Fortran solutions.

----

Once again, a big thanks to Steve and Tim for their suggestions.

Brian.

[Les_Neilson]

Quoting - Brian Francis

We have decided to upgrade to VS2008 because of its greater stability,

Sorry I just had to smile.
I too am usually tasked with doing the VS/IVF upgrades, iron out the (usually C++) code changes needed to make the system build. But since I have been running VS2008 I have had regular instances of the IDE dying for no apparent reason. I get a dialog telling me Visual Studio has encountered a problem and that it will see if it can find out why, then it dies and restarts the IDE. Maybe it's a VS + Vista thing :-(
Sometimes it occurs immediately when I click on the sln file, sometimes when I am stepping through debug code, sometimes before coffee, sometimes after.
I don't usually use the words Stability and Visual Studio in the same sentence ;-)

However I can honestly say I do not remember ever having this error -unable to open an include file.

Les