concurrent compiles failing intermittently on common includes

[Steve_Nuchia]

This just started a few days ago after months of flawless operation. On my build server, randomly, I'm getting errors about failure to open one of our most heavily used include files. It's source controlled, allong with all the other build inputs, and doesn't change from one build to the next. I've rebooted the build server, there's nothing in the event log so it's not the disk. The common element seems to be that it's building two fortran projects in parallel when it happens, I've not yet seen it when it's fortan in one thread and something else in the other.

Anybody else seen anything like this?

It's a physical Intel Core2 Duo machine with Server 2008 x64, VS2008, IVF 10.1.019,running underTeam Build. On my workstation with a very similar configuation (except no team build) I've not been able to reproduce it. I don't have enough on it yet to file a bug report but if somebody else has seen it maybe we can combine our observations.

[TimP]

If Team Build relies on environment variable search paths, this looks like a frequently encountered problem with scripted builds: checking environment variables in the middle of a large build finds them empty, even though they were set and working correctly earlier in the build.

The problem comes up frequently with the use of scripted builds using tools not supported by Microsoft. This was cited as a reason for not looking into it (it would make life easier for users of non-Microsoft scripted builds), even though we submitted a reproducer (based, as I recall, on SFU).

[jimdempseyatthecove · Accepted solution]

Quoting - Steve Nuchia

This just started a few days ago after months of flawless operation. On my build server, randomly, I'm getting errors about failure to open one of our most heavily used include files. It's source controlled, allong with all the other build inputs, and doesn't change from one build to the next. I've rebooted the build server, there's nothing in the event log so it's not the disk. The common element seems to be that it's building two fortran projects in parallel when it happens, I've not yet seen it when it's fortan in one thread and something else in the other.

Anybody else seen anything like this?

It's a physical Intel Core2 Duo machine with Server 2008 x64, VS2008, IVF 10.1.019,running underTeam Build. On my workstation with a very similar configuation (except no team build) I've not been able to reproduce it. I don't have enough on it yet to file a bug report but if somebody else has seen it maybe we can combine our observations.

I've seen a similar problem but it was related to (presumed) different causes. In one case it was (is) when running with multi-threaded build (or coccurrent buildsfrom different processes)the dependency checker whereit appeared to open a file (include file) with the wrong mode (i.e. with a mode for exclusive use). In that case there was a narrow window where a build from a 2nd thread/process would get a file in use error. To correct for this consider locally caching the include files. e.g. make a top level project that checks the server for more recent include files, and if found simply copies the include file to the local cache. Then make all the other projects dependent on the caching project.

A second situation is were parallel builds attempt to concurrently build the same .MOD file. To correct for this, have one project make the .MOD files and seperate dependent projects containing the USE.

And a third issue is when the output files(.mod, .obj, .exe, .lib...) are written to a remote system and where the build receives a BAD_NET_PATH error when the real condition is something else and the something else is something to the effect "Unable to perform the request right now, please try again right now" i.e. "temporarily busy". The problem is related to writing accross the network.

Jim Dempsey

[Steve_Nuchia]

Quoting - jimdempseyatthecove

I've seen a similar problem but it was related to (presumed) different causes. In one case it was (is) when running with multi-threaded build (or coccurrent buildsfrom different processes)the dependency checker whereit appeared to open a file (include file) with the wrong mode (i.e. with a mode for exclusive use). In that case there was a narrow window where a build from a 2nd thread/process would get a file in use error. To correct for this consider locally caching the include files. e.g. make a top level project that checks the server for more recent include files, and if found simply copies the include file to the local cache. Then make all the other projects dependent on the caching project.

I'm pretty sure that's it, though I haven't caught it with filemon yet. Haven't tried very hard, been working on the workaround since it doesn't matter much what's causing it, I have to make my builds work now and a fix will take time.

I don't understand the local cache idea: if each project made a copy that would seem to address the issue; perhaps you are writing from the concurrent single-threaded builds perspective.

What I'm doing now is running an extra invokation of devenv on a solution containing only my Fortran projects, and ignoring the exit code. That pre-builds enough of the libs so I don't hit the problem in my real solution build. Ugly but functional.

I had been thinking in terms of something I could drop into the precompile / postcompile actions that would act as a semaphore. There are easy Unix shell idioms for this but I shudder at the idea of trying to write one in DOS Batch script. I like copying the real header to the project intermediate folder better; thanks for the suggestion.

And thanks for seconding the sighting. If I can confitm the hypothesis that it is opening the include in exclusive mode I'll submit a bug report on it.