VS2008 command-line build problem

[Steve_Nuchia]

Hello,

I am having a problem with command-line builds in VS2008. I know of only one other person from the Microsoft forums with a similar problem and I don't know what languages he is using. I'm posting this to see if anybody here might have seen it, since IVF is a major part of our system.

The main part of our build is a very large VS2008 solution with a deep and complex dependency graph. 20+ IVF projects are near the bottom of that graph, with mainly C++ above. Our C++ involves a lot of COM/automation interfaces with MIDL-generated header files. After much effort, the solution's encoding of the dependecy information is close to right and we can build interactively in the IDE, full and incremental, without problems.

When I try to run the same builds from the command line: devenv solution.sln /build "Release|Win32" say, it will run for a while, sometimes building one or two projects and sometimes building dozens. If I have the Visual Studio "maximum parallel projects" set to 1 it will sometimes even finish. But more frequently it will crash.

The stack trace when it crashes shows a call to a bad address from msenv.dll (installed with VS) CDepNode::FDependsOn(). Now I don't know but I'm betting that has something to do with file-level dependency analysis.

I have not been able to reproduce the problem with a simpler solution. It seems to happen most often when a project build completes while the other thread is still analyzing dependencies. If that's the case it will be very hard to reproduce without a realistically complicated large code base.

One possibility is that there is a discrepancy in the vtable between Microsoft's DLL and the file-level dependency analysis support in IVF -- a header that didn't get updated or something similar. Another is a concurrency control gap in the implementation of either Microsoft's or Intel's data structures or code in this area. Another is that one of the many resource leaks and other bugs in VS is squashing a live value that contributes to computing the call address.

Anyway, if you are working on a codebase that involves numerous projects in a VS solution, and especially if it is code that you could submit as a repro case, I'd be grateful if you would try a few command-line builds and see what happens. Try a mix of full and incremental, with the parallel build number set to 1, #cores, and #cores+1.

Thank you,

-swn

[TimP]

I think you are implying that the problem becomes more serious as the number of parallel threads in the build increases, making me wonder if there is a threading correctness issue in msenv or something associated with it. That seems likean issue Intel Thread Checker was made for. My understanding from a brief past discussionof a superficiallyanalogous situationwas that Microsoft would not work on such an issue unless it could be reproduced without involvement of non-Microsoft tools (and was important to a high profile customer).

[Steve_Nuchia]

I consider the problem pretty serious with any number of threads :-) but yes, it reproduces more reliably with more threads.

I hope we can get some action of of MS on this; that's why I'm hoping to surface more people with the same problem.

It wouldn't surprise me too much if they refuse to work on it but I'm still hopeful. The fact that it works in interactive mode suggests it may be something pretty simple -- an uninitialized semaphore, for instance, or a DLL being loaded with the wrong threading model, or ... well it could be anything.

[Steve_Nuchia]

My trouble ticket at Microsoft was updated this morning as "found and fixed" -- color me pleasantly surprised.

-swn

[ma_asghar]

Out of interest, did Microsoft say when they will be releasing the fix?

I've had a similar problem on a mixed Fortran/C++ solution with around 14 projects in VS2005 (IVF 10.1, MSC++ 2005).

Builds in the IDE would succeed but the command line build would continuously fail. It seemed that either the one of the parallel threads was racing ahead of the other or the build had trouble correlating the two threads.

Aftera bit of experimentation and searching the Microsoft website (e.g. http://support.microsoft.com/kb/921538/en-us), setting "maximum number of parallel project builds" to 1, worked.

Of course, it wouldbe nice to use the parallel build on the command line as well.

M Asghar

[Steve_Nuchia]

No word on fix availability. The guy who closed my ticket is the same one who closed the ticket on the same problem in VS2005 SP1. That's probably why he was able to find and fix it so quickly. I'm hoping it shows up in VS9 SP1 and if not I'll have to rattle their cage to get a hotfix. Or try to; I don't have any leverage.

[Steve_Nuchia]

Microsoft released SP1 for VS2008 yesterday. It installed cleanly with IVF 10.1.019 in place and the bugs I'd seen with the beta SP1 versus IVF have not shown up. And it fixed the command line problem!!!

If you've installed the beta SP or hotfixes see: http://blogs.gotdotnet.com/heaths/archive/2008/08/11/vs-2008-sp1-beta-must-be-removed-prior-to-installing-the-release-of-vs-2008-sp1.aspx

-swn

[Matt_F_1]

Anyone heard about anything for VS2005?

Thanks,

Matt

[Steve_Nuchia]

The QFE (hotfix) for VS2005 is supposed to be out, see connect.microsoft.com issue #332669

https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=332669