Visual Studio 2019 crashes when debugging mixed C++/Fortran program

[Brian_Francis]

A few months back we upgraded Visual Studio from 2015 to 2019 and Intel Fortran from 2016 to 2019. Everything compiles, links and runs as expected. Quite pleased about how smooth the upgrade went.

Less pleased though about instability when debugging mixed language C++/Fortran programs. Setting a breakpoint in a Fortran function and running to that breakpoint causes VS to display a message box with title "devenv.exe - Application Error" and with message "The instruction at 0x00000000545AE283 referenced memory at 0x000000002BA20000. The memory could not be read." The memory address and instruction change at each occurance. The same problem happens when hitting a breakpoint in a C++ function when the Call Stack is active and a Fortran function is somewhere on the call stack. This instability occurs for both 32-bit and 64-bit builds.

I recorded the problem and submitted it using the VS "Report a problem" feature. It has been stuck in "Under Investigation" status now for a month. See https://developercommunity.visualstudio.com/content/problem/714437/vs2019-crashes-at-breakpoint.html

We have 7 developers with different workstations, all running Windows 10, who all have the same problem. One of our key products extensively uses Fortran, so not being able to debug it is a real showstopper. As a workaround we had to make the following changes to all C++ projects in the mixed language solution:

• Platform Toolset: Changed from "Visual Studio 2019 (v142)" to "Visual Studio 2015 (v140)"
• Windows SDK Version: Changed from "10.0 (latest installed version)" to "10.0.14393.0"

With the latest VS 16.3 update, VC16 binaries are no longer compatible with VC14. To continue using the workaround we would have to make the same changes to C++ projects in all other solutions (because we statically link libraries produced by other solutions). We cannot do this, so we've rolled back out of the latest VS update.

Is this a known problem? Are other developers experiencing this problem? Other suggested workarounds or solutions?

Thanks,

Brian.

[FortranFan]

Brian Francis wrote:

.. As a workaround we had to make the following changes to all C++ projects in the mixed language solution:

• Platform Toolset: Changed from "Visual Studio 2019 (v142)" to "Visual Studio 2015 (v140)"
• Windows SDK Version: Changed from "10.0 (latest installed version)" to "10.0.14393.0"

With the latest VS 16.3 update, VC16 binaries are no longer compatible with VC14. To continue using the workaround we would have to make the same changes to C++ projects in all other solutions (because we statically link libraries produced by other solutions). We cannot do this, so we've rolled back out of the latest VS update.

Is this a known problem? Are other developers experiencing this problem? Other suggested workarounds or solutions?

What I find is Visual Studio 2019 and Intel Fortran integration are not quite ready to "tango" on "prime-time", particularly if you plan on continuously updating Visual Studio with each notification from Microsoft which is often a recipe for disaster.

Re: "Other suggested workarounds or solutions?," my suggestion will be to "freeze" your version of VS, say to 16.2 (or as appropriate), or see if Platform Toolset change to "Visual Studio 2017 (v141)" can help.

[Steve_Lionel]

I tried a couple of mixed-language applications with ifort 19.0.5 and VS2019 16.2.5 and 16.3 and couldn't reproduce a problem. Can you create and attach a ZIP of a sample application (include the projects so that they can be built and run) and instructions for where to set breakpoints and demonstrate the issue?

If you haven't already you should open a case with Intel Support, but I am sure they will want to see a reproducer.

[Brian_Francis]

Hi Steve,

I am unable to reproduce the problem in smaller mixed language solutions, but these are all console applications.

The solution that is giving us grief is a Win32 DLL that is in its 36th year of continuous development. It has 860 Fortran files (290K lines of code) and 2700 C++ files (1M lines of code). To complicate things further, it links in various static and dynamic libraries (a further 200K Fortran and 1M C++ lines of code). So it is not possible to package anything up (except perhaps the workstation) to reproduce the issue.

I like the VS "Report a problem" feature, since I was able to start a recording, get VS to crash, and then save the recording. However, I don't like their response time. I don't doubt that they're busy.

If lots of people are having this problem, then I'm sure you would have heard about it by now. So perhaps its just us. We will devote some time here to further investigate the issue. I'm hoping the problem is related to having a particular function in the call stack. I'll share whatever I find.

Thanks,
Brian.

[Steve_Lionel]

*I* might not have heard about it, but I haven't seen any other complaints about it here or in the other places I follow. Good luck!

[Brian_Francis]

An update...

In August I filed a bug report with the VisualStudio team showing the crash I was getting. After a series of periodic updates, last Friday I was notified that the problem has been fixed and that I can test the fix in a preview version.

https://developercommunity.visualstudio.com/content/problem/714437/vs2019-crashes-at-breakpoint.html?childToView=822065#comment-822065

This is my first time using the preview feature. I made a full backup of my computer so I could back out of any trouble. I then ran their preview installer. It installs to "C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview", so that it won't interfere with the current version. Nice! However (there is always a "however"), when I load a solution containing Fortran projects, a window pops up listing them all as Unsupported.

Has anyone else used Intel Fortran with a preview version of VisualStudio?
Do I simply uninstall and then reinstall Fortran?
Will it add Fortran integration to both my regular and preview installations?

Thanks,
Brian.

[Steve_Lionel]

The install doesn't understand "Preview" versions.

[Brian_Francis]

Where can I log an enhancement request for that capability?

[Steve_Lionel]

If you have current support, you can submit it through the support portal. But don't get your hopes up... The integration complexity is bad enough as it is. To be honest, I had not heard of "preview" versions of VS before.

[Devorah_H_Intel]

MS VS preview versions are not supported. Preview versions are used for testing, verifying bug fixes.

[jimdempseyatthecove]

I presume your main project is C++. Because the error appears to be stack related, it might be related to C++'s Structured Exception Handling. While you need SEH in your production code, can you test compiling your Debug build without SEH (and #define try and catch such that they "evaporate" the SEH code).

Jim Dempsey

[JohnNichols]

MS VS preview versions are not supported. Preview versions are used for testing, verifying bug fixes.

I use the latest Windows 10 Preview, the latest VS Preview and the latest Intel Compiler

There are limited problems with using this combination, but it is impossible to go back to VS 17 once you have successfully updated, do not go this route unless you need to for a good reason, the error on unloaded modules being unsupported can be fixed by creating a new SLN in VS 19 Preview.  I get this every time, but have got everything running I needed to after a few minutes work.

I have a good reason to take this step, but it has problems, but I want to know about problems well in advance. 

Also the Fortran error generator is really upset with VS Preview and a minor change with error can generate 30 or 40 errors - it takes a while to get used to this problem. 

VS Preview is just like a cut throat razor - works well in some places

Very old Fortran code has generated a few interesting bugs in the latest Intel Fortran, but this is normal.  

At least we are passed the days of Powerstation 

John

[Brian_Francis]

As Steve suggested, I logged an Intel support request asking that they support Preview versions of VS.

Some of my team have had success by disabling the Fortran Expression Evaluator extension. The jury is still out on that one. Hopefully we are not looking at a second intermittent problem. We've also had some success by deleting the .suo file and rebuilding before each debug run (yes, painful). The plan is to fudge along with what we have until the new release of VS comes out. I hear some of you saying "that's not a plan!"

If we really get stuck, I'm going to give Jim's idea a whirl.

Thank you all for your comments and suggestions.

Brian.

[Christian]

Hello Brian,

Brian Francis wrote:

If lots of people are having this problem, then I'm sure you would have heard about it by now. So perhaps its just us. We will devote some time here to further investigate the issue. I'm hoping the problem is related to having a particular function in the call stack. I'll share whatever I find.

I just wanted to state that it's not just you. We also have the same problem - and no solution (and no small reproducible project). We have to disable the Fortran extensions and work (more worse than better) with the C debugging engine.

Hopefully the fix in the preview version will hit a future update of 16.3 or 16.4.

Best regards,

Christian

[jimdempseyatthecove]

Christian,

This may be completely unrelated....

Several years back I had similar issues with the VS debugger. The program would crash with illegal instruction. Upon examination of the disassembly code I could find no error in the instructions in range of the failure point. This led me on a false trail of assuming an errant branch took the execution to the middle of the instruction sequence (branch to middle of byte sequence constituting an instruction). After several days of searching, and failing, I added some trace capture code. What I discovered was that the debugger startup code was placing an INT03 (break point) instruction in the middle of the byte sequence constituting an instruction, and by happenchance caused an illegal instruction as opposed to modifying what the instruction did, which would have been worse. Upon examination of the set break points (from property box) there was no break point set at/near that location. I tried deleting all break points individually, then adding the desired break points. This did not correct the problem. Even with no break points set, still illegal instruction. Then out of frustration, I do not know why I thought of this, I added a break point, then from the debugger break points property window I clicked on the red to delete all break points. Ran in debug mode and problem cured.

Suggestion: click on the red to delete all break points in the Break Points property box.

Jim Dempsey

[Christian]

Dear Jim,

as a matter of fact - some of my colleagues used exactly this method to get it working. On some other machine this does not work.

Now some good news – in Visual Studio 16.4.0 there is a huge improvement. I only had one crash in two days with the Fortran Expression Evaluator enabled.

There is only a problem with the watch window – it doesn’t update some variable values when stepping through the code. Deleting and re-entering the variable shows the new value.

But it’s a progress.

Christian

[van_der_Merwe__Ben1]

There are other crash problems as well, see here for example:

https://developercommunity.visualstudio.com/content/problem/829550/visual-studio-2019-crashes-when-trying-to-debug-an-1.html

This one does not happen in the original release of Visual Studio 2019, but in the 16.3.10 release.

[jimdempseyatthecove]

Christian,

I think the watch window issue (in mixed language program) may have similar causes to the debug issue. From what you describe it appears that a "rememberance" is being used to obtain the variable's address.

I also experienced a related but different issue with a C#/C++/Fortran(DLL) application whereby on program startup (in C#) that while I could specify break points in the Fortran DLL, that they would not break when that section of code was run. My fix (hack) was to insert a function call to the Fortran DLL that did nothing. By placing a break point on this function call in C#, I could then open a disassembly window and then at break (from startup) perform a step into. The step into, would load the DLL, then once loaded (what I assume) the correct symbol table was known (replaced the remembered table), and then the break points could be set properly.

The hack was a hassle, but it worked. Keep this in mind should your mixed language program experience similar issues.

Jim Dempsey

[David_Billinghurst]

Some "me to" notes.

1. Jim's trick to disable/enable breakpoints works for me on two machines. 
2. Resetting breakpoints doesn't work for a colleague.  He strongly believes the Fortran Expression Evaluator causes his problem, as breakpoint==crash with expression evaluator enabled, while breakpoint /= crash with expression evaluator disabled
3. Upgrading to VS 16.4.0 hasn't fixed the problem.  Too soon to tell if it is better/worse.
4. Our product has a C# wpf main and Fortran dll.  It is a large application.  I have a few test cases with the same structure.  I am trying to find a reproducer.  I can't trigger the issue in small cases, even when the product is crashing on breakpoints in a parallel VS session
5. Also seen the watch window not updating with VS 16.4.0.  Don't recall it with 16.3.9.

[burel__byron]

I have been having the same problem with my mixed C++/Fortran code. What seems to have worked for me:

In my C++ projects, under Configuration Properties - Advanced - Use Debug Libraries, one project had "Yes", all the others had "No". I changed the yes to a no and so far things are working. Keeping my fingers crossed.

[CKE]

My team and me are having the same issue for mixed language software development with Fortran/C and C++. Using

Microsoft Visual Studio Professional 2019 Version 16.4.3

Intel® Parallel Studio XE 2019 Update 5 Composer Edition for Fortran Windows* Integration
for Microsoft Visual Studio* 2019, Version 19.0.0052.16

the projects compile and link without problems and have no runtime errors.

If debugging of a Fortran application is started in Visual Studio then it freezes and is closed after some seconds. As mentioned above, the Expression Evaluator might trigger this behavior. But I wanted to know, what is the root cause of this behaviour and is there any hope to get this fixed by Intel?

I use the following compiler and linker settings for my C++ debug configuration (Placeholder <MyDynamicLibraryName>.dll):

Compiler:

/FR"Debug\" /GS /analyze- /W3 /Gy /Zc:wchar_t /ZI /Gm- /Od /Fd"Debug\vc142.pdb" /Zc:inline 
/fp:precise /D "WIN32" /D "_DEBUG" /D "_WINDOWS" /D "_USRDLL" /D "RWINI_EXPORTS" 
/D "_VC80_UPGRADE=0x0600" /D "_WINDLL" /D "_MBCS" /errorReport:prompt /WX- /Zc:forScope 
/RTC1 /Gd /Oy- /MTd /FC /Fa"Debug\" /EHsc /nologo /Fo"Debug\" /Fp"Debug\RWINI.pch" 
/diagnostics:column 

Linker:

/OUT:"<MyDynamicLibraryName>.dll" /MANIFEST /NXCOMPAT /PDB:"<MyDynamicLibraryName>.pdb" 
/DYNAMICBASE "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib" 
"advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "odbc32.lib" "odbccp32.lib" 
/IMPLIB:"<MyDynamicLibraryName>.lib" /DEBUG /DLL /MACHINE:X86 /SAFESEH:NO /INCREMENTAL 
/PGD:"<MyDynamicLibraryName>.pgd" /SUBSYSTEM:CONSOLE /MANIFESTUAC:"level='asInvoker' 
uiAccess='false'" /ManifestFile:"<MyDynamicLibraryName>.dll.intermediate.manifest" 
/ERRORREPORT:PROMPT /NOLOGO /TLBID:1 

The compiler and linker settings for the Fortran code with debug configuration are (Placeholder <MyStaticLibraryName>.lib):

Compiler:

/nologo /debug:full /Od /warn:interfaces /iface:cvf /module:"Debug\\" /object:"Debug\\" 
/Fd"Debug\vc160.pdb" /traceback /check:bounds /check:stack /libs:static /threads /dbglibs /c

Linker:

/OUT:"<MyStaticLibraryName>.lib" /LIBPATH:"<MyStaticLibraryPath>" /NOLOGO <MyStaticLibraryName>.lib

Debug compiler and linker settings of the Fortran program which uses the mixed language library (Placeholder <MyProgramName>.exe):

Compiler:

/nologo /debug:full /MP /Od /I"<MyStaticLibraryPath>"  /D_TRACE /warn:unused /warn:noalignments 
/warn:interfaces /real-size:64 /iface:cvf /module:"Debug/" /object:"Debug/" /Fd"Debug\vc160.pdb" 
/traceback /check:bounds /libs:static /threads /dbglibs /c

Linker:

/OUT:"Debug/<MyProgramName>.exe" /INCREMENTAL:NO /NOLOGO /LIBPATH:"<MyStaticLibraryPath>" /MANIFEST 
/MANIFESTFILE:"Debug\<MyProgramName>.exe.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' 
uiAccess='false'" /DEBUG /PDB:"Debug/<MyProgramName>.pdb" /MAP:"Debug/<MyProgramName>.map" 
/SUBSYSTEM:CONSOLE <MyStaticLibraryName>.lib

Do you see anything in my compiler and linker settings which might cause the crash during debugging in Visual Studio 2019?

Is it possible to get a log file from Intel Fortran Expression Evaluator before it crashes?