IO issues on Windows Server 2012

[Richard_S_6]

We recently added a Dell Poweredge R420 to our compute server mix. Because of the newer hardware we had to install Windows Server 2012 STD. To which we added the HPC addon to make it similar to our other compute servers running Windows 2003 Compute Cluster Edition.

Our existing software built with Intel Visual Compiler XE 13.1.2.190 runs without problems on the older setup. But on the new configuration we get an IO error at what seem to be random (maybe load dependent). If the application writes a very large file followed by a close then open it again we get an IOSTAT=30. The writes are to a local raid 1 disk with a controller with 1GB cache. Which is similar to our older compute servers.

Is there a compatibility issue between Visual Fortran and Windows Server 2012.

[Steven_L_Intel1]

We have seen that issue with many different Windows versions - it seems that Windows doesn't fully close a file until "sometime after" it returns back to the code that called CloseHandle. Our usual recommendation here is to add a SLEEPQQ call after the CLOSE for, say, 500ms. That usually takes care of it.

[Richard_S_6]

Already tried that with the following code.

100 continue
        open(unit=11,file=szFile,IOSTAT=ios999,buffered='yes',form='unformatted',action='write')
        if (ios999.ne.30 .and. ios999.ne.0) then
            goto 666
        endif
        openCount = openCount + 1
        if (ios999.eq.30) then
            if (openCount.gt.300) then ! try for 5 minutes
                goto 666
            endif
            ! wait up to 1 seconds
            iWait = 1000
            call SLEEPQQ(iWait) ! wait (integer in milliseconds)
            goto 100
        endif

Only happening on the Server 2012 box, never needed this before on 2003 Compute Cluster boxs.

[Steven_L_Intel1]

I don't know what to tell you, then. We did test on that version of Windows and did not see any problems. But there isn't much we can do here - if the call to CloseHandle (the Windows API used to close a file) returns success, we have to assume it is closed. It might be useful to call the Windows API routine GetLastError (the one from KERNEL32, not the one in module IFPORT) and see what the Windows status is after the failed open.

[Bernard]

It seems that handle to file is not closed thus from the POV of Windows it is still in use.As it was told try to call GetLastError WinApi function and inspect its return value.More advanced option will include the usage of windbg and scanning for not released handles.You can also use AppVerifier for the same purpose.

[Richard_S_6]

Added a c call to GetLastError, it seems to always return zero. Do I need to try something else to get more granular information than the IOSTAt=30?

[Richard_S_6]

Going to add a call to execute a script when this happens. Going to use the Sysinternals handle routine to see who has this file open.

[Steven_L_Intel1]

Which GetLastError did you call? It needs to be the one from kernel32.

[Richard_S_6]

Used the following c code:

#include <process.h>
#include <errno.h>
#include <Windows.h>
#pragma warning( push )
#pragma warning( disable : 4996 )
extern "C"
{
    void __stdcall GET_LAST_ERROR(int *err)
    {
        DWORD thisError = GetLastError();
        *err = thisError;
    }
}

From Fortran added:

call Get_Last_Error(lastError)

[Bernard]

Sorry for late answer.Try to use Process Explorer and look for open handles In your code.I think that there is only one function named GetLastError and it is exported by Kernel32.DLL. Regarding the result try to convert it to HRESULT. Here is the link http://msdn.microsoft.com/en-us/library/windows/desktop/ms680746(v=vs.85).aspx

[Richard_S_6]

Not sure how this will help since GetLastError is returning a zero. Just in case I redid my code as follows.

#include <process.h>
#include <errno.h>
#include <Windows.h>
#include <WinError.h>
#pragma warning( push )
#pragma warning( disable : 4996 )
extern "C"
{
    void __stdcall GET_LAST_ERROR(int *err)
    {
        HRESULT result = HRESULT_FROM_WIN32(GetLastError());
        *err = (DWORD)result;
    }
}

[Bernard]

In this case please use either Handle.exe or Process Explorer to look for orphaned handles.

[Richard_S_6]

Looks like the low level fortran open code has a bug and when it gets an IOSTAT=30 type error it does not close the openHandle it has.

Modified the wait code to have a close on the unit and things are working.

100 continue
        open(unit=11,file=szFile,IOSTAT=ios999,buffered='yes',form='unformatted',action='write')
        if (ios999.ne.30 .and. ios999.ne.0) then
            goto 666
        endif
        openCount = openCount + 1
        if (ios999.eq.30) then
            if (openCount.gt.300) then ! try for 5 minutes
                goto 666
            endif
            ! wait up to 1 seconds
            iWait = 1000
            call SLEEPQQ(iWait) ! wait (integer in milliseconds)
            close(11) ! add for bug in fortran open
            goto 100
        endif

[Steven_L_Intel1]

There is no bug in the OPEN code - you get error 30 only if the CreateFile API returns -1 as the handle, indicating that the operation failed. There is no path through the code where you'd get error 30 and a handle was successfully returned to the OPEN code.

The CLOSE you added would have no effect. The SLEEPQQ is what helps, as I suggested earlier.

[Richard_S_6]

Sorry your wrong without the close I get IOSTAT=30 300+ times and then the program quits.

[Richard_S_6]

Same thing in another place in the program. Without the close it would loop 240 times them give IOSTAT=30 error message. Remember this only seems to be happening on Server 2012.

! Random unit for output
5 continue
outFile = 12
call GETTIM (i,j,k,l)
call SEED(i+j+k+l)
call RANDOM(xFile)
outFile = (xFile * 70) + 20.0
if (outFile.lt.20.or.outFile.gt.99) then
    goto 5
endif
close(outFile)

! Check files
inquire(file=TemplateFile,EXIST=EXISTS)
if (.not.EXISTS) then
    goto 1300
endif
! open files
open(unit=11,file=TemplateFile,err=1000,form='formatted',    &
    share='DENYNONE',action='read',IOSTAT=ios999)
100 Continue

! make sure unit not in use
inquire(unit=outFile,OPENED=UNITINUSE)
if (UNITINUSE) then
    goto 5
endif

open(unit=outFile,file=DeckFile,form='FORMATTED',    &
     action='WRITE',STATUS='REPLACE',IOSTAT=ios999)
if (ios999.ne.30 .and. ios999.ne.0) then
    goto 1100
endif
openCount = openCount + 1
if (ios999.eq.30) then
    if (openCount.gt.240) then ! try for 4 minutes
        call Get_Last_Error(lastError)
        goto 1100
    endif
    ! wait up to 1 seconds
    iWait = 1000
    call SLEEPQQ(iWait) ! wait (integer in milliseconds)
    close(outFile) ! fix for Fortran bug in failed open
    goto 100
endif

[Richard_S_6]

Forgot to add that I wrote a little routine that does a SYSTEM call to run handle.exe on the file that gets the IOSTAT=30 and writes it to a log fiel. It always shows that the running program has a handle to the file when the error happens.

[jimdempseyatthecove]

There used to be an issue in ca V8.nn of IVF where IOSTAT=ios999 would not update ios999 on success. As a test for this lingering bug, insert ios999=0 in front of the open, remove the close(11), then see if you get a successful open after multiple retries. If this changes the behavior (open works), then please report this back to Steve.

BTW, I've seen a similar issue in Windows with C++ when the O/S is buffering writes. Note, when you goto arround the writes, you effectively have bypassed any buffering. Try inserting a FLUSH(11) prior to close, though I suspect the buffered write will still be in effect.

Jim Dempsey

[Bernard]

If there is call to CloseHandle and if the operation fails the return value will be zero.And it this case detailed error description should be given by GetLast Error.Maybe somehow your file handle is being CV loser two times it will also throw an error.

[Richard_S_6]

Tried setting ios999=0 just before open seems to have no effect. Been using Resource Monitor to watch the exe and it looks like there may be a close problem. Does the normal close(11) throw a runtime error is it fails? I'm seeing a handle for a file I know should be closed.

[Richard_S_6]

Definitely an issue with Fortran close on our Server 2012 box. Watching same program using Process Explorer on Server 2003 and Server 2012 you can see orphan handles accumulate on 2012. In this case we have a readonly shared file that gets opened 5 times by 5 calls to a subroutine witch writes out 5 separate files to 5 directories. First pass there is a handle to the file left over on 2012 not on 2003. Over time (like over night) you end up with 4 or 5 extra handles.  Problem comes with the output files, after enough time if an output file name is repeated in the same directory enough times eventually a handle gets left over also. Which causes an IOSTAT=30 since it can not be overwritten because of the open handle. In all cases even error conditions the file units are being closed before exiting the subroutine.