Intel® Fortran Compiler
Build applications that can scale for the future with optimized code designed for Intel® Xeon® and compatible processors.

buffered IO

Scott_L_
New Contributor I
1,416 Views

 

The code I am working on is running fine on Windows.  When it is ported and  built on Linux, it fails on the  on eof our unit tests  with an error  – cannot write a file to working directory - in a directory that is open to write in.

Running the same Linux executable with strace, I get "too many open files" error trying to write the file.

Changing all buffered IO open statements from BUFFERCOUNT=4 to =1, now it runs with no errors.   Is there any known issue with buffered IO on Linux as opposed to WIndows?

 

thanks,

Scott

 

 

0 Kudos
24 Replies
Kevin_D_Intel
Employee
1,220 Views

The number of physical buffers and too many open files and/or cannot write to a working directory don’t seem related to me. Maybe there’s a shell limit being exhausted leading to the cannot open and files and too many open files errors.

What version of the compiler are you using?
How many files are opened at one time?   Are files also being closed during execution?
Can you provide a complete runnable reproducer?

0 Kudos
Scott_L_
New Contributor I
1,220 Views

I am using ifort 17.1 on Windows and on Linux and the intel C compiler same version.   Code is almost all fortran except for a C xml parser.  Only a handful of files open at once.   Put in a check that any logical unit opened had been closed.  Tried check=all and undefined initialization debug options.   Too many files message shows up with strace;  without strace, the code fails to open (in C) a file to write to in working directory (writeable).  

0 Kudos
jimdempseyatthecove
Honored Contributor III
1,220 Views

Is this application Fortran (has PROGRAM entrypoint) or C (has main entrypoint)?
Is threading involved?
If yes to C main, and threading, then does the C main instantiate threads that call the Fortran subroutines (that are further parallelized)?

Jim Dempsey

0 Kudos
Scott_L_
New Contributor I
1,220 Views

Main program and bulk of code is Fortran; 600k lines.  Small C project to read/write xml.  No threading or parallization.

thanks,

scott

 

0 Kudos
Scott_L_
New Contributor I
1,220 Views

 

Don't you really love those bugs that come and go when you add a printf...  OK, so it's not buffered IO in Fortran.  Intel C compiler though, with -O0 optimization.

 

thanks,

Scott

 

0 Kudos
Lorri_M_Intel
Employee
1,220 Views

You've got I/O going on in both Fortran and C, correct?  You need to be careful not to try and be accessing the same file and/or the same units.  The runtime libraries for the two languages do not share data, and in particular, don't share I/O "infrastructure" details.

               --Lorri

 

0 Kudos
Scott_L_
New Contributor I
1,220 Views

Yes, there is IO in both Fortran and C, but only to a distinct set of files/units.   the C code manages all IO to xml files only.  Fortran never touches them (xml)  directly.   logical units or file handles are never shared between Fortran and C.   At least for 2 tries each, I can get the test case to run all by itself, but when I run a set of 30 cases, the same test case always (2 times) fails.

Thanks

Scott

 

0 Kudos
jimdempseyatthecove
Honored Contributor III
1,219 Views

In some of the situations where I experienced crash going away when inserting PRINT or other statements to locate the error. (also known as Heisenbug), the error usually was manifested by program writing beyond array bounds or to uninitialized reference. Have you run the full compiler diagnostics and runtime diagnostics to check for call with incorrect interface and/or subscript out of bounds errors? That is a good first start (prior to diagnostic PRINT statements).

Jim Dempsey

0 Kudos
Scott_L_
New Contributor I
1,219 Views

Thanks,  yes I have run check=all and all the undefined initialization (so glad to see this feature since the CRAY-1 had it!!) on ~600 test cases on both windows and Linux.   Wish Rational Software Purify still supported Fortran.   Heisenburg would be proud as I consistently get the code to "pass" if I run a single job, but consistently fail when I run ~30 different jobs (test cases) on multiple Linux servers  all directed to the same file system.  Am I having fun yet?

0 Kudos
Scott_L_
New Contributor I
1,219 Views

A bright young guy found the problem.  I was calling GETFILEINFOQQ with a fixed string (name of one file, no wildcards) to get the file timestamp.   I neglected to continue calling GETFILEINFOQQ until I got the last file handle, since I just would see one file.  In the documentation fine print:

GETFILEINFOQQ must be called with the handle until GETFILEINFOQQ sets handle to FILE$LAST, or system resources may be lost.

Symptoms were too many files open, or unable to open a file to write to, or unable to open a readable file.   The API for this function is not the best, not that I didn't screw up.   Would be a kindness to have a clean function to get file timestamps, or if calling GETFILEINFOQQ with a fixed string filename just released its stuff since you can't have more than one file with the same name.

phewwwwwwwwwwww

scott

 

0 Kudos
Scott_L_
New Contributor I
1,219 Views

 

Question for intel Fortran developers:

Can getfileinfoQQ be used in a recursive function?    Is there a difference in its behavior on Windows versus Linux?

The goal is to write a Fortran function to recursively delete directories with a possible wildcard filename at the top level.

My code seems to be running on WIndows, but only deletes 1 of 3 subdirectories at a certain subdirectory level.

thanks

scott

 

0 Kudos
jimdempseyatthecove
Honored Contributor III
1,219 Views

A sub directory will (can) not be deleted if a file in the directory is in use. Sometimes this also includes if a Windows Explorer or file browse window (Dialog) is looking at the directory. Additionally, you may have a hidden file in there.

Does an error status return when deleting the directory?

Jim Dempsey

0 Kudos
Scott_L_
New Contributor I
1,219 Views

The recursive delete I have at this point works on Windows, but not on Linux, a recursive GETFILEINFOQQ at a lower recursion level seems to corrupt the GETFILEINFOQQ at the higher recursion level (on Linux but not on Windows).   No files should be open since this process is run as part of our test suite that creates new directories and then runs the test, all in batch.   It does not seem to fail to delete the directory using DELDIRQQ.   It seems to forget that at the upper recursion level GETFILEINFOQQ  found 3 files to delete and only deleted the first.

BTW, excel is notorious for locking a file it has open on windows, like .csv files.

thanks,

scott

 

0 Kudos
Steve_Lionel
Honored Contributor III
1,219 Views

Make sure that you have declared your routine recursive and that you're using a local variable for the "handle" argument. It ought to work.

0 Kudos
Scott_L_
New Contributor I
1,219 Views

 

ifort compiler options:   -noauto  

export FFLAGS = -c -I ../Includes -DLINUX -assume nounderscore  -names lowercase                                              -noauto   \
               -fp-model=precise -fp-speculation=strict -fpconstant  -traceback   -O0 -check all   -debug full

routine

!*******************************************************************************

!

!*******************************************************************************

recursive logical function ac_del(filename)

use ifport

use messages

!

IMPLICIT NONE

character(len=*), intent(in) :: filename

! local variables

integer :: ierr, result

integer(kind=2) :: count

character(len=SIZEOFFILENAME) :: filestr, subdir

 

TYPE (FILE$INFO) :: info

INTEGER(KIND=INT_PTR_KIND( )) :: handle

character(len=8) :: rout = 'ac_del: '

logical :: local_debug = .true., file_exists, delresult,wildcard

 

are info and handle auto or static?  on both windows and Linux ifort 17.1

manual:

Scalar variables of intrinsic types INTEGER, REAL, COMPLEX, and LOGICAL are allocated to the run-time stack. Note that the default changes to auto if one of the following options are specified:

 

does the recursive in the function definition make local variables auto instead of static?

 

thanks

scott

 

  • recursive

  •  

 

 

 

0 Kudos
jimdempseyatthecove
Honored Contributor III
1,219 Views

Without recursive (or other options that implicitly set recursive)

character(len=SIZEOFFILENAME) :: filestr, subdir

may default to SAVE (similar behavior to arrays).

Using the implicit SAVE filestr, subdir would be safe to use as long as upon popping up a recursion level that you clear out any left over tail-end text appended to the buffer

lenBefore = LENTRIM(subdir)
delresult = ac_del(subdir)
if(lenBefore < LEN(subdir)) subdir(lenBefore+1:) = ' '

Jim Dempsey

0 Kudos
Scott_L_
New Contributor I
1,219 Views

Adding AUTOMATIC attribute to the local variables in the recursive delete function did not change the functions behavior.   It works on Windows but on Linux, it fails to delete two of the three subdirectories in a directory selected for recursive delete.   I could provide the function if intel would like to test it.  

0 Kudos
Kevin_D_Intel
Employee
1,219 Views

Yes, if you can share a small but complete reproducer with us that will help us investigate if we have an issue on Linux with this routine.

0 Kudos
Scott_L_
New Contributor I
1,219 Views

Thank you Kevin,  I will as soon as I can pass the export control process.

thanks

scott

 

0 Kudos
Scott_L_
New Contributor I
1,036 Views

Kevin D,

Attached is the function source with a dummy main that should work.  The case I am failing on Linux is recursively delete 'junk'  where there exists this directory tree:

 

junk/

              junk1/

                          junk2/

                          junk2a/

                         junk2b/

 

I have some files under junk2, junk2a, and junk2b, but it may not matter.

On windows junk/ and all files/directories under junk/  get deleted.

On Linux,  junk2 under junk1 gets deleted but not junk2a or junk2b.  From the log, it does not seem to try deleteing junk2a & junk2b on Linux.

thanks,

Scott

 

0 Kudos
Reply