Running out of valid logical units from newunit

[Greg_Thoreson]

I ran into this issue where the program appears to run out logical units provided with newunit=, for example:

integer :: fileUnit
open(newunit=fileUnit, file='myfile.txt')

The problem occurs when newunit is used many times on a file (or files) that do not exist. In other parts of the code, where the file does exist, the newunit number keeps decreasing (more and more negative) eventually using newunit= will "reset" and return something like -1, which is not a valid one for us to use from what I can tell. It seems to either asymptote near either -(2^15)/2 or -(2^16)/2. Almost like it is underflowing.

Here's a small(ish) program that replicates the issue on Windows (I tried both 2019.5.281 and 2020.4.311).

You need to make a file (can be empty) called "ExistingFile.txt" and DO NOT make a file called "WontExist.txt".

program newunitbug
	implicit none
	integer :: i, j, k
	integer :: fileUnit1, fileUnit2
	integer :: istat
	integer :: val
	logical :: isopen
	character(len=64) :: fileName
	real :: start, finish
	
	do i = 1, 100
		open(newunit=fileUnit1,file='ExistingFile.txt', status='old', action='read', iostat=istat)
		write(*,*) 'fileUnit1=',fileUnit1
		if (istat .ne. 0) then
			write(*,*) 'failed to open on iter i=',i
			write(*,*) 'iostat=',istat
			write(*,*) 'open files:'
			do k = -1, -32000, -1
				inquire(unit=k, opened=isopen, name=fileName)
				if (isopen) write(*,*) k,fileName
			end do
			stop
		end if
		
		call cpu_time(start)
		do j = 1, 1000
			call OpenNonExistantFileNoClose()
			!call OpenNonExistantFileWithClose()
			!call OpenExistantFileWithClose()
		end do
		call cpu_time(finish)
		
		write(*,'(A,F10.3)') 'inner loop time (s):',finish-start
		close(fileUnit1)
	end do
	write(*,*) 'program finished'
	
contains

	subroutine OpenNonExistantFileNoClose()
		integer :: istat, fileUnit2
		open(newunit=fileUnit2,file='WontExist.txt', status='old', action='read', iostat=istat)
		if (istat .ne. 0) return
		close(fileUnit2)
	end subroutine OpenNonExistantFileNoClose
	
	subroutine OpenNonExistantFileWithClose()
		integer :: istat, fileUnit2
		open(newunit=fileUnit2,file='WontExist.txt', status='old', action='read', iostat=istat)
		close(fileUnit2)
	end subroutine OpenNonExistantFileWithClose
	
	subroutine OpenExistantFileWithClose()
		integer :: istat, fileUnit2
		open(newunit=fileUnit2,file='ExistingFile.txt', status='old', action='read', iostat=istat)
		close(fileUnit2)
	end subroutine OpenExistantFileWithClose
end program newunitbug

Using either OpenNonExistantFileNoClose or OpenNonExistantFileWithClose in the inner most loop will cause the program to eventually fail to open the existing file. Using OpenExistantFileWithClose appears to work just fine.

Here's the output I get:

 fileUnit1=        -129
inner loop time (s):     0.016
 fileUnit1=       -1130
inner loop time (s):     0.016
 fileUnit1=       -2131
inner loop time (s):     0.016
 fileUnit1=       -3132
inner loop time (s):     0.016
 fileUnit1=       -4133
inner loop time (s):     0.016
 fileUnit1=       -5134
inner loop time (s):     0.016
 fileUnit1=       -6135
inner loop time (s):     0.016
 fileUnit1=       -7136
inner loop time (s):     0.016
 fileUnit1=       -8137
inner loop time (s):     0.016
 fileUnit1=       -9138
inner loop time (s):     0.016
 fileUnit1=      -10139
inner loop time (s):     0.016
 fileUnit1=      -11140
inner loop time (s):     0.016
 fileUnit1=      -12141
inner loop time (s):     0.016
 fileUnit1=      -13142
inner loop time (s):     0.016
 fileUnit1=      -14143
inner loop time (s):     0.031
 fileUnit1=      -15144
inner loop time (s):     0.016
 fileUnit1=      -16145
inner loop time (s):     1.609
 fileUnit1=      -16145
inner loop time (s):     2.375
 fileUnit1=      -16145
inner loop time (s):     2.859
 fileUnit1=      -16145
inner loop time (s):     3.219
 fileUnit1=      -16145
inner loop time (s):     3.547
 fileUnit1=      -16145
inner loop time (s):     3.875
 fileUnit1=      -16145
inner loop time (s):     4.359
 fileUnit1=      -16145
inner loop time (s):     5.141
 fileUnit1=      -16145
inner loop time (s):     5.375
 fileUnit1=      -16145
inner loop time (s):     5.828
 fileUnit1=      -16145
inner loop time (s):     6.750
 fileUnit1=      -16145
inner loop time (s):     7.797
 fileUnit1=      -16145
inner loop time (s):     8.234
 fileUnit1=      -16145
inner loop time (s):     9.641
 fileUnit1=      -16145
inner loop time (s):     9.312
 fileUnit1=      -16145
inner loop time (s):    10.328
 fileUnit1=      -16145
inner loop time (s):    10.516
 fileUnit1=      -16145
inner loop time (s):     2.094
 fileUnit1=      -16145
 failed to open on iter i=          35
 iostat=         104
 open files:
          -1
          -2
          -3
          -4
forrtl: info (91): INQUIRE of internal unit-number is always an error (NOTE: unit does not identify a file)
          -5

It's worth noting a few other things:

• The time to open files with newunit= seems to drastically increase as we approach this limit
• On program end, I write all open units to the screen, there does not appear to be any lingering/opened file units
• Sometimes fileUnit1 will not open and error out with iostat=104, other times it will open but cannot be read (iostat=39) when fileUnit1 gets "reset" to -1

My current workaround is to use inquire to check if the file exists before calling open. But on my giant codebase, this is difficult.

Also, using the "open(unit=" with some constant logical unit on a non-existant file seems to work fine as well.

[Steve_Lionel]

I've seen issues with NEWUNIT over the years, suggesting that there's an issue with the underlying implementation. If you have paid support, I'd recommend reporting this to Intel at the Online Service Center.

[Greg_Thoreson]

Thanks, Steve.

According to the Online Service Center, when I select Intel Fortran Compiler, it states "Support for the product selected is through the Intel Community Forums.", a link back to here. And it doesn't let me proceed further.

I've entered a real-life infinite loop!

[JohnNichols]

Can you edit your first post and turn the code into Fortran as I cannot copy the code as formatted with Opera - I will not select and copy - but the first one does.  

Re infinite loop -- as Hawking postulated in his last paper there is finiteness to the universe. 

[Steve_Lionel]

Greg, I did say IF you have paid support. Apparently you don't, so reporting it here is correct. One of the Intel support engineers should pick this up. (And it remains annoying that the forum makes it extremely difficult to copy a code example, clearing a selection on the slightest mouse movement. I have managed to do it with ^C after a few tries.) It would be great if the forum offered a download button for code inserts.

[Greg_Thoreson]

I'm not positive I have paid support. Intel Parallel Studio is provided as an enterprise license from my company. I would assume that includes support, but I'm not sure of the pathway from lowly worker -> enterprise account -> Intel support.

If needed I can try to provide details on our enterprise license to the appropriate entity on Intel's end to escalate support.

[Steve_Lionel]

The user who registered the support license needs to be the one to submit the report to Intel. 

[Greg_Thoreson]

Apologies, I updated the post and also uploaded the source code as an attachment.

Apparently Hawking never had the pleasure of maintaining legacy fortran code

[JohnNichols]

Note had he would have had an endless stream of grad students and people who just wanted to get their name on one of his papers.  

Looking at your output someone has coded in a hard limit and you hit it.  

Interestingly it happens sometimes.  People always want more than programmers intend.  

I was talking to one of my Intel friends a few days ago, about this forum, my description was the 

"Knights of the Infinite Do Loop "

we know who is King. 

[Steve_Lionel]

I can reproduce this from the code sample. One thing is obvious is that closing a unit that was opened with NEWUNIT doesn't make that unit number available for reuse. The library eventually exceeds some limit on the number it can keep track of. I hope someone from Intel will pick this up and send it to development. It isn't the first time I have seen similar behavior.

[Devorah_H_Intel]

Thank you for reporting this issue. It has been escalated to engineering. 

[JohnNichols]

Is it just my imagination, or are there a lot of these ICE type issues lately. My memory seems to be that we had hardly any when I first joined the forum and now they seem almost weekly?

[Devorah_H_Intel]

Probably your imagination. We get the usual amount of ICE issues if not less than usual. 

[Greg_Thoreson · Accepted solution]

Just to close the loop on this for others' benefit, Devorah followed up with this response via email (thanks, Devorah)

"The issue has been fixed and it will be included in the future release, not the one that is coming out in two weeks but the following version. 2022.2

There is a work-around: declare the newunit variable 'fileUnit1' as 'INTEGER*1'.

INTEGER*1 is supported. And for INTEGER*1, we keep a map of the newunits assigned, and do reclaim numbers on CLOSE or error.

A bit-map of 128 bits is only 16 bytes. A bit-map of 2**15 bits would be 2**12 bytes, 4096 bytes. We could extend the INTEGER*1 mechanism and reclaim numbers perfectly."

[JohnNichols]

The comment from Intel would suggest they are running on a 3 month or quarterly update system.   Why do we never say quinerly.