Problems using the PXFOPEN() posix routine

[Evan_de_Kock]

Dear All

I am trying to use the PXFOPEN() routine from the posix module IFPOSIX, but experience problems compiling the code. The problem arises since I am trying to use the parameters O_RDONLY, O_WRONLY, O_RDWR as described in the documention, but the compiler complains that these parameters are not defined. These parameters are only defined in the fcntl.h file under the \VC\include directory of my Visual Studio 2005 installation.

Any suggestions on how to resolve this PXFOPEN() problemwill be appreciated. Is the use of the IFPOSIX module portable to Linux systems?

Evan de Kock

[Steven_L_Intel1]

How very interesting - those symbols are indeed not defined. I'll have to look into this when I get back to the office on Monday. I see that there is an IPXFCONST function which is supposed to return the value of "constants defined in the C POSIX standard", but that doesn't seem to work for this.

Yes, the Posix calls are portable to Linux.

[Evan_de_Kock]

Quoting - Steve Lionel (Intel)

How very interesting - those symbols are indeed not defined. I'll have to look into this when I get back to the office on Monday. I see that there is an IPXFCONST function which is supposed to return the value of "constants defined in the C POSIX standard", but that doesn't seem to work for this.

Yes, the Posix calls are portable to Linux.

Thanks Steve, I will greatly appreciate that since I would like to use the IFPOSIX module in full.

In the meantime I have simply copied the definitions of these symbols from the fcntl.h file over to my source code (with some translations of cource) to get my posix test program compiled. Everything seems to be working fine, expect that PXFREAD() does not work properly. Upon calling PXFREAD (ifildes,buf,nbyte,nread,ierror), the number that is returned for nread is always less than nbyte, and is sometimes even zero when it is not suppose to be. In all these cases, ierroris alwayszero. On the other hand, PXFWRITE seems to be working fine. Could you please look into this PXFREAD issue as well?

Thanks,
Evan

[Steven_L_Intel1]

I'm not sure what I did wrong before, but indeed calling IPXFCONST with the name of the constant (in upper case) is the way to retrieve the value. When I do this I get:

O_RDONLY
0
O_WRONLY
1
O_RDWR
2

Would you please supply a test program for the PXFREAD and PXFWRITE issue? Thanks.

[Evan_de_Kock]

Quoting - Steve Lionel (Intel)

I'm not sure what I did wrong before, but indeed calling IPXFCONST with the name of the constant (in upper case) is the way to retrieve the value. When I do this I get:

O_RDONLY
0
O_WRONLY
1
O_RDWR
2

Would you please supply a test program for the PXFREAD and PXFWRITE issue? Thanks.

I would guess that you have made the same mistake as I did, namely using IPXFCONST(O_RDONLY) instead of IPXFCONST("O_RDONLY") - the documentation refers to the values O_RDONLY, O_WRONLY etc., instead of the names "O_RDONLY"..., which is a bit confusing. The documentation on the IFPOSIX routinesis generaly very scant, especialy on the error codes that might bereturned.

I have solved my PXFREAD problem by also using "O_BINARY" in addition to "O_RDONLY" when opening the file with PXFOPEN - when doing so PXFREAD correctly reads the specified number of bytes from the file. However, the documentation indicates "O_BINARY" as Windows only feature. Will the PXFREAD work correctly for unformatted files on a Linux system without the "O_BINARY" switch?

Evan

[Steven_L_Intel1]

I think the mistake I made was to type the name in lowercase. I agree that the documentation is scant. It doesn't help, really, that the POSIX bindings for Fortran were targeted at F77 and have not been updated since then - it is pretty much abandoned and few vendors support it.

Please provide a test case I can look at that shows the PXFREAD problem.

[Evan_de_Kock]

Quoting - Steve Lionel (Intel)

I think the mistake I made was to type the name in lowercase. I agree that the documentation is scant. It doesn't help, really, that the POSIX bindings for Fortran were targeted at F77 and have not been updated since then - it is pretty much abandoned and few vendors support it.

Please provide a test case I can look at that shows the PXFREAD problem.

Steve, please see the attachment MyTestPosix.F90. It contains a simple routine that can be used to copy a file. It is almost exclusively based on the Posix library, illustrating the use of PXFOPEN, PXFREAD, etc.You will notice the differention that I have madebetween Windows and Linux platforms. I am not sure whether the Linux section will work since the I know that the Windows section does not work without the "O_BINARY" switch in the PXFOPEN statement.

[Steven_L_Intel1]

Evan,

Your code works on Linux if you also conditionalize the open of the old file. If you don't use O_BINARY on Windows, then when the new file gets written, the POSIX routines add an extra for each record, so each line ends with . I don't have ready access to a POSIX spec to know if this is the way it is supposed to work or not - I will ask.

[Evan_de_Kock]

Quoting - Steve Lionel (Intel)

Evan,

Your code works on Linux if you also conditionalize the open of the old file. If you don't use O_BINARY on Windows, then when the new file gets written, the POSIX routines add an extra for each record, so each line ends with . I don't have ready access to a POSIX spec to know if this is the way it is supposed to work or not - I will ask.

Thanks Steve. Apart from the extra being added during the writes when the old file is no opened without the O_BINARY,eachPXFREAD(from the old file) only reads up to the end of the record and not the specified number of bytes (NBYTES in my example code). It is just strange that the PXFREAD returns no error code in this case.

I have still one question left on this matter. How do one conditionalize the open of the old file for linux to avoid the same problem when O_BINARY is absent in the Windows case? As I have mentioned earlier, the O_BINARY is stated to be a Windows feature only.

[Steven_L_Intel1]

You would conditionalize it the same way you did for the new file. However, I have done some further testing and believe that you don't need this. I need to make some modifications to your code as I know why it doesn't work if you omit O_BINARY. Stay tuned.

[Steven_L_Intel1 · Accepted solution]

Ok, this took me a little longer than I thought because of several issues.

First, on Windows one needs to open the file in binary mode if you're copying a file. Text mode, which is the default, will give you odd values for the number of bytes read (it seems to subtract one for each line in the file) but will "undo the damage" when you write. However, this means you cannot use the number of bytes read to judge how far through the file you are.

Second, if you don't use binary mode on Windows, it will fail to copy non-text files.

Third, you need to add O_TRUNC when you create the output file, as if it already exists with a longer length, you'll get the leftovers at the end.

I modified your routine to not care about the total file size and also to avoid the need for conditional compilation. I have tested this on Windows and Linux with files of various types and sizes and it seems to work. Enjoy.

[Evan_de_Kock]

Quoting - Steve Lionel (Intel)

Ok, this took me a little longer than I thought because of several issues.

First, on Windows one needs to open the file in binary mode if you're copying a file. Text mode, which is the default, will give you odd values for the number of bytes read (it seems to subtract one for each line in the file) but will "undo the damage" when you write. However, this means you cannot use the number of bytes read to judge how far through the file you are.

Second, if you don't use binary mode on Windows, it will fail to copy non-text files.

Third, you need to add O_TRUNC when you create the output file, as if it already exists with a longer length, you'll get the leftovers at the end.

I modified your routine to not care about the total file size and also to avoid the need for conditional compilation. I have tested this on Windows and Linux with files of various types and sizes and it seems to work. Enjoy.

Thanks for all the help Steve. Yes, I have realized after sending you the attachment that O_TRUNC is needed. Your trick in handling the O_BINARY issue in a portable manner is very neat! I also like your modification to the DO loop to copy the file contents, making it unnecessary to determine the total file size beforehand. However, that highlights again the imprecise documentation on the PXFREAD function. Making the call PXFREAD(ifildes,buf,nbyte,nread,ierror), the documentation cleary states that "if no error occurs, the value of nread will equal the value of nbyte." Your implementation of the COPY_FILE routine clearly exploits the fact that in general, nread < nbyte with ierror=0 on thepenultimate iteration. Maybe I just don't know how to read the documentation properly?

Also, if it is possible that the situation of nrbyte=0 and ierror /= 0 may occur in some situation, then your implementation will mask the error. I have therefore added an extra test after the DO loop to explicitly check for this.
Besides this comment, your code for COPY_FILE is very nice. I am especially pleased to know that it is working on Linux too. Thanks again for your help on the posix stuff!

Evan

[Steven_L_Intel1]

Ah, right, I should have done the test for NRBYTES==0 after the error check.

I agree that either the documentation is inadequate or there's a bug in the implementation somewhere. I have raised the issue with the developers.

[Evan_de_Kock]

Quoting - Steve Lionel (Intel)

Ah, right, I should have done the test for NRBYTES==0 after the error check.

I agree that either the documentation is inadequate or there's a bug in the implementation somewhere. I have raised the issue with the developers.

Steve, I have still one major issue with the posix implementation of the COPY_FILE routine. If the old and new files are the same, then the O_TRUNC in the opening of the "new" file oviously destroys the file to be copied. I cannot think of any way how to test whether the old and new files are the same (using the input arguments OLD_FILE and NEW_FILE directly is obviouslyutterly unrealiable). On a linux system one can in principle use a combination of PXFFDOPEN() and INQUIRE to establish whether the files are the same before opening the "new" file. However, PXFFDOPEN is rather useless on Windows because of the "zero Posix I/O flag" issue. Another way is to STAT both OLD_FILE and NEW_FILE and then compare their inodes. Again, this is useless on Windows because the inode of a file is always zero on Windows.

Any ideas on this problem will be appreciated.
Evan