Q about direct access files

[WSinc]

When I open a direct access file, i.e.

OPEN(8,file="test.dat",access="direct,recl=512)

I don't really know how many bytes are in it,

so when it gets to the end, I cant access the last record.

If RECL does not divide evenly into the actual number of bytes,

it generates an error, but does not try to read in those last bytes.

Example: If the file is 1100 bytes long, and RECL is = 200, then

it cant read the last 100 byes. Well, someone would say

"why not make RECL = 100 bytes?" The problem there is

that the file byte count might be a PRIME NUMBER, so RECL

cannot be made to divide it evenly.

So the burning question is: Is there a way to get the leftover

 bytes read in?

[WSinc]

I tried to correct a typo error, but it won't let me edit my post.

[Steven_L_Intel1]

By default, RECL= is in units of 4 bytes for unformatted files. If this file was created using some other compiler, try adding the option /assume:byterecl (in Visual Studio, this is Fortran > Data > Use bytes as RECL= unit.. > Yes.

But I would suggest using ACCESS='STREAM' to read a file that is a stream of bytes.

[WSinc]

OK, but should I assume there is no way to read in the leftover bytes?

Better I guess to use ACCES=STREAM as you suggested.

[Steven_L_Intel1]

No, not with direct-access and a RECL larger than one byte. It assumes all records are the same length (as that's how they would be written.)

STREAM is what you want here,

[John_Campbell]

Bill,

The following should work:

[fortran]
      integer(1) one_byte   ! or character
      open (unit   = 8,                &
            file   = 'test.dat',       &
            status = 'OLD',            &
            form   = 'BINARY',         &
            iostat = iostat)
      write (*,2000) 'Opening file ',trim(file_name),' iostat = ',iostat
2000  format (a,a,a,i0)
!
      do
!
!      Get next byte from file
         read (unit=8, iostat=iostat) one_byte
          if (iostat /= 0) exit
! ...
[/fortran]

[Steven_L_Intel1]

STREAM is the standard way to do this - no need to use FORM='BINARY'.

Bill, what are you doing with this file?

[WSinc]

I am reading it in as a text file. That method seems to work a lot better than using a FORMAT

statement on a regular TXT file.

I did get the STREAM to work ok, but with one glitch. I still have not figured out

a good way to determine the actual number of bytes read in when it reads in the last "chunk."

The Fortran Help doesn't go into much detail about that. It does return 

an EOF, though, and tries to read all the bytes in.

To get around that, I have to zero out the buffer, then see which bytes are "clobbered" when I read

the next set of bytes in. Perhaps there is a "clean" way to do that?

Here is a routine I wrote for getting a HEX dump.

We need a set of articles, one article for each type of file

one would want to work with.

[John_Campbell]

Bill,

Why not set irecl=1, ie read 1 byte at a time. Then you have no issue with the block length.

You could write a simple routine "get_file_size" that counts the number of bytes in the file (=ipos1), then another routine "store_file" that allocates an array and reads and stores the file contents.

John

[WSinc]

Hi Johnl

Sure - I understand about IREC=1, but I am concerned about the large number of I/O calls it would generate.

Wouldn't that have a large overhead?

What if I am working with a large file? Hopefully I would want to be able to read the bytes in , in large chunks.

A single pass would give me the right FILE size, but I am still stuck if its a prime number.

actually if I am getting the file size, I might as well store the contents as well, right?

[andrew_4619]

This seems like a strange problem. If it is a DAC file the data record length would normally have some logical meaning relating to its use and creation. This comes back to Steve comment that without knowing what you are trying to do with the file it is hard to give advise.

BTW there must be an integer number of bytes per record and an integer number of records, so the total number of bytes can only be prime if one of those values is 1!

[John_Campbell]

Bill,

To answer your question: "Wouldn't that have a large overhead?" my recent experience with Windows 7 is that there has been significant improvement in file I/O buffering (compared to XP) and so multiple short reads and even multiple reads of files can be very efficient.
Depending on how much memory you are using, you don't notice a decline in buffering efficiency until the file size approaches the size of installed memory, which can be a fairly large file, at least more than 4gb. Then again my change to Windows 7 also saw a change to a SSD drive.
The simplest approach is to try it. I'm sure you will not notice the run time, unless we are talking about thousands or millions !! of files.  (You still have not explained what you are using the files for.

John

[WSinc]

Actually the files I am working with are TEXT files, something of the nature of general correspondence that needs to be encrypted.

So the app is an encryption/decryption program, hopefully unbreakable.

Text goes in, and text comes out.........

Actually, I don't expect large files, so perhaps I should not

worry about overhead at all.

I will experiment with RECORD lengths of 1 byte, see what happens.

Thanks for your feedback ! ! Greatly appreciated !

[Steven_L_Intel1]

You should look at the Win32\Crypto sample we provide. All the work is done for you.

[WSinc]

Do you have a link to the Crypto example? How do I find it?

[Steven_L_Intel1]

It's part of the product. In the ZIP Program Files (x86)\Intel\Composer XE 2013 SP1\Samples\en_US\Fortran\Win32.zip  This sample was done in 2007 and uses FORM='BINARY' - I should update it to use stream I/O. It uses the Windows Crypto API and you have a wide choice of algorithms.

[WSinc]

I tried copying and pasting that link, and doing a search on "Zip program files crypto"

but could not get to that particular sample. Gave up after 30 minutes.

That URL has embedded blanks, that's why it wont go directly there.

I did find something submitted by John Mechalas on 9/26/13.

Called LIBCRYPTORANDOM

How do I get to the master directory for

those Zip program files?

[andrew_4619]

That isn't a URL they are part of the install on your PC

"C:\Program Files (x86)\Intel\Composer XE 2013\Samples\en_US\Fortran\Win32.zip"

The exact path depends on the version installed...