Read data directly from a .zip file

[davidspurr]

Is there Fortran code available to read data directly from a file within a zip? Basic zip - no encryption.

(be nice if IVF had something akin to MKL that included such a utility!)

With "sparse" GB files becoming quite common, it would make life a lot easier if could specify a file name within a zip file and read the data without having to first extract the file.

I realise that zlib has C code for the task, but I have no C code experience, so it is incomprehensible to me.

I did look at this several years ago, reading data from zipped files on a CDR. In that case there were multiple "small" zip files with a data single file in each zip. Eventually I settled for running an external command to unzip to a temp file & then read the data from the temp. That worked OK as there were multiple similar "small" files & I only had to extract 1 at a time to read them. Would not be nice if there were massive individual files.

With increasing data file sizes it would be nice to directly read the data without having to extract to a temp file first.

Seems to be something that would be useful in many instances, but I have not located any Fortran code to do the trick.

David

[Intel_C_Intel]

It's not Fortran code, but there is a public domain DLL that will read ZIP files. It is possible to call DLLs from Fortran, so you should be able to use this. By the way, this is from the people who wrote the code in ZLib.

http://www.info-zip.org/UnZip.html

[joerg_kuthe]

Here is some source code that might help you to use ZLIB:

MODULE qtZLibKindsModule
INTEGER, PARAMETER :: qt_K_ULONG = SELECTED_INT_KIND(9) ! 4-Byte Integer
INTEGER, PARAMETER :: qt_K_ULONGF = SELECTED_INT_KIND(9) ! 4-Byte Integer
END MODULE qtZLibKindsModule

MODULE qtZLibInterfaceModule
! Fortran Interface to ZLIB1.dll
! Calling convention according
! ...LIBsrczlib1.2.3zlib-1.2.3win32DLL_FAQ.txt
!
! * The exported symbols are exclusively defined in the source
! files "zlib.h" and "zlib.def", found in an official zlib
! source distribution.
! * The symbols are exported by name, not by ordinal.
! * The exported names are undecorated.
! * The calling convention of functions is "C" (CDECL).
! * The ZLIB1.DLL binary is linked to MSVCRT.DLL.
!
! Source:
! http://www.zlib.net/
!
USE qtZLibKindsModule
IMPLICIT NONE

! ---------
! Constants

INTEGER, PARAMETER :: Z_NO_FLUSH = 0 ! #define Z_NO_FLUSH 0
INTEGER, PARAMETER :: Z_PARTIAL_FLUSH = 1 ! #define Z_PARTIAL_FLUSH 1
! /* will be removed, use Z_SYNC_FLUSH instead */
INTEGER, PARAMETER :: Z_SYNC_FLUSH = 2 ! #define Z_SYNC_FLUSH 2
INTEGER, PARAMETER :: Z_FULL_FLUSH = 3 ! #define Z_FULL_FLUSH 3
INTEGER, PARAMETER :: Z_FINISH = 4 ! #define Z_FINISH 4
! /* Allowed flush values ; see deflate() below for details */

INTEGER, PARAMETER :: Z_OK = 0 ! #define Z_OK 0
INTEGER, PARAMETER :: Z_STREAM_END = 1 ! #define Z_STREAM_END 1
INTEGER, PARAMETER :: Z_NEED_DICT = 2 ! #define Z_NEED_DICT 2
INTEGER, PARAMETER :: Z_ERRNO =(-1) ! #define Z_ERRNO (-1)
INTEGER, PARAMETER :: Z_STREAM_ERROR =(-2) ! #define Z_STREAM_ERROR (-2)
INTEGER, PARAMETER :: Z_DATA_ERROR =(-3) ! #define Z_DATA_ERROR (-3)
INTEGER, PARAMETER :: Z_MEM_ERROR =(-4) ! #define Z_MEM_ERROR (-4)
INTEGER, PARAMETER :: Z_BUF_ERROR =(-5) ! #define Z_BUF_ERROR (-5)
INTEGER, PARAMETER :: Z_VERSION_ERROR =(-6) ! #define Z_VERSION_ERROR (-6)
! /* Return codes for the compression/decompression functions. Negative
! * values are errors, positive values are used for special but normal events.
! */

INTEGER, PARAMETER :: Z_NO_COMPRESSION = 0 ! #define Z_NO_COMPRESSION 0
INTEGER, PARAMETER :: Z_BEST_SPEED = 1 ! #define Z_BEST_SPEED 1
INTEGER, PARAMETER :: Z_BEST_COMPRESSION = 9 ! #define Z_BEST_COMPRESSION 9
INTEGER, PARAMETER :: Z_DEFAULT_COMPRESSION =(-1) ! #define Z_DEFAULT_COMPRESSION (-1)
! /* compression levels */

INTEGER, PARAMETER :: Z_FILTERED = 1 ! #define Z_FILTERED 1
INTEGER, PARAMETER :: Z_HUFFMAN_ONLY = 2 ! #define Z_HUFFMAN_ONLY 2
INTEGER, PARAMETER :: Z_DEFAULT_STRATEGY = 0 ! #define Z_DEFAULT_STRATEGY 0
! /* compression strategy ; see deflateInit2() below for details */

INTEGER, PARAMETER :: Z_BINARY = 0 ! #define Z_BINARY 0
INTEGER, PARAMETER :: Z_ASCII = 1 ! #define Z_ASCII 1
INTEGER, PARAMETER :: Z_UNKNOWN = 2 ! #define Z_UNKNOWN 2
! /* Possible values of the data_type field */

INTEGER, PARAMETER :: Z_DEFLATED = 8 ! #define Z_DEFLATED 8
! /* The deflate compression method (the only one supported in this version) */

INTEGER, PARAMETER :: Z_NULL = 0 ! #define Z_NULL 0 /* for initializing zalloc, zfree, opaque */

! #define zlib_version zlibVersion()
! /* for compatibility with versions less than 1.0.2 */

! ----------
! INTERFACEs

! ________
! int compress (Bytef *dest, uLongf *destLen, const Bytef *source, uLong sourceLen);
! Compresses the source buffer into the destination buffer. sourceLen is the byte
! length of the source buffer.
! Upon entry, destLen is the total size of the destination buffer, which must be at
! least 0.1% larger than sourceLen plus 12 bytes.
! Upon exit, destLen is the actual size of the compressed buffer.
INTERFACE
! int compress (Bytef *dest, uLongf *destLen, const Bytef *source, uLong sourceLen);
INTEGER FUNCTION com press( dest, destLen, source, sourceLen )
!x DEC$ ATTRIBUTES C, DLLIMPORT :: compress
!DEC$ ATTRIBUTES C :: compress
!DEC$ ATTRIBUTES REFERENCE :: dest, destLen, source
USE qtZLibKindsModule
CHARACTER(1), INTENT(OUT) :: dest(*)
INTEGER (qt_K_ULONGF), INTENT(INOUT) :: destLen
CHARACTER(1), INTENT(IN) :: source(*)
INTEGER (qt_K_ULONG), INTENT(IN) :: sourceLen
END FUNCTION compress
END INTERFACE

! __________
! int uncompress (Bytef *dest, uLongf *destLen, const Bytef *source, uLong sourceLen);
! Decompresses the source buffer into the destination buffer. sourceLen is the byte
! length of the source buffer.
! Upon entry, destLen is the total size of the destination buffer, which must be
! large enough to hold the entire uncompressed data. (The size of the uncompressed
! data must have been saved previously by the compressor and transmitted to the
! decompressor by some mechanism outside the scope of this compression
! library.)
! Upon exit, destLen is the actual size of the compressed buffer.
INTERFACE
! int uncompress (Bytef *dest, uLongf *destLen, const Bytef *source, uLong sourceLen);
INTEGER FUNCTION uncompress( dest, destLen, source, sourceLen )
!x DEC$ ATTRIBUTES C, DLLIMPORT :: uncompress
!DEC$ ATTRIBUTES C :: uncompress
!DEC$ ATTRIBUTES REFERENCE :: dest, destLen, source
USE qtZLibKindsModule
CHARACTER(1), INTENT(OUT) :: dest(*)
INTEGER (qt_K_ULONGF), INTENT(INOUT) :: destLen
CHARACTER(1), INTENT(IN) :: source(*)
INTEGER (qt_K_ULONG), INTENT(IN) :: sourceLen
END FUNCTION uncompress
END INTERFACE

CONTAINS

CHARACTER(100) FUNCTION qtZL_CF_GetZLibErrorMessage( iError ) RESULT(cMsg)
INTEGER, INTENT(IN) :: iError
INTEGER iS

cMsg = 'ZLib :: '
iS = LEN_TRIM(cMsg) + 2
SELECT CASE ( iError )
CASE (Z_OK)
cMsg = ' '
CASE (Z_STREAM_END)
cMsg(iS:) = 'Stream end (Z_STREAM_END).'
CASE (Z_NEED_DICT)
cMsg(iS:) = 'Need dictionary (Z_NEED_DICT).'
CASE (Z_ERRNO)
cMsg(iS:) = 'There is an error reading or writing the files (Z_ERRNO).'
CASE (Z_STREAM_ERROR)
cMsg(iS:) = 'Invalid compression level is supplied (Z_STREAM_ERROR).'
CASE (Z_DATA_ERROR)
cMsg(iS:) = 'Input data is corrupted or invalid (Z_DATA_ERROR).'
CASE (Z_MEM_ERROR)
cMsg(iS:) = 'Memory could not be allocated for processing (Z_MEM_ERROR).'
CASE (Z_BUF_ERROR)
cMsg(iS:) = 'Not enough room in the output buffer (Z_BUF_ERROR).'
CASE (Z_VERSION_ERROR)
cMsg(iS:) = 'Version of zlib.h and the version of the library linked do not match (Z_VERSION_ERROR).'
CASE DEFAULT
WRITE (cMsg(iS:), 10000) iError
10000 FORMAT('Unknown error; error code = ', I0)
END SELECT

END FUNCTION qtZL_CF_GetZLibErrorMessage

END MODULE qtZLibInterfaceModule

And here is a simple test program to show how to use it. Sorry for the German texts, but since language doesn't matter here, I leave it as it is:

PROGRAM T_ZLib1
USE qtZLibInterfaceModule
IMPLICIT NONE

INTEGER, PARAMETER :: ISLEN = 80
CHARACTER(1) :: c1aSource(ISLEN)
CHARACTER(ISLEN) cSource

INTEGER, PARAMETER :: IDLEN = ISLEN * (1. + 1./1000) + 12
CHARACTER(1) :: c1aDest(IDLEN)
EQUIVALENCE(cSource,c1aSource)

INTEGER (qt_K_ULONGF) :: iDestLen
INTEGER (qt_K_ULONG) :: iSourceLen

INTEGER iRet, j

WRITE(*,"(/'Quelltext eingeben: ')")
READ(*,'(A)') cSource

iSourceLen = LEN_TRIM(cSource)
iDestLen = IDLEN
iRet = compress( c1aDest, iDestLen, c1aSource, iSourceLen )

WRITE(*,6000) iSo urceLen, iDestLen
6000 FORMAT(/'Laenge des urspruenglichen Textes: ', I0/ &
'Komprimierter Text (hexadez.), Laenge:', I0)
DO j = 1, iDestLen
WRITE(*,6001) c1aDest(j)
6001 FORMAT(Z2.2, 'h ')
END DO

iSourceLen = iDestLen
c1aSource(1:iSourceLen) = c1aDest(1:iDestLen)
iDestLen = IDLEN
iRet = uncompress( c1aDest, iDestLen, c1aSource, iSourceLen )

c1aSource(1:iDestLen) = c1aDest(1:iDestLen) ! -> cSource
WRITE(*,6010) iSourceLen, iDestLen, cSource(1:iDestLen)
6010 FORMAT(/'Laenge des komprimierten Textes: ', I0/ &
'De-komprimierter Text (Laenge: ', I0, '): ', A)

END PROGRAM T_ZLib1

You need to link the library ZLIB.LIB to get this to work. Hope this helps.

Kind regards

Jrg Kuthe
http://www.qtsoftware.de

[davidspurr]

Thanks for the sample code. Looks like it will be a wet weekend task to work through it.

In the meantime, I proceded as before, shelling to System and running an exe to decompress to a temp file.

My hopes of reading each file within the "zip" as needed to save disk space were dashed by the fact that some of the files are ".tar.gz" type. I eventually twigged to the fact that I had to unzip everything each time I wanted to extract any included file. With up to 400 files in each zip that meant unzipping the whole thing up to 400 times :-( Orders of magnitude slower than unzipping everything to a temp file first, reading as needed, then deleting to temps when finished. Neat PD Win versions of tar & gzip programs avialable though.

Even with standard zip files, it proved much slower to extract individual files separately. Faster than for ".tar.gz" but basically, the unzip program appears to sequentually unzip all included files till it finds the one specified. OK if the one you want is at the front of the zip, but on average have to unzip half the files each time?

Seems that accessing separate files on a disk is orders of magnitude faster than accessing from within a "file container" (at least for zip - not certain if that is true for tar). Appears to be no way to rapidly home in on a particular file to extract. Pity, it would be very handy to keep multiple files relating to a project wrapped in a zip or other cotainer.

For now I will pay the price of unzipped everything each time I need to access some or all the contained files. Much preferable to dealing with 1000's of individual files.

I am still interested in the challenge of an "in core" solution (hopefully that is faster than using temp files), but it will have to wait till I get more spare time.

Again, many thanks for the code
David