define variable during runtime

[tropfen]

Hello,

i need to define a variable during runtime. But all conditions i need, i only know at runtime (like type (real*4, real*8), number of dimension( [x,y] )).

Is there a way to build this kind of program? Is it inside the fortran standard?


Frank

PS: Why do i need such feature?
I getting a netcdf data set and all information about size and type of the dataset is inside the file. Currently i have to use a noautomatic 2 step procedure.

[Jugoslav_Dujic]

Quoting - tropfen

Hello,

i need to define a variable during runtime. But all conditions i need, i only know at runtime (like type (real*4, real*8), number of dimension( [x,y] )).

Is there a way to build this kind of program? Is it inside the fortran standard?


Frank

PS: Why do i need such feature?
I getting a netcdf data set and all information about size and type of the dataset is inside the file. Currently i have to use a noautomatic 2 step procedure.

Fortran is a strongly typed language. That means that type of every variable needs to be known in advance so that the compiler knows what is doable with it.

You're not without options though. For example, you can read the data as a raw binary stream (i.e. array of INTEGER(1) or CHARACTER) and use TRANSFER intrinsic to "cast" the contents into appropriate type once you find it out. Or devote a pool of routines (one for handling real(4) scalar, one for real(4) 1D-array) etc, that would "do the right thing" with the given raw data. But we ought to know a little bit more about the nature of the problem to be able to suggest most viable solution(s): for example, do you know in advance -- at least something -- what kind of input you expect, or is it totally random? Do you know in advance the number of variables? What you need to do with the converted data? Etc.

[tropfen]

Quoting - Jugoslav Dujic

Fortran is a strongly typed language. That means that type of every variable needs to be known in advance so that the compiler knows what is doable with it.

You're not without options though. For example, you can read the data as a raw binary stream (i.e. array of INTEGER(1) or CHARACTER) and use TRANSFER intrinsic to "cast" the contents into appropriate type once you find it out. Or devote a pool of routines (one for handling real(4) scalar, one for real(4) 1D-array) etc, that would "do the right thing" with the given raw data. But we ought to know a little bit more about the nature of the problem to be able to suggest most viable solution(s): for example, do you know in advance -- at least something -- what kind of input you expect, or is it totally random? Do you know in advance the number of variables? What you need to do with the converted data? Etc.

Hello Jugoslav Dujic,

thanks for the fast reply.

unfortunatly i know near to nothing about the dataset.
It may have up to 4 dimensions and any type between integer*2 to real*16 is possible.

Based on the knowledge that i have to define all varibles in the beginning, i will define a set of predefined allocatables and use those during the program.

Thanks a lot
Frank

[jimdempseyatthecove]

Quoting - tropfen

Hello Jugoslav Dujic,

thanks for the fast reply.

unfortunatly i know near to nothing about the dataset.
It may have up to 4 dimensions and any type between integer*2 to real*16 is possible.

Based on the knowledge that i have to define all varibles in the beginning, i will define a set of predefined allocatables and use those during the program.

Thanks a lot
Frank

Frank,

Too much is unknown for us to offer advise.

Is the dataset binary or ascii characters?
If binary, is the data little-endian or big-endian?
If binary, is the floating point IEEE compliant?
When floating point and IEEE compliant is the size supported internally by IVF? (4,8,16)
Note, older insturmentation data may be REAL(10) or REAL(2).

Is the dataset one large block of data expressing an array (one or more dimensions unknown at compile time) of one type of INTEGER or REAL of size and extentunknown?
Is there a header record?

Can your code be written such that it reads the array as a single dimension array (as binary file)?
then interpret the size and extent using a function. example ARRAY(index(I,J,K))

For this type of work I have a union of typesbut look at using TRANSFORM as it can produce a descriptor without moving thedata.

Or,with interface checking off,you can pass thefirst element of anINTEGER array(plusextents) to subroutine expecting a REAL(4) of a specific rank plus the extents.

SUBROUTINE FOO(A,nX,nY,nZ)
INTEGER :: nX,nY,nZ
REAL(4) :: A(nX,nY,nZ)
...

Jim Dempsey

[Jugoslav_Dujic]

Quoting - tropfen

Hello Jugoslav Dujic,

thanks for the fast reply.

unfortunatly i know near to nothing about the dataset.
It may have up to 4 dimensions and any type between integer*2 to real*16 is possible.

Based on the knowledge that i have to define all varibles in the beginning, i will define a set of predefined allocatables and use those during the program.

Thanks a lot
Frank

Still, it seems a bit odd -- at least at first sight -- to have as input a bunch of variables whosestructure and semantics you don't know. When we write programs, we try to define the data model first, but you don't seem to have one. Would you be so kind to satisfy my curiosity -- what's this used for?

[tropfen]

Hello Jim,

thanks for your reply.

The format for the dataset is netcdf (*.nc). This is a spezific scientific format. Commonly used in the climate modeling community. Unfortunatly 99.9% of the routines are based on unix/linux machines.

The files are including a meta part. Saying x variables in y dimensions. Each variable does have it own attributes like type of variable, unit system, scaling numbers and so on.

One Example: the temperature at 2m above ground for 192 x 144 gridpoints covering the earth. Inside the dataset is also saved the position of each gridpoint, an array for the timesteps (eg. 1971-1-1 to 2100-12-31 saved as hours since a given date) and so on.

The grid may be saved as array(192,144,number of timesteps) or arrayX(192,144,number of timesteps,1,1). Each variable has it own array.

The position of each variable is randomly. eg. time at no. 1 londitute at no. 2 latitude at no. 3 tempearure at no. 4. In the next dataset each position (var_id) may differ.

All the needed information to construct the needed array is inside the metadata.

Dont ask me why the have designed this erratic file format.

My solution is, based on the knowledge that fortran cannot define variable during runtime, providing a number of different allocatable arrays, designing a huge select case tree and hoping it will work.

Thanks for your help

Frank



Quoting - jimdempseyatthecove

Frank,

Too much is unknown for us to offer advise.

Is the dataset binary or ascii characters?
If binary, is the data little-endian or big-endian?
If binary, is the floating point IEEE compliant?
When floating point and IEEE compliant is the size supported internally by IVF? (4,8,16)
Note, older insturmentation data may be REAL(10) or REAL(2).

Is the dataset one large block of data expressing an array (one or more dimensions unknown at compile time) of one type of INTEGER or REAL of size and extentunknown?
Is there a header record?

Can your code be written such that it reads the array as a single dimension array (as binary file)?
then interpret the size and extent using a function. example ARRAY(index(I,J,K))

For this type of work I have a union of typesbut look at using TRANSFORM as it can produce a descriptor without moving thedata.

Or,with interface checking off,you can pass thefirst element of anINTEGER array(plusextents) to subroutine expecting a REAL(4) of a specific rank plus the extents.

SUBROUTINE FOO(A,nX,nY,nZ)
INTEGER :: nX,nY,nZ
REAL(4) :: A(nX,nY,nZ)
...

Jim Dempsey

[jimdempseyatthecove]

Read your data into an INTEGER(1) :: INPUTARRAY(bigSizeHere)

Define a type containing

[cpp]    ! Align real 8 at front of context
    union
    map
    real(8) :: valREAL8
    end map
    map
    real(4) :: valREAL4
    end map
    map
    integer(8) :: valINT8
    end map
    map
    integer(4) :: valINT4
    end map
    map
    integer(2) :: valINT2
    end map
    map
    integer(1) :: valINT1
    end map
    map
    integer(1) :: ArrayINT1(8)
    end map
    map
    character(8) :: Name
    end map
    end union
[/cpp]

Then for the particular type copy from the current position in INPUTARRAY to the instance of the type variable for the size indicated. To read next REAL(4)

MyConversion.ArrayINT1(1:4) = INPUTARRAY(POS:POS+3)
POS = POS+4
RetVal = MyConversion.valREAL4

Add code to check for time to read more data, eof, errors, etc...

Jim Dempsey

[Steven_L_Intel1]

In the future, you could perhaps take advantage of the Fortran 2003 feature of "polymorphic" (run-time variable) datatype. This isn't in Intel Fortran yet, though. Perhaps later this year.

[Jugoslav_Dujic]

Quoting - jimdempseyatthecove

You're reinventing a Variant :-)
http://www.canaimasoft.com/f90vb/OnlineManuals/UserManual/TH_34.htm

[tropfen]

Hello,

first of all, thank you very much.

To Jim
your way will produce me a sevre headache, but i think on this way i will solve my problem.

To Jugoslav
your link is very helpfull to understand the way of thinking.

Frank

[jimdempseyatthecove]

Quoting - tropfen

Hello,

first of all, thank you very much.

To Jim
your way will produce me a sevre headache, but i think on this way i will solve my problem.

To Jugoslav
your link is very helpfull to understand the way of thinking.

Frank

Frank,

The headache is only for a very small piece of code. Once written, you wrapper functions will make it completly transparent (i.e. much easier to code).

In a project I wrote, it required writting compressed data of the internal state of a simulation program. Each save state might be a few megabytes of data.The individual data elements were compressed (or omitted) depending on value and use. The first consideration was what was the runtime format of the data element (INTEGER(1,2,4,8,16), REAL(4,8,16), CHARACTER(size), Array of typeor scalar of type) and use: constant, variable, with limited range, variable floating point but always integer, demotable floating point (computed as REAL(8) but can be archived as REAL(4)) and a few other considerations for arrays of these items which could be stored as a min within array followed by deltas from min for the elements within the array, etc....

The datbase would contain a header record with an array of INTEGER(1) values holding the conversion rule code number. The remainder of the database would contain blobs of records. Depending on how it was used in the application a REAL(8) variable might be stored using 0, 1, 2, 4 or 8 bytes, an array of REAL(8) might be stored in 0 or much fewer than 8 x number of elements in array.

The goal of the database was to maintain random access, reasonably good compression ratio, either lossless or acceptable lossy-ness as preference, and have very low compress/decompress overhead.

The higher level code simply reads and writes an instance of the internal state of the simulation program. The conversion rules are driven off the header record. The table driven conversion rules were relatively easy to implement (although some time was spent on debugging).

Once written though, the database read and wrote faster and was consideably smaller than the raw data.

I have forgotten about my headache long ago.

Jim Dempsey