Thread-safe code

[Intel_C_Intel]

Hello,

I have a fortran code (25 source files, 50KB each) that uses a significant number of COMMON blocks. This code is used as a DLL and called from C++ (VS 2005) from multiple threads.

I want to make this fortran code threadsafe and after reading the documentation, I think the best approach is as follows:

Put all COMMON block variables in a user Type.
From C++, call fortran to allocate this user type and get a pointer to it. Pass the pointer to other fortran routines that will be called by C++. The other fortran routines will use the data from the pointer variable.
At the end of the program, in C++, call a second fortran routine to deallocate.

(I've been using this approach for a C dll and it works flawlessly, with 6 simultaneous threads).

Please can you comment on this approach. Is this the best approach ?
How do I go about actually implementing this in Fortran ? Any sample code would be helpful.

Thank you for your advice.
PS - I am not a fluent fortran programmer.

va.

[Jugoslav_Dujic]

It sounds sensible; however, it would require huge (if mechanical) changes in the fortran code: you have to pass the said type through every single argument list where COMMONs used to be, and prepend every former common variable with NewType%.

On retrospect, however, I can't think of a simpler approach. You could investigate Thread Local Storage, which provides thread-safe pointer(s) which can be used in a global manner (sparing the need to pass the type through all argument lists), but as result you would have to make a wrapper to accessing and dereferencing the TLS pointer, call that wrapper in every routine where needed, and end up with an unportable code; probably not worth the hussle. Your proposal seems to have the virtue of simplicity and portability.

[jimdempseyatthecove]

Jugoslav and original poster,

You can avoid the prepention of NewType% to the variables by using the Fortran Preprocessor

#define FOO NewType%foo

Or similar such defines. I have used this quite successfuly.

Note, you can also use this technique to rearrange subscripting order if you want and do other manipulations that would othewise require extensive edits to working code.

Using this technique you have the flexibility of choosing a method (TLS, psudo-this pointer, other...) without having to alter the soruce statements (and subject working code to new errors).

Jim Dempsey

[jimdempseyatthecove]

Va,

The suggestions by yourself, Jugoslav, and myself will provide multi-thread accessible data and would have nothing to do with making your Fortran DLL code threadsafe. Even if each thread had its own private context there would be issues that will interfere with threadsafe coding. Care must be taken with multiple thread updates toshared variables. The use of OpenMP ATOMIC, CRITICAL, and REDUCTION directives may be requied as well as use of calls toInterlockedXXX subroutines/functions. A bugaboo that often catches me is omitting to place AUTOMATIC on subroutine local arrays. Depending on compiler switches or directives local arrays may end up in a static location instead of on stack.

Jim Dempsey

[Intel_C_Intel]

Jim,

In my case, the callers of the DLL cannot use the DLL functions unless they allocate and pass this user-defined type to each function. Thus each function uses the information from this user-type and returns certain values back. Hence making the functions fully re-entrant. There wont be any cases where the user-type holding the (COMMON) block data is modified by two threads at the same time. (This will actually be handled by mutex in C++ code).

This will result in threadsafe code, right ? Or am I missing something here ?

Thank you.

VA

[Jugoslav_Dujic]

vikrant@wam.umd.edu:

This will result in threadsafe code, right ? Or am I missing something here ?

If you can manage to have no shared/global/static (SAVEd) variables whatsoever within the dll execution tree, (it can often turn out to be difficult with non-trivial codes, but it's doable), I'd say yes. Admittedly, Jim is better versed in threading and optimization issues than myself, so he can confirm.

[jimdempseyatthecove]

Va,

From my understanding of your messages your DLL is written in Fortran.

It is not clear if each of your DLL functions is wholly contained or nested several levels deep. Although the caller to the DLL may execute the call to the interface in a thread safemanner it would be presumptuous to assume that all calls within the DLL are thread safe without careful examination. Too often is the case where a DLL is derived from a working single threaded application and sadly even with aggressive testing a race condition won't show up until after the program is in production.

It would be nice if there were a compiler warning that could be used (and selectively disabled) whereby if no placement attribute is given (SAVE or AUTOMATIC) that a warning would be emitted if the compiler produces placement as if withSAVE. With the warning you could then edit the code accordingly to include SAVE or AUTOMATIC as the case may be.

Also it is not clear is if your user defined type only contains values or if it may contain references (pointers). If containing references then not only do you have to worry about the DLL being thread safe but you also have to worry about the caller(s) mannaging the data referenced by the DLL in a manner that does not adversely affect the execution of the DLL. An example of which might be passing in by reference a filespec to the DLL, the DLL executing in a thread safe manner, but having another thread trash the filespec buffer outside the DLL. Another example would be passing into the DLLa videoframe buffer for analysis and having a different thread stomping on the video frame buffer prior to the DLL completing the analysis.

Having a thread safe interface is only your first line of defense.

Jim Dempsey

[vikrantca]

This is followup question to the above discussed approaches.
I have replaced the COMMON blocks with a user defined type.
Now, in my subroutine whereever a COMMON block variable is used, I would like to replace it with: myTypeVariable%variableName

I tried the #define approach, but this doesnt work. What is the best way to address this without significant Replace/Paste of the code. I was hoping to use something like this:

#define commonVariable myTypeVarialbe%variableName

Once I define this, then I just need to comment out the COMMON blocks and the code should compile fine, right ? But when I #define, that variable has a zero value (integer type).

Any help is appreciated.

Thank you.
-VA

[jimdempseyatthecove]

VA,

Try this. Compile as-is for using COMMON.

Then remove "!" on first "#define" to compile for using module.

Remember to enable preprocessing file in project property page

Jim Dempsey

!#define USE_MODULE
#ifdef USE_MODULEmodule MOD_DefineSample type YourType  real :: rREALVAR  integer :: iINTVAR end type YourType type(YourType) :: aYourTypeend module MOD_DefineSample
#define REALVAR aYourType%rREALVAR
#define INTVAR aYourType%iINTVAR
#endifprogram DefineSample
#ifdef USE_MODULE use MOD_DefineSample implicit none
#else implicit none COMMON REALVAR, INTVAR REAL :: REALVAR INTEGER :: INTVAR
#endif ! Variables INTEGER :: I ! Body of DefineSample do 100 I = 1, 5
  INTVAR = I
  REALVAR = 3.14159 * REAL(INTVAR)  WRITE(*,*) INTVAR, REALVAR
100  CONTINUE end program DefineSample

[Jugoslav_Dujic]

Jim, does it work with REALVAR and INTVAR as arrays too? (Just asking, lazy to check ). If not, there might lie the dragons...

[jimdempseyatthecove]

Jugoslav,

Yes it does work with arrays.

There is one potentialproblem with this. That is it may expose a bug in IVF.

On W_FC_C_9.1.037 I cannot use

do INTVAR = 1, 100

Because this expands to aMyType%iINTVAR and for some reason IVF W_FC_C_9.1.037 balks at using an integer within a derived type for a loop control variable. You get the error messege if you type in the expanded text as well.This might be fixed in the newer version. I haven't filed an issue due to me running an older revision. (I am waiting for V10.nn to settle down before upgrading).

I have been doing this for several years now. I have a solution with maybe 13 projects, ~700 files. And was written in F77 with buku number of COMMONS. I changed to using modules and used FPP such that the statemnts did not change but would compile under conditional control into either the F77 with COMMON or F90 with modules. Worked real slick. Also the motivating force for going to F90 was to get OpenMP and using the FPP hack I could also invisibly prepend what amounts to a this pointer to the variable. i.e. make it thread safe too.

Debugging is a bit of a problem but not unworkable as you have to prepend the invisible part of the variable name. Usualy a conditional compile with code to initialize a pointer to the context is all that is required.

Jim Dempsey

[Steven_L_Intel1]

Good thing that using a derived type component as a DO loop control variable doesn't work - it isn't supposed to.

[jimdempseyatthecove]

I would suppose that permitting a derived type component to be a DO loop control variable might cause more problem reports than not. Like temporal issues of an out called subroutine/function referencing the derived types component for the iteration number. Optimizations would be hosed.

It would be nice though for the error message to state something to the effect that a derived type component cannotbe used asa DO loop control variable. As it is now, the error message looks like token fault in the compiler.

Jim Dempsey

[jimdempseyatthecove]

Jugoslav,

An example in my code is

#define TENSEG pTether.pFiniteSolution.rTENSEG

Where TENSEG used to be in common. This was an array of segment tension variables in a finite element solution program simulating tethers. The old F77 code would block copy into a working set of data (position, velocity, acceleration and about 100 other massive chunks of data). F90 can use pointers. So I can have a pointer to a Tether which contains tether state information one of which is a pointer to Bead Model Finite Solution state and one of the Bead Model Finite Solution states is a list of segment tension (pointer to rTENSEG).

Not wanting to introduce typographical errors or having a change in code from old code to new code I chose to use #defines.

TENSION = TENSEG(JSEG)

works for the old code as well as for the new code. Other than for some #includes at the top of the source file the old code was sufficient as-was.

Also note, if you have a loop with many such references you can insert a local pointer to the rTENSEG, insert the => for the pointer and then redefine the TENSEG macro. Thus the optimized code loop is identical to the unoptimized and is identical to the old source code.

Macros can also help avoid tedious and error prone repetitive typing

subroutine AVRDB_Restore_FiniteSolution(pTether)
 use MOD_AVRDB
 use MOD_TOSS
 type (TypeTether) :: pTether
 type (TypeFiniteSolution), pointer :: pFiniteSolution

...
 pFiniteSolution => pTether.pFiniteSolution ! make local copy
...
! use #define to specify structure containing storage type flag
#define fFS AVRDB.in.SaveAs.FiniteSolution
#define pFS pFiniteSolution
! use #define to produce uncluttered code for saving REAL
#define Restore_REAL(x) if((fFS.x) .ne. NotRecorded) call AVRDB_Restore_REAL(fFS.x, pFS.x)
#define Restore_REALarray(a) if((fFS.a) .ne. NotRecorded) call AVRDB_Restore_REAL_array(fFS.a, pFS.a, size(pFS.a))
...
 ! TETH DENSITY AT DEPLOY END (SEG 1) FOR USE IN MASS-FLOW FORCE CALCS
 Restore_REAL(rDENSEG1)

 ! POSITION VECTOR BETWEEN ATT PTS (INER FRAME)
 Restore_REALarray(rELI)
...

Jim Dempsey

[vikrantca]

Jim, thank you for the detailed code. It seems to work, but here is where I am stuck.
In order to write threadsafe dll, below is the sample code from my approach:

************ BEGIN FORTRAN CODE **************************

MODULE GLOBAL

parameter (ncmax=20) !max number of components in mixture
parameter (nrefmx=10) !max number of fluids for transport ECS
parameter (n0=-ncmax-nrefmx,nx=ncmax)
parameter (nrf0=n0) !lower limit for transport ref fluid arrays
parameter (nrefluids=4) ! number of ref fluids available for transport properties model

TYPE MYTYPE
INTEGER :: A,B,C
REAL*8 :: D1,D2
CHARACTER*255 :: string1, string2, string3
END TYPE

END MODULE GLOBAL




! Use this function to allocate the pointer and fill it with data

subroutine INITFUNC(handle)

USE GLOBAL

TYPE(MYTYPE), POINTER :: handle


! Expose subroutine FortranCDll to users of this DLL
!
!DEC$ ATTRIBUTES DLLEXPORT::INITFUNC
ALLOCATE ( handle, STAT=ierr)
! Variables

ha = 1;
handle%B = 2;
handle%C = 3;

! Body of FortranCDll

end subroutine INITFUNC


! Deallocate the pointer
subroutine DESTROYFUNC(handle)

USE GLOBAL

TYPE(MYTYPE), POINTER :: handle

DEALLOCATE(handle)

! Expose subroutine FortranCDll to users of this DLL
!
!DEC$ ATTRIBUTES DLLEXPORT::DESTROYFUNC

! Body of FortranCDll

end subroutine DESTROYFUNC


! READ the data from the handle and use it for some computation.
PURE subroutine MANFUNC(handle, res);
USE GLOBAL

TYPE(MYTYPE), POINTER :: handle
INTEGER, INTENT (INOUT) :: res

! Expose subroutine FortranCDll to users of this DLL
!
!DEC$ ATTRIBUTES DLLEXPORT::MANFUNC


! Variables

res = handle%A + handle%B + handle%C

! Body of FortranCDll

end subroutine MANFUNC


************** END FORTRAN CODE ***************************


******************************************************************
// C Code


HINSTANCE handle = LoadLibrary("FortranDLL.dll");

fpinit = (FPINIT)GetProcAddress(handle,"INITFUNC");

fpman = (FPMAN)GetProcAddress(handle,"MANFUNC");


int thandle = 0;
fpinit(thandle)

// Now thandle points to MyType type variable

// Send the thandle to a function that will use the data in thandle


int res = 0;

fpman(thandle, &res);

// Here we get the computation result in res.

// Call destroy function to deallocate the thandle.


******* END C Code **********************************

Above is the code for the f90 dll and the corresponding C code that will call the DLL.
The issue is with lines such as (from above code):

res = handle%A + handle%B + handle%C

instead of handle%A, the original variable A is in the common block. In the code, I want to replace (thr u preprocessor) all the occurences of A with handle%A. I am new at fortran, and for some reason I cannot get the # define to work in this case.

Thank you very much for your help.

- VA

[jimdempseyatthecove]

VA,

General comments.

Your C code loads the FortranDLL- this is good.

Your C codegets the entry point to INITFUNC and places the entry point into fpinit- this is good.

Your Cgets the entry point to MANFUNC and places the entry point into fpman - this is good

Your C code in initializing thandle to NULL - this is good practice.

Your C calls fpinit(thandle) - this is NOTgood as you want to use fpinit(&thandle).

Your C call fpman(thandle) without the & - this is good.

Alternately you could declare INITFUNC in Fortran as a function taking no arguments and returning a pointer to MYTYPE object. Then use:

int thandle = fpinit(); // this is prefered

In the Fortran source module you need to insert the #defines for the token substitution. #defines are global to the source file from the point of #define downwards. Therefore if your source file contains the MODULE GLOBAL you would place the #defines into your source code _after_ the declaration for the variable types containing the variable names to be substituted.

#define A thandle%a

Would be placed after end of MODULE GLOBAL and before the first use of A. Also, if you have several subroutines in the source file you would NOT insert multiple #defines for the symbole. As in doing so A might become thandle%thandle%thandle%a. I tend to use a filename.DEF file containing the #defines and then use #include "filename.DEF".

Note, you cannot insert the #defines into the MODULE GLOBAL (after end of module)if the module source code is not included in the source file. That is to say, if you compile the module files seperately (standard practice) then the #defines must be located in the source files that contain the USE of the modules (provided your intentions are to use the text substitutions).

The Fortran Preprocessor is case sensitive, Fortran generally is not. Therefore I tend to use low casecharacters in the structure such as to avoid naming conflicts. I found it handy to prepend lowcase i, r, b, c, etc... to the symbol name

TYPE MYTYPE 
 INTEGER :: iA,iB,iC
 REAL*8 :: rD1,rD2
 CHARACTER*255 :: cstring1, cstring2, cstring3
END TYPE 

Then

#define A thandle%iA

This will avoid recursive text substitution if you mistakenly define something twice.

See how far you get with these suggestions.

Jim Dempsey

[vikrantca]

Hello Jim,
Thanks for the comments. I was missing the part where the #define is declared after the module definition.
The code works fine now. WRT the first few comments, I am using the address of operator from the C code, just forgot to type it here in the post.

Thank you.

-VA