Need Help with Pointer to One-Dimensional Array in SMP Processing Application

[Eric_S_4]

Hello.  I am in need of some technical guidance regarding appropriate OpenMP directives for an SMP application.  I am currently working with a commercial finite element software to develop a user-defined material model.  Thus, I am given a collection of SMP object files.  I input my code into a specific subroutine and build an executable with the rest of the provided object files.  I'm just mentioning this to provide some background and to let you know that I don't have access to all of the source code.  During a finite element analysis, the material subroutine gets called many times.  Within the subroutine that I'm working on, I declare a pointer to a one-dimensional array that is given the SAVE attribute.  The objective of this one-dimensional array is to collect data during each call to the subroutine and, at a certain point in the analysis, I want to operate on the array.  This is why I don't want the array to go out of scope once the subroutine returns.  When I build the executable and run a finite element simulation with a single CPU the user-defined subroutine works great.  However, when I try running the same simulation using multiple CPUs in SMP format my SAVED pointer gets garbled up.  I've confirmed this via various WRITE statements throughout the subroutine.  This makes me think that perhaps the array is getting broken up in memory due to the SMP processing.  I've been reading about the THREADPRIVATE compiler directive.  I tried declaring the pointer to the one-dimensional array as THREADPRIVATE, but the values in the array still get garbled up during a multi-CPU SMP simulation.  Are there other compiler directives or anything else specific I need to be doing to ensure that my one-dimensional array remains in tact and meaningful during SMP processing?

Thank you in advance for your help!

[jimdempseyatthecove]

From your description, your SAVE array behaves like a buffer accumulating data in one part of your program. If the accumulation is performed in a parallel section, you must be certain that the "fill" section is thread safe. While you can use a critical section, an atomic InterlockedExchangeAdd can be more efficient should a next index be sufficient.

Likewise, when after building the SAVE array, you transition into processing the data, the "empty" (get next thing to work on) must be thread safe. In addition to this, the concurrent processing of things from the array must be thread safe. An example of this is interacting particles where one thread is forming the interaction between particles A and B and another thread is forming the interaction between particles B and C. You may have concurrency issues of accumulating Force in particle B in this situation.

If each thread in the processing of data needs use of a temporary array that needs to be preserved between calls, then either thread private, or pass the (thread unique) array in and out of your subroutine. If you many such temporaries, then consider placing them into a user defined type, create an array of these types (one for each thread), then either pass the reference to the thread's entry or let the thread use it's OpenMP thread number as an index. **** caution, OpenMP thread numbers are actually team member numbers. If you enable nested parallelism, each team has its team member numbers starting at 0.

Jim Dempsey

[Greg_T_]

Hi Eric,

Could you use the solution dependent state variables in Abaqus to save data at the element Gauss integration points, instead of using a local array?  I've used the *DEPVAR command in the *MATERIAL data to save and access values in a user subroutine during the analysis.  Since the state variables are managed in Abaqus, perhaps that could help avoid the problems when using multicore?

Regards,
Greg

[Eric_S_4]

Hi Everyone,

Thank-you for your helpful responses.  I've been investigating this issue today and determined that the best fix (at least for now) is to simply define a critical region with the !$omp critical and !$omp end critical compiler directives while operating on the SAVED arrays.  This appears to be working, but may not be most efficient as Jim pointed out. 

Thanks again!

[jimdempseyatthecove]

Eric,

Are you able to show us what you placed into the critical section (with copious comments)?

Jim Dempsey