topic How to minimize thread creation overhead in Intel Fortran/OpenM in Intel® Fortran Compiler

How to minimize thread creation overhead in Intel Fortran/OpenMP?

virtualmemory — Thu, 07 Jun 2012 19:21:20 GMT

I have a compute-bound Fortran pgm which I am attempting to parallelize using OpenMP. The outline of this pgm is below. The parallel form below takes almost 3 times longer than the serial form, and according to VTune, it is due to thread creation overhead.

I am wondering how I can bring the thread creation outside all the DO-loops, yet still execute all but the iSP loop (the parallel region in the code below) serially.

The outer loops cannot be parallelized because the values of array a at each time depend on values at the preceeding time (this is a PDE with time and position as independent variables). Also, the 'iTry' loop has a conditional EXIT, which is usually taken.

DO iTime=1,nTimes
...
DO iTry=1,nTries
...
!$OMP PARALLEL
!$OMP DO
DO iSp=1,nSp
DO j=1,4000
a(iSp,j)=...
END DO ! j
END DO ! iSp
!$OMP END DO
!$OMP END PARALLEL
...
END DO ! iTry
...
END DO ! iTime

How to minimize thread creation overhead in Intel Fortran/OpenM

jimdempseyatthecove — Thu, 07 Jun 2012 21:32:23 GMT

If nTries is very large, give the following a try

!$OMP PARALLEL PRIVATE(iTime, iTry, j)
DO iTime=1,nTimes
!$OMP MASTER
...
!$OMP END MASTER
DO iTry=1,nTries
!$OMP MASTER
...
!$OMP END MASTER
!$OMP DO
DO iSp=1,nSp
DO j=1,4000
a(iSp,j)=...
END DO ! j
END DO ! iSp
!$OMP END DO
!$OMP MASTER
...
!$OMP END MASTER
END DO ! iTry
!$OMP MASTER
...
!$OMP END MASTER
END DO ! iTime
!$OMP END PARALLEL

The above will move the parallel region to outside your nTimes and nTriesloops.
*** each thread executes full range of nTimes and nTries...
*** however, only master thread executes the ...
*** and there is an implied barrier at !$OMP END DO

Jim Dempsey

How to minimize thread creation overhead in Intel Fortran/OpenM

jimdempseyatthecove — Thu, 07 Jun 2012 21:33:41 GMT

In your original code make j PRIVATE

How to minimize thread creation overhead in Intel Fortran/OpenM

virtualmemory — Thu, 07 Jun 2012 22:28:00 GMT

Jim -

nTries is a user-set constant, generally 4 or less. Under normal circumstances, only a single try is necessary, so the loop exits after the first try.

There are many private variables, including j, but I did not show these in the interest of simplicity.

I did try something like your MASTER approach, but was unable to get it to compile due to the interleaving of Fortran and OMP blocks. I will have another look.

If I were not using OpenMP, I would just start nSp worker threads (this is Windows) and have them be idle until the iSp loop starts.

How to minimize thread creation overhead in Intel Fortran/OpenM

jimdempseyatthecove — Fri, 08 Jun 2012 02:43:15 GMT

>>If I were not using OpenMP, I would just start nSp worker threads (this is Windows) and have them be idle until the iSp loop starts.

In OpenMP, the first !$OMP PARALLEL region creates the thread pool. In an application this first time thing happens only once. For your timing insert

!$OMP PARALLEL
write(*,*) omp_get_thread_num() ! or some code that does not optimize out
!$OMP END PARALLEL
....
Now run your timed session

Note, the initial thread startup is generally negligible...
unless you have some initialization going on...
like a large thread private area
and/or large stack that gets touched

The above code will eliminate those variables from your test.

Jim Dempsey

How to minimize thread creation overhead in Intel Fortran/OpenM

TimP — Fri, 08 Jun 2012 11:03:51 GMT

It should be sufficient to set KMP_BLOCKTIME long enough that the threads persist between entries to parallel regions (default 200 ms). Both environment variable and subroutine call alternatives are available.

How to minimize thread creation overhead in Intel Fortran/OpenM

virtualmemory — Thu, 21 Jun 2012 20:38:52 GMT

I tried increasing KMP_BLOCKTIME to 10,000 msec (10 sec), with no change in execution time.

I also tried decreasing the stack size from the default 2 MB to 1 MB, also with no effect.

I need some tools to help me understand what's happening. Parallel execution is taking about 1.5 _longer_.

How to minimize thread creation overhead in Intel Fortran/OpenM

Steven_L_Intel1 — Thu, 21 Jun 2012 21:15:00 GMT

Intel VTune Amplifier XE is just what you need to analyze the thread performance

How to minimize thread creation overhead in Intel Fortran/OpenM

TimP — Thu, 21 Jun 2012 21:30:04 GMT

Inspector should catch some threading errors, but, as Steve hinted, you may find some simply by Amplifier showing where all threads are contending for access to a variable.

How to minimize thread creation overhead in Intel Fortran/OpenM

virtualmemory — Thu, 21 Jun 2012 22:13:59 GMT

I have tried VTune, but can't get it to work for this pgm. See my post of 6/20, http://software.intel.com/en-us/forums/showthread.php?t=106106, for details.