I have a Fortran program which heavily utilizes OpenMP for multithreading. The program performs some heavy computations on the input data. I've started to see performance degradation when several successive runs are performed in a single session. This problem does not happen all the time but occasionally. I further investigated the issue via VTune and the result are (OpenMP is told to use 22 threads in both sessions):
Here is a sample of when the performance stays the same across successive runs:
and here is a sample of when the performance drastically decreases between runs (I've omitted some of the threads in this pic to make it smaller):
It appears that in the first session OpenMP reuses the same threads in the second run that where originally created for the first run but in the second session OpenMP creates new threads for the second run. Also the CPU time (load) in the threads created for the second run are considerably lower than those for the first run.
I was wondering if you could point me in the right direction for resolving this issue?