Hi to everyone,
I am working on a Monte Carlo code, written in Fortran77 in order to make it parallel using OpenMP. Now I am in the testing phase of the development process, but I am facing problems with the overhead costs of the code. For example, when I analice it using Vtune Amplifier XE I obtain the following summary:
Elapsed Time: 43.352s
Total Thread Count: 5
Overhead Time: 16.560s
Spin Time: 0.847s
CPU Time: 157.369s
Paused Time: 0s
Well, the systems complains that the overhead time is too much. What is worst is that I have tested this code against gfortran and this effects are less pronunciated using the later. This is sad, because without parallelization the code compiled with ifort is much faster than gfortran, but as I increase the number of OMP threads (maintaining the load per thread constant) the overheads costs render the ifort version slower than the gfortran one.
What I have found is that the threads get "stalled" in a very disorder fashion, for example, you can see this in the image bellow
The code has several subroutines that controls all the Monte Carlo simulation process (for example, random number generation, electron and photon transport, geometry description, etc). This subroutines communicate each other using COMMON blocks, therefore I have had to flag some of them as private using the THREADPRIVATE statement when needed. The idea is to maintain the original structure of the code as much as possible, considering that this is a wide used code and the idea is to offer an easy transition to parallelization with OpenMP without changing the core of the program.
I have created a small code that runs only the random number generator and use them to estimate the value of PI. This code has also the same problem as the original code. In this last one what have I found is that the function _kmp_get_global_thread_id_reg has a great part of the overhead time:
Well, I would really appreciate if someone has a tip to face this problem. I have tried to search info about this problem without success. Thanks for your help!!
Link Copied
I did a mistake putting the images, here are the correct
and the second
Well, I still have overhead problems with the Monte Carlo code, looking at the amplifier concurrency test results I found that most of the overhead occur when some functions of the MC code calls this function : __kmp_get_global_thread_id_reg
Inside this functions the only OMP statements are of the THREADPRIVATE type, for example for AUSGAB:
subroutine ausgab(iarg)
implicit none
integer*4 iarg,irl,jp
real*8 aux
common/score/ sc_array(102),sc_array2(102), sc_tmp(102),sc_last(10
*2), sc_pulse_height(200,102), icase,ipulse,de_pulse
real*8 sc_array, sc_array2, sc_pulse_height, sc_tmp,de_pulse
integer*4 sc_last,icase,ipulse
C$OMP0THREADPRIVATE(/score/)
COMMON/EPCONT/EDEP,TSTEP,TUSTEP,USTEP,TVSTEP,VSTEP, RHOF,EOLD,ENEW
*,EKE,ELKE,GLE,E_RANGE, x_final,y_final,z_final, u_final,v_final,w_
*final, IDISC,IROLD,IRNEW,IAUSFL(31)
C$OMP0THREADPRIVATE(/EPCONT/)
DOUBLE PRECISION EDEP
real*8 TSTEP, TUSTEP, USTEP, VSTEP, TVSTEP, RHOF, EOLD, ENE
*W, EKE, ELKE, GLE, E_RANGE, x_final,y_final,z_final, u_final,
*v_final,w_final
integer*4 IDISC, IROLD, IRNEW, IAUSFL
COMMON/STACK/ E(50),X(50),Y(50),Z(50),U(50),V(50),W(50),DNEAR(50),
*WT(50),IQ(50),IR(50),LATCH(50), LATCHI,NP,NPold
C$OMP0THREADPRIVATE(/STACK/)
DOUBLE PRECISION E
real*8 X,Y,Z, U,V,W, DNEAR, WT
integer*4 IQ, IR, LATCH, LATCHI, NP, NPold
can be the threadprivate clause so costly for the program, the strange thing is that now I do not see this behavior in the example codes that I gave above, that also have threadprivate clauses. Well, I suppose that porting legacy codes is not so simply... and finally, where could I find information about the __kmp_get_global_thread_id_reg (or other omp library) functions? Thanks for your help!.
Thanks to both for your comments, well, to obtain the assembly code I had to use the Debug mode (as suggested by the link).
using just the Release mode I obtain a similar pattern as the pictures shown above, looking at the compiling options of the Release mode I did not find any runtime check or diagnostics enabled, in the Command Line section the following appears:
/nologo /O2 /fpp /Qopenmp /module:"Release\\" /object:"Release\\" /Fd"Release\vc120.pdb" /libs:dll /threads /c
Well, just to be sure I decided to compile the program outside VS2013 (i.e. using the command prompt) with the following flags:
/fpp /Qopenmp /O3 /fast
and the I ran a concurrency analysis using the command prompt. I obtained the following results:
so now it seems that the overhead have decreased a lot comparing with previous results. There are still some overhead, now I feel more satisfied with the results. Anyway, also this program have many small subroutines where I need to use THREADPRIVATE clauses (for example, subroutine SHOWER calls PHOTON that calls BREMS that calls UPHI and then AUSGAB etc etc etc, each of them having their own THREADPRIVATE calls), so I think that this structure is also a major contribution to the total overhead cost.
Thanks for your help!
thanks for the comments,
I compiled with /fpp /Qopenmp /O3 /fast /debug:all /check:none flags obtaining the following.
If I double click on any subroutine like UPHI, MSCAT, etc I jump to the declaration of each subroutine (subroutine UPHI ....). If I doule clock on __kmp_get_global_thread_id_reg its showed what it seems to be assembly code of the libomp5md.dll
well, I will study more the code, but I have had a great advance thanks to you!, I really appreciate your help ;)
Well, Jim, as far as I know I do not have nested parallel regions, is there a way to have (unintentionally) nested regions without explicitly using the PARALLEL directive?. I only have one parallel region:
C$OMP0PARALLEL PRIVATE(OMP_IAM) COPYIN(/egs_vr/,/EPCONT/) C$OMP*COPYIN(/ET_control/,/PHOTIN/,/randomm/,/UPHIOT/,/USEFUL/,/score/) C$OMP0SINGLE #ifdef _OPENMP OMP_TOT = OMP_GET_NUM_THREADS() WRITE(6,1060)OMP_TOT 1060 FORMAT(//'Number of OpenMP threads = ',I2//) C$OMP0END SINGLE NOWAIT #endif #ifdef _OPENMP OMP_IAM = OMP_GET_THREAD_NUM() #endif ixx = 1802 jxx = 9373 + OMP_IAM call init_ranmar C$OMP0DO SCHEDULE(static) DO 1071 icase_aux=1,ncase icase = icase_aux call shower(iqin,ein,xin,yin,zin,uin,vin,win,irin,wtin) 1071 CONTINUE 1072 CONTINUE C$OMP0CRITICAL DO 1081 I=1,102 r_sc_array(I) = r_sc_array(I) + sc_array(I) r_sc_array2(I) = r_sc_array2(I) + sc_array2(I) r_sc_tmp(I) = r_sc_tmp(I) + sc_tmp(I) DO 1091 j=1,200 r_sc_pulse_height(j,i) = r_sc_pulse_height(j,i) + sc_pulse_hei * ght(j,i) 1091 CONTINUE 1092 CONTINUE 1081 CONTINUE 1082 CONTINUE C$OMP0END CRITICAL C$OMP0END PARALLEL
The shower subroutine takes almost all the time length of the program. I thought about the THREADPRIVATE (or other clauses) because this program is composed of many small subroutines, and for each loop the program enters and exists these subroutines many many times. The other strange thing is that the overhead is much less when using gfortran, for example running this code for an AMD FX8350 (8 cores) CPU under Debian 7.5 I obtained the following graph when increasing the number of OMP threads (the ammount of particles is fixed):
(The BQS stands for batch queuing system, is the original parallelizing method) So one can see that at the beginning the ifort performance is much better than gfortran, but as the number of OMP Threads increases the runtimes start to get closer... the compilation flags were:
gfortran : -cpp -fopenmp -O3 -ffast-math
ifort: -fpp -qopenmp -O3 -fast -check none (the last just in case)
at the beginning of this thread the problems were much higher, but I still have this ifort v/s gfortran performance issue... thanks for your help!
