Re: Problem with OMP CRITICAL code generation

Dogbite · ‎05-28-2009

Having migrated my program to IVF, I'm now preparing to add OMP directives. I am aiming at the large loop that consumes some 80% of processing time and calls a few dozen subroutines. As several of these subroutines will be writing to the four temporary data files, I expected to have to set up some CRITICAL regions in these subroutines.

Working first with a sample program, I have had difficulty getting the compiler to generate code for CRITICAL regions in subroutines. I have included below a scaled down version of the sample program. It contains the main procedure and the subroutine Process. These reside in separate disk files, but I have combined them into a single file for inclusion here.

The problem is that while the compiler recognizes the Critical region in the main program it does not recognize the Critical region in the subroutine -- however, as I recompiled and tested the snippet below, with the two routines in a single disk file, Lo! and Behold!, it generated code for both regions. It even executes as expected.

How can I make this work using separate disk files?

[plain]      PROGRAM EXAMPLES
      
      USE EXAM_PARAMS
      include "omp_lib.h"
      INCLUDE 'link_fnl_static.h'
!DEC$ OBJCOMMENT LIB:'libiomp5md.lib'
      INTEGER LOOP_CTR
!$OMP THREADPRIVATE (TID)

      PRINT*,' OMP_ACTIVE 1 = ',OMP_IN_PARALLEL()     
      CALL OMP_SET_NUM_THREADS(4)
!$OMP PARALLEL      
      TID = OMP_GET_THREAD_NUM()
      PRINT*,' TID/ACTIVE =', TID, OMP_IN_PARALLEL()    
!$OMP DO  
      DO LOOP_CTR = 1,4
         CALL PROCESS(LOOP_CTR,TID)
!$OMP CRITICAL (CPX)
         PRINT*,' -- Ex 1 !  TID = ',TID
         PRINT*,' -- Ex 2 !  TID = ',TID
!$OMP END CRITICAL (CPX)     
      END DO
!$OMP END DO
!$OMP END PARALLEL
      END
C ---------------------------------------------------      
      SUBROUTINE PROCESS(LOOP_CTR_ARG,TID_P2_ARG)
C
      USE EXAM_PARAMS
      include "omp_lib.h"
      INCLUDE 'link_fnl_static.h'
      INTEGER TID_P2_ARG
      INTEGER LOOP_CTR_ARG
      INTEGER I
      
      PRINT*,' '
      PRINT*,' P2: CTR/TID/ACTIVE =',TID_P2_ARG,LOOP_CTR_ARG,
     &   OMP_IN_PARALLEL()
!$OMP CRITICAL (CP2)
      DO I=1,3
         PRINT*,'-- CP2: TID/ARG = ',OMP_GET_THREAD_NUM(),TID_P2_ARG
      END DO
!$OMP END CRITICAL (CP2)     
      RETURN
      END
[/plain]

jimdempseyatthecove · ‎05-29-2009

Check the properties on the project that builds the subroutine, as well as the file within the project itself, and verify that you are compiling with Process OpenMP Directives enabled.

If that fails, then in your dummy MAIN place a break at CALL PROCESS. At break in debugger, open Threads Window and Freeze threads that are notthe current thread (marked with yellow arrow). To freeze, right-click on thread in Threads window, pick Freeze. Thaw is the reverse of this.

Once you have the other threads frozen, use F11 to step into the subroutine. When you reach the !$OMP CRITICAL, open the Dissassembly Window, look for a call to a omp_... or kmp_... subroutine with the name containing critical. Or simply F10 in the Dissassembly window until you either reach the call or reach the source code of a statement after !$OMP Critical. If you do not call in to omp_... or kmp_... before you reach the first statement within the critical section then the compiler is not honoring the !$OMP CRITICAL statement.

Jim Dempsey

Dogbite · ‎05-29-2009

Quoting - jimdempseyatthecove

Check the properties on the project that builds the subroutine, as well as the file within the project itself, and verify that you are compiling with Process OpenMP Directives enabled.

If that fails, then in your dummy MAIN place a break at CALL PROCESS. At break in debugger, open Threads Window and Freeze threads that are notthe current thread (marked with yellow arrow). To freeze, right-click on thread in Threads window, pick Freeze. Thaw is the reverse of this.

Once you have the other threads frozen, use F11 to step into the subroutine. When you reach the !$OMP CRITICAL, open the Dissassembly Window, look for a call to a omp_... or kmp_... subroutine with the name containing critical. Or simply F10 in the Dissassembly window until you either reach the call or reach the source code of a statement after !$OMP Critical. If you do not call in to omp_... or kmp_... before you reach the first statement within the critical section then the compiler is not honoring the !$OMP CRITICAL statement.

Jim Dempsey

Thanks, Jim.

Not only do I have Process OpenMP Directives set, but I also have OpenMP Diagnostic Level set to /Qopenmp_report:2. When compiling with the single input file (containing both the main and the subroutine) I get the notations:

[plain]1>C:Ex_letters.for(39): (col. 7) remark: OpenMP multithreaded code generation for CRITICAL was successful.
1>C:Ex_letters.for(18): (col. 7) remark: OpenMP multithreaded code generation for CRITICAL was successful.
1>C:Ex_letters.for(15): (col. 7) remark: OpenMP DEFINED LOOP WAS PARALLELIZED.
1>C:Ex_letters.for(12): (col. 7) remark: OpenMP DEFINED REGION WAS PARALLELIZED.
[/plain]

When compiling with separate input files, the compiler generates a notation for only the Critical region in the main program. This is why I believe the compiler is perhaps not recognizing the !$OMP directive, or not recognizing that the directive is in the dynamic scope of the Parallel region defined in the main program.

The code I included above is simplified from my earlier test version that also included data manipulation. In those early versions, the compiler did not recognize Threadprivate data in the called subroutine. I removed that code to simplify the question, and in the belief/hope/expectation that the two problems share a common cause.

jimdempseyatthecove · ‎05-31-2009

Dogbite,

When I compile on my system (had to comment out two of your missing header files) I do enter the critical section...

The critical section seems to be working OK on my system

[cpp] -- CP2: TID/ARG =            1           1
 -- CP2: TID/ARG =            1           1
 -- CP2: TID/ARG =            1           1
 -- CP2: TID/ARG =            0           0
  -- Ex 1 !  TID =            1
 -- CP2: TID/ARG =            0           0
  -- Ex 2 !  TID =            1
 -- CP2: TID/ARG =            0           0
  -- Ex 1 !  TID =            0
 -- CP2: TID/ARG =            2           2
  -- Ex 2 !  TID =            0
 -- CP2: TID/ARG =            2           2
 -- CP2: TID/ARG =            2           2
  -- Ex 1 !  TID =            2
  -- Ex 2 !  TID =            2
 -- CP2: TID/ARG =            3           3
 -- CP2: TID/ARG =            3           3
 -- CP2: TID/ARG =            3           3
  -- Ex 1 !  TID =            3
  -- Ex 2 !  TID =            3
[/cpp]

Although the Ex 1 and Ex 2 lines have text printed between the lines this is due to the interviening print line being issued within a different named critical section.

Jim Dempsey

Dogbite · ‎06-01-2009

Quoting - jimdempseyatthecove

Dogbite,

When I compile on my system (had to comment out two of your missing header files) I do enter the critical section...

The critical section seems to be working OK on my system

[cpp] -- CP2: TID/ARG =            1           1
 -- CP2: TID/ARG =            1           1
 -- CP2: TID/ARG =            1           1
 -- CP2: TID/ARG =            0           0
  -- Ex 1 !  TID =            1
 -- CP2: TID/ARG =            0           0
  -- Ex 2 !  TID =            1
 -- CP2: TID/ARG =            0           0
  -- Ex 1 !  TID =            0
 -- CP2: TID/ARG =            2           2
  -- Ex 2 !  TID =            0
 -- CP2: TID/ARG =            2           2
 -- CP2: TID/ARG =            2           2
  -- Ex 1 !  TID =            2
  -- Ex 2 !  TID =            2
 -- CP2: TID/ARG =            3           3
 -- CP2: TID/ARG =            3           3
 -- CP2: TID/ARG =            3           3
  -- Ex 1 !  TID =            3
  -- Ex 2 !  TID =            3
[/cpp]

Although the Ex 1 and Ex 2 lines have text printed between the lines this is due to the interviening print line being issued within a different named critical section.

Jim Dempsey

Jim,

Thanks forthe persistence of your assistance!

While I had stated above that I had the Process OpenMP Directives set, as well as the OpenMP Diagnostic Level set, it turns out that what I had set was the OpenMP Conditional Compilation flag for the Preprocessor. I was surprised to discover that there was a third OpenMP entry, this one under the Language tab, to direct the compiler to generate the desired code -- and which I had not set. When I set this flag, the program performs properly for me as well.

Setting the Process OMP Directives flag also sets the OMP Conditional Compilation condition, but I find it curious that the CondComp flag has the effect of setting the Process flag only halfway: that is, the compiler generated code within the lexical extent of a region in the main procedure, but did not recognize anything in the dynamic extent of that same region. It was confusing enough to cost me a couple of days of tail-chasing, and I wonder if thecompiler is intended to function that way, and if so, why?

Thanks again for your help, Jim.

--That dizzy dalmation,

Dogbite

Steven_L_Intel1 · ‎06-01-2009

Yes, this is intentional. Some people want to use the OMP conditional compilation facility without generating OpenMP calls in the code.

Dogbite · ‎06-01-2009

Quoting - Steve Lionel (Intel)

Yes, this is intentional. Some people want to use the OMP conditional compilation facility without generating OpenMP calls in the code.

Steve,

There's probably something I'm still not understanding here, but the starting point of my confusion was that by setting the CondComp flag for the Preprocessor, the compiler responded by generating OpenMP calls in the code even though the Process OMP flag was not set for the compiler.

Doesn't that conflict with your statement above? You seem to be saying that, for source including OMPdirectives, selecting the CondComp flag ( /assume:cc_omp ) will not result in the compiler generating OMP calls. The compiler certainly generated OMP calls for me when I set the CondComp flag but not the Process OMP flag.

What am I missing?

(scratches behind left ear)

Steven_L_Intel1 · ‎06-01-2009

Well, maybe I'm missing something. I'll go back through this thread and see what I can work out.

jimdempseyatthecove · ‎06-02-2009

Quoting - Dogbite

Steve,

There's probably something I'm still not understanding here, but the starting point of my confusion was that by setting the CondComp flag for the Preprocessor, the compiler responded by generating OpenMP calls in the code even though the Process OMP flag was not set for the compiler.

Doesn't that conflict with your statement above? You seem to be saying that, for source including OMPdirectives, selecting the CondComp flag ( /assume:cc_omp ) will not result in the compiler generating OMP calls. The compiler certainly generated OMP calls for me when I set the CondComp flag but not the Process OMP flag.

What am I missing?

(scratches behind left ear)

Perhaps the itch is behind the right ear.

A possiblereason for the difference is you may want to compile with the OpenMP directives enabled but with OpenMP stubs active. This is a single threaded version of your application passing through the OpenMP regions. Sometimes it is helpful to do this while tracking down a problem.

Jim Dempsey