Intel® Fortran Compiler
Build applications that can scale for the future with optimized code designed for Intel® Xeon® and compatible processors.

OpenMP Module Variable visibility

philliard
Beginner
945 Views

I am trying to set up an OpenMP fortran code with a task thread that listens for input from the user and when received sets a variable that the main thread uses to determine when to stop running.  I created the following code that seems like it should work.  There is a module level logical that is set to .false. by default, but the task thread sets it to true when receiving a specific string from the user.  This code works as expected in Debug, however, in release, the main thread doesn't ever see that the variable has changed.

module threads
    implicit none
    
    logical :: killThread 
    integer :: testVariable
    
contains
    
    subroutine taskThread()
        character(2) :: killme
        
        do while (.true.)
            
            ! waiting for user to input
            read(*,*) killme
            print *, killme
            
            ! if user inputs exact string 'OK' then exits infinite loop and 
            ! sets module variables
            if (killme == 'OK') then
                killThread = .true.
                testVariable = 1
                exit
            end if
        end do 
        
        print *, 'exiting thread 0, testVariable = ', testVariable, ', address = ', loc(testVariable)
        
    end subroutine
    
 
end module
    
program threadtest
    use omp_lib
    use threads
    implicit none

    integer :: thread
    integer :: flushStart, flushStop, ticksPerSecond
        
    ! these should be shared module level variables that can be written and read by all threads
    killThread = .false.
    testVariable = 0
    
    call SYSTEM_CLOCK(flushStart, ticksPerSecond)
    
    ! create two threads
    call omp_set_num_threads(2)
    
    !$omp parallel
    !$omp single
        thread = omp_get_thread_num()
    
        !$omp task
            call taskThread()
            print *, 'exited, thread 0, testVariable = ', testVariable, ', address = ', loc(testVariable) 
        !$omp end task
        
            
        do while (.true.)
                
            ! infinite loop that just prints to the screen once per second until the module variable
            ! "killThread" is set to true
            call SYSTEM_CLOCK(flushstop, ticksPerSecond)
            if (real(flushStop-FlushStart) / ticksPerSecond > 1.0) then
                print *, 'tick thread 1, testVariable = ', testVariable, ', address = ', loc(testVariable)
                call SYSTEM_CLOCK(flushStart, ticksPerSecond)
            end if
                
            if (killThread) then
                exit
            end if
                
        end do

    !$omp end single
    !$omp end parallel
    

    print *, 'exiting program'
    
end program

 

The issue is that the variable killThread is being set to true in the task thread, but in the main thread, it never sees it and never stops running.  Everything I have seen with OpenMP and fortran is that module variables are supposed to be shared unless set to THREADPRIVATE (which these are not).  I also set a test integer to see if it is getting changed.  The value does get changed in the task thread, but the main thread still sees it as 0.  The address of the variable reports the same in both cases, so I am confused.

 

0 Kudos
16 Replies
Steve_Lionel
Honored Contributor III
945 Views

You need to add the VOLATILE attribute to the module variable. Otherwise the compiler sees there is nothing in the loop that sets the variable and therefore doesn't check it again.

0 Kudos
philliard
Beginner
945 Views

Thanks Steve.   That worked and I think I understand why.

I actually had thought it was something like this (in Release due to optimization).  I had tried to put in a condition in the loop that would change the value to true after the 500th iteration, but it seems like that did not work.

            if (count > 500) then
                killThread = .true.
            end if

why would that not tell the compiler it can't optimize that check away?

0 Kudos
Steve_Lionel
Honored Contributor III
945 Views

I'd have to see the whole modified code to analyze, since as you originally posted there is no count.

0 Kudos
philliard
Beginner
945 Views

OK, here you go.  Just an integer counter that is initialized outside the infinite loop that increments at every tick.  Like I said, your original fix worked fine, I'm just trying to understand.

module threads
    implicit none
    
    logical :: killThread 
    integer :: testVariable
    
contains
    
    subroutine taskThread()
        character(2) :: killme
        
        do while (.true.)
            
            ! waiting for user to input
            read(*,*) killme
            print *, killme
            
            ! if user inputs exact string 'OK' then exits infinite loop and 
            ! sets module variables
            if (killme == 'OK') then
                killThread = .true.
                testVariable = 1
                exit
            end if
        end do 
        
        print *, 'exiting thread 0, testVariable = ', testVariable, ', address = ', loc(testVariable)
        
    end subroutine
    
 
end module
    
program threadtest
    use omp_lib
    use threads
    implicit none

    integer :: thread, count
    integer :: flushStart, flushStop, ticksPerSecond
        
    ! these should be shared module level variables that can be written and read by all threads
    killThread = .false.
    testVariable = 0
    
    call SYSTEM_CLOCK(flushStart, ticksPerSecond)
    
    ! create two threads
    call omp_set_num_threads(2)
    
    !$omp parallel
    !$omp single
        thread = omp_get_thread_num()
    
        !$omp task
            call taskThread()
            print *, 'exited, thread 0, testVariable = ', testVariable, ', address = ', loc(testVariable) 
        !$omp end task
        
            
        count = 0
        do while (.true.)
                
            ! infinite loop that just prints to the screen once per second until the module variable
            ! "killThread" is set to true
            call SYSTEM_CLOCK(flushstop, ticksPerSecond)
            if (real(flushStop-FlushStart) / ticksPerSecond > 1.0) then
                print *, 'tick thread 1, testVariable = ', testVariable, ', address = ', loc(testVariable)
                count = count + 1
                print *, count
                call SYSTEM_CLOCK(flushStart, ticksPerSecond)
            end if
                
            if (count > 500) then
                killThread = .true.
            end if
            
            if (killThread) then
                exit
            end if
                
        end do

    !$omp end single
    !$omp end parallel
    

    print *, 'exiting program'
    
end program
0 Kudos
Steve_Lionel
Honored Contributor III
945 Views

And you waited 500 seconds to see if it exited? I would have expected that to work, but haven't tried it myself.

By the way, you can replace "do while (.true.)" with just "do".

0 Kudos
gib
New Contributor II
945 Views

I changed 500 to 10 (why wait 500 sec?) and added the line

if (killThread) exit

before the read statement in taskThread.  Then after 10 seconds it is necessary to type something and hit Enter to kill the thread, because it is always waiting for console input.  The added line can be before or after the read statement.

0 Kudos
Steve_Lionel
Honored Contributor III
945 Views

I tried the program and, while I don't really understand OpenMP well, it appeared to do what it was supposed to. I added some more print statements to make sure I knew what was happening. Here's where I modified it:

        myloop: do while (.true.)
                
            ! infinite loop that just prints to the screen once per second until the module variable
            ! "killThread" is set to true
            call SYSTEM_CLOCK(flushstop, ticksPerSecond)
            if (real(flushStop-FlushStart) / ticksPerSecond > 1.0) then
                print *, 'tick thread 1, testVariable = ', testVariable, ', address = ', loc(testVariable)
                count = count + 1
                print *, count
                call SYSTEM_CLOCK(flushStart, ticksPerSecond)
            end if
                
            if (count > 10) then
                killThread = .true.
                print *, "Setting killthread true"
            end if
            
            if (killThread) then
                print *, "killThread is true"
                exit myloop
            end if
                
        end do myloop
        
        print *, "Outside loop"

When I run this I see:

 tick thread 1, testVariable =            0 , address =        140694945976440
          11
 Setting killthread true
 killThread is true
 Outside loop
OK
 OK
 exiting thread 0, testVariable =            1 , address =
       140694945976440
 exited, thread 0, testVariable =            1 , address =
       140694945976440
 exiting program

Isn't this what you want?

0 Kudos
philliard
Beginner
945 Views

Thanks, Steve.

The point of this program is to:

  1. execute some function on a main thread (in this case an infinite loop that ticks once per second) - while simultaneously...
  2. kick off a second thread that will wait for user input and exit everything if the user enters a certain string

If the user enters "OK" at any time then the program exits.

In your implementation, I think the main loop is just stopping when the counter reaches 11.  I made it 500 because I don't want it to stop on a timer, rather I ONLY want it to stop when the user enters the string "OK" and presses enter.   I only put that counter and extra exit condition in there because you were saying that the reason that if check wasn't done is the compiler saw that killthread could never change in the loop and optimizes that check away.  Like I said, in its original form it works in debug mode. So if I had the counter and extra exit condition in the code, even if I never expected to let it go that long, why was volatile still required on the variable?

what I want is this...

 tick thread 1, testVariable =            0 , address =        140695989965200
           1
 tick thread 1, testVariable =            0 , address =        140695989965200
           2
 tick thread 1, testVariable =            0 , address =        140695989965200
           3
OK tick thread 1, testVariable =            0 , address =        140695989965200
           4

 OK
 exiting thread 0, testVariable =            1 , address =
       140695989965200
 killThread is true
 exited, thread 0, testVariable =            1 , address =
       140695989965200
 Outside loop
 exiting program

Note, that as soon as I press OK the code exits.  But what the code (in Release) does is this...

 tick thread 1, testVariable =            0 , address =        140696553034056
           1
 tick thread 1, testVariable =            0 , address =        140696553034056
           2
 tick thread 1, testVariable =            0 , address =        140696553034056
           3
 tick thread 1, testVariable =            0 , address =        140696553034056
           4
OK
 OK
 exiting thread 0, testVariable =            1 , address =
       140696553034056
 exited, thread 0, testVariable =            1 , address =
       140696553034056
 tick thread 1, testVariable =            0 , address =        140696553034056
           5
 tick thread 1, testVariable =            0 , address =        140696553034056
           6
 tick thread 1, testVariable =            0 , address =        140696553034056
           7

Note, after pressing OK, I can see that the taskthread exited (exited, thread 0, testVariable =...), but the main loop keeps ticking.  Making killThread volatile fixes this behavior, but I'm not sure why it is required since there is a clear condition in the code that CAN change it. I am ok with making the variable volatile.  I'm just trying to understand why it is necessary.

0 Kudos
Steve_Lionel
Honored Contributor III
945 Views

If I understand the assembly correctly, the compiler created a shadow variable (called r14b in my assembly listing) and it tested that. The code that set killThread based on count set both the module variable and the shadow variable. Since the test only looks at the shadow variable, that's why it doesn't trigger when you enter OK. Adding VOLATILE alerts the compiler that it really needs to check the module variable.

Here's the assembly when count>10:

        mov       DWORD PTR [THREADS_mp_KILLTHREAD], -1         ;75.17
        mov       r14b, -1                                      ;75.17

and here's the test:

        test      r14b, 1                                       ;79.17
        je        .B1.40        ; Prob 80%                      ;79.17

If I make killThread VOLATILE, I get this instead:

        mov       DWORD PTR [THREADS_mp_KILLTHREAD], -1         ;76.17
        mov       r8d, DWORD PTR [THREADS_mp_KILLTHREAD]        ;80.13
        test      r8d, 1                                        ;80.17
        je        .B1.41        ; Prob 80%                      ;80.17

 

0 Kudos
mecej4
Honored Contributor III
945 Views

Steve, r14b is the MASM style name for the lowest 8 bits of register R14 in X64; see, for example, https://software.intel.com/en-us/articles/introduction-to-x64-assembly . As with X86, there are no instructions that involve purely memory operands; i.e., with no registers involved. 

0 Kudos
Steve_Lionel
Honored Contributor III
945 Views

Ah, I didn't know that. Makes sense that it is using a register for the shadow. I never studied x64 assembly. Thanks.

0 Kudos
philliard
Beginner
945 Views

I read a book on Assembly in around 1990, so I won't pretend I understand all of what you wrote.  :)

So is the bottom line reason the original variable is not checked in lieu of the shadow variable is because it is being set in a separate thread.  Similar to a situation where something is coming from the operating system or something like that?

This sort of seems like a bug in the OpenMP implementation.  It is my understanding that module variables are supposed to be shared by all threads in OpenMP.  So any module variable I set in any thread seems like it should be interpreted correctly by any other thread.  But I am satisfied with using the volatile attribute to make it work correctly.

Thanks again for the help.

0 Kudos
Steve_Lionel
Honored Contributor III
945 Views

It's not a bug. Without VOLATILE, the compiler is allowed to assume that a variable isn't modified in ways not visible to it, such as other threads. This permits better optimization. Situations such as this are exactly what VOLATILE was designed for.  Fortran 2018 expanded the use of ASYNCHRONOUS to be a "smaller hammer", however. I tried it with your example and it worked there too.

The difference is that with ASYNCHRONOUS, the compiler is allowed to use a register after it fetches the value for a reference, with VOLATILE it always has to go to memory.

0 Kudos
philliard
Beginner
945 Views

That makes a lot of sense.  Thanks for all your help.

0 Kudos
IanH
Honored Contributor II
945 Views

VOLATILE is probably overkill for this.  What is missing is an OpenMP flush operation that ensures that the thread executing the infinite loop has a a consistent view of the program's memory.  (There is an implicit flush when the task ends, but that just ensures that the temporary view of the thread running the task is consistent with memory, it doesn't affect the other thread running the loop.)

Add !$OMP FLUSH(killThread) prior to the reference to killThread inside the conditional expression of the if statement inside loop and see what happens.  See the section of your chosen edition of the OpenMP specification on the "Memory Model", and also the documentation of the FLUSH construct.

(Note that writing to killThread from within the infinite loop also introduces a race condition.)

0 Kudos
gib
New Contributor II
945 Views

IanH (Blackbelt) wrote:

Add !$OMP FLUSH(killThread) prior to the reference to killThread inside the conditional expression of the if statement inside loop and see what happens.

That does it.

0 Kudos
Reply