Problems with OpenMP: internal compiler error in 9.1

[mhermanns]

Hi,

I have a problem in parallelizing a do-loop using OpenMP. I have a do-loop with a lot of operations in it. The used data for the operations is read from an array which IS NOT altered by the do-loop. At the end of the do-loop an entry in a different array is updated. A sketch could be as follows:

do j = 0, Ny

!read data from a never-changing array
dummy1 = A(:,j)

do i = 0, Nx

!read more data from another never-changing array
dummy2 = B(i,j)

!perform computations with the read data

!write output to a different array
C(i,j) = dummy3

enddo
enddo

Looking at the structure, the use of OpenMP for parallelizing the outer do-loop should be straightforward, but surprisingly it is not the case. The do-loop works in serial and in OpenMP with only one thread (so private and shared variables should be OK), but if I use 2 threads, the content of the final matrix C changes randomly at certain stripes C(:,j).

I have used a default(private) clause and then defined shared the minimum number of them (basically in the above example, Nx, Ny, A, B, and C). I have seen that the problem seems to be one of the read processes and not the final write step, but things change when I introduce a write statement here or there. It really looks like a race condition, but I see no reason for that .

I'm already a week with this and start to do have no idea about what is going on. Any clue about what it could be? I know that it would be great to have a look at the code, but it is a very long one with a lot of subroutine-calls.

I'm using Intel Fortran Compiler 9.0 Build 20060222 on a P4 machine with debian etch.

Thanks in advance and greetings from Spain,

Miguel

[jimdempseyatthecove]

Miguel,

In your sketched code please include the placement and contentsof the$OMP statements as well as where and how you declared the arraysA, B and C.

Additionaly, have you checked your code for dependencies? Example:

dummyDependent = B(i, j-1)

Jim Dempsey

[mhermanns]

Ok Jim,

here is the second version of my sketch :-), which includes the OMP clauses as well as the definitions of the different types of variables involved in the loop:

subroutine dummy_sub(A,B,C)
type(A_type), intent(in) :: A
real, intent(inout) :: B(:,0:,0:)
real, intent(out) :: C(:,0:,0:)

real :: dummy1(...), dummy2(...), ...

if(First_Time_Called) then
allocate(A%A1(2,0:Nx,0:Ny), &
A%A2(2,2,0:Nx,0:Ny), &
A%A3(2,2,2,0:Nx,0:Ny), &
A%A4(2,0:Nx),A%A5(2,0:Ny))

!Fill A1, A2, A3, A4, and A5 with never-again-changing data
endif

!$OMP PARALLEL DO SCHEDULE(STATIC) DEFAULT(PRIVATE) &
!$OMP SHARED(Nx,Ny,A,B,C)
do j = 0, Ny

!reads data from never-changing arrays and modifies it
dummy1 = B(:,:,j)
dummy2 = dummyfunction1(B(:,:,j1:j2),A%A5)

do i = 0, Nx

!reads more data from more never-changing arrays
dummy3 = dummyfunction2(dummy2,A%A4)
dummy4 = A%A1(:,i,j)
dummy5 = A%A2(:,:,i,j)
dummy6 = A%A3(:,:,:,i,j)

!perform computations with the read data

!write output to a different array
C(:,i,j) = dummy7

enddo
enddo
!$OMP END PARALLEL DO

...

I hope this sketch is more usefull than the previous one. The arrays B and C are allocated in the calling program, while part of A is allocated in the calling program (not shown in the sketch) and another part is initialized by the subroutine. All the dummy variables are declared in the body of the subroutine using "ubounds" from the dummy arguments of the subroutine. With dummyfunction1 and dummyfunction2 I just want to represent that operations are performed with their arguments (normally do-loops or matmuls).

Regarding the dependencies you mention, it is true that in the assignment of dummy2 in the sketch values for different "j" are involved, but those are only read processes from arrays, which are not affected by the j-do-loop, so in principle there should be no problems, or?

Thanks for your help and greetings from Spain,

Miguel

[mhermanns]

Wow, ok, I have some interesting news about this issue.

I was waiting to have access to a computer with an Intel Fortran Compiler 9.1 installed on it to see if my OpenMP problem was solved in the new version of the compiler, or it was a problem in my code. The results are as follows.

When I compile my code using the Intel Fortran Compiler 9.1.032 and without the -openmp flag, then everything works without problems. BUT if I add the -openmp flag, then I get an internal compiler error while processing the file with the above mentioned loop (is anyway the only one with OpenMP clauses):

LOC_type is: 1
101000_1

fortcom: Severe: **Internal compiler error: internal abort** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error.

compilation aborted for Finite_Difference_Formulation/Finite_Difference_Formulation_2D.f90 (code 3)

I hope this helps while I wait to be able to make more tests and to write a bug report at the end of August (when I will have access to the 9.1 version of the compiler and to the corresponding premier support service).

Greetings from Spain,

Miguel

[Steven_L_Intel1]

The message doesn't really help - we'd need to see a test case.

[mhermanns]

Ok,

I have taken my original files, merged them and started to eliminate parts of the code which do not "eliminate" the error message. At the end I have the following quite short example, which of course has no practical use :-) :

---

!-----------------------------------------------------------------------!
module Finite_Difference_Formulation_2D
implicit none

type Structured_Grid_2D

real, allocatable :: Grid_Nodes(:,:,:), Second_Metric(:,:,:,:,:)

end type Structured_Grid_2D

contains
!-----------------------------------------------------------------------!
subroutine Finite_Difference_Method_2D(Grid)
type(Structured_Grid_2D), intent(inout) :: Grid

integer :: Number_Nodes_x, Number_Nodes_y, j
real :: Grid_Derivatives(2,0:ubound(Grid%Grid_Nodes,2),0:5)

Number_Nodes_x = ubound(Grid%Grid_Nodes, 2)
Number_Nodes_y = ubound(Grid%Grid_Nodes, 3)

allocate(Grid%Second_Metric(2,2,2,0:Number_Nodes_x,0:Number_Nodes_y))

!$OMP PARALLEL DO SCHEDULE(STATIC) DEFAULT(PRIVATE) &
!$OMP SHARED(Number_Nodes_y, Grid)
do j = 0, Number_Nodes_y

Grid_Derivatives(:,:,0) = Grid%Grid_Nodes(:,:,j)

enddo
!$OMP END PARALLEL DO

end subroutine Finite_Difference_Method_2D
!-----------------------------------------------------------------------!
end module Finite_Difference_Formulation_2D

---

The command I have used to compile the above example (stored in a file called like the module) is:

ifort -xN -openmp Finite_Difference_Formulation_2D.f90 -c -o output.o

I have seen that not using the switch -xN leads to a success in the compilation process.

Hope this helps more :-). Greetings from Spain,

Miguel

[mhermanns]

Hello again to everybody,

I have downloaded and installed the latest version of the fortran compiler (9.1.036) and have tried to compile the above test case. Now the compilation is succesfull, which is a good sign :-)

Regarding the original problem in my code it is still not working. I will work on it and let you know about the results.

Greetings from Spain,

Miguel

[jimdempseyatthecove]

I think the problem is that Grid_Derivatives is not declared as being AUTOMATIC. Use:

real, automatic:: Grid_Derivatives(2,0:ubound(Grid%Grid_Nodes,2),0:5)

Without "automatic" Grid_Derivatives will likely be created as a static object in the name space of your subroutine (Finite_Difference_Method_2D). As such (without automatic) all threads will overwrite the data each thinks is private. This same thing happens when you create small vectors

real :: vector(3)

Can cause problems in OpenMP. Use

real, automatic:: vector(3)

Jim Dempsey

[mhermanns]

Hi Jim,

Thanks for the hint. I have tried it out on my code by setting all local variables as automatic, but the problem still persists :-(

If your hint would have worked, then I would think that the compiler has somewhere a bug, because having specified the clause DEFAULT(PRIVATE) I don't see why I need to do something else. I would expect the compiler to do what it has to do in order to ensure the private nature of the variables, or?

Thanks for your help and greetings from Spain,

Miguel

[mhermanns]

Here I am again,

I have managed to reduce my code to 300 LOC while still preserving a minimal functionalty and still showing the bug. I have submitted a bug report through the premier support service with the resulting example code, as I do not see any obvious violation of the OpenMP standard and the Fortran 95+ standards.

The issue ID is 391012

Hope the supplied information is useful :-)

Thanks again and greetings from Spain,

Miguel