Using !$omp declare simd in an elemental subroutine

[eos_pengwern]

Quite recently (in May), I decided to update some of my codebase to use OpenMP 4.0 and, in particular, try out the "!$omp declare simd" directive. After a few teething troubles documented in a question I asked here, I got it working well and it has been running smoothly ever since. As it's such a short subroutine, I'll reproduce it here for convenience:

elemental subroutine Calculate_Sines(normal_x, normal_y, normal_z, &
                                     rzero_x, rzero_y, rzero_z,    &
                                     es_x, es_y, es_z, sin_theta)
 
!$omp declare simd(Calculate_Sines) uniform(normal_x, normal_y, normal_z)
    real(kind(1d0)), intent(in) :: normal_x, normal_y, normal_z
    real(kind(1d0)), intent(in) :: rzero_x, rzero_y, rzero_z
    real(kind(1d0)), intent(out) :: es_x, es_y, es_z, sin_theta
       
    es_x = normal_z * rzero_y - normal_y * rzero_z
    es_y = normal_x * rzero_z - normal_z * rzero_x
    es_z = normal_y * rzero_x - normal_x * rzero_y       
     
    sin_theta = sqrt( es_x**2 + es_y**2 + es_z**2 )
 
end subroutine Calculate_Sines

However, I've just installed Parallel Studio XE 16.0 and everything has come to a juddering halt. The same piece of code now fails to compile, with the error message:

error #8807: An OpenMP* directive may not appear in a PURE procedure.

I assume that an elemental subroutine is pure by implication. My basic syntax isn't at all original, as it very closely follows an example given in the Intel documentation here, so I can't really figure out what I should do. Is the example in the documentation, which dates from May 2014 no longer valid? Or am I just doing something really dumb?

[jimdempseyatthecove]

Look in the documentation under VECTOR | ATTRIBUTES VECTOR

Consider using

!DIR$ ATTRIBUTES VECTOR : uniform(normal_x, normal_y, normal_z)  :: Calculate_Sines

Read the documentation, I havn't tried this in your code example.

Jim Dempsey

[eos_pengwern]

Thank you Jim. That does work, but it's not standard Fortran 2003 any more. That seems like a step backward.

So, either:

-   this is genuinely non-conforming code, and the Intel compiler is wrong to allow it with a non-standard directive (not to mention the Intel documentation being wrong to recommend it).

or

-  this is in fact standard-conforming code and the fact that the new Fortran compiler won't compile it, when the old Fortran compiler did, is a new regression bug in the 16.0 compiler.

I think I need to wait for someone from Intel to reply, hopefully on Monday.

[Kevin_D_Intel · Accepted solution]

I believe this is coming from improved/corrected OpenMP 4.0 support in 16.0. According to the OpenMP 4.0.0 spec, chapter 2 Directives, under Restrictions (pg. 26 paragraph 3) it says:  "OpenMP directives may not appear in PURE or ELEMENTAL procedures."

In our 16.0 release we added support for Fortran 2008 IMPURE ELEMENTAL. If you declare the routine with IMPURE it will compile.

[eos_pengwern]

Thank you Kevin, that nails it!

[Kevin_D_Intel]

Glad to hear that.

Just a few tidbits from following up on the earlier comments about documentation.

You wrote earlier “this is genuinely non-conforming code, and the Intel compiler is wrong to allow it with a non-standard directive (not to mention the Intel documentation being wrong to recommend it).” and I want to ensure we correct the documentation. I read this as saying we might be suggesting using !omp$ simd within an ELEMENTAL. Can you point me to what was not clear in the cited article or documentation so we make sure to review/revise that?

With IMPURE, your test case also compiles with -qopenmp-simd (a suggested option in the earlier cited article) which is a nice validation of that option for this small routine. In some extended notes about the new support for IMPURE ELEMENTAL, there is a caution that the Vectorization quality could be impacted with IMPURE as the compiler may not be able to optimize with the routine now having side-effects. I checked your routine and find it still vectorizes with IMPURE with the 16.0 compiler.

I posted a reply to your earlier post to redirect any reader to this updated discussion.

[eos_pengwern]

Thank you Kevin.

The relevant part of the cited article (for the avoidance of doubt, this one: https://software.intel.com/en-us/articles/explicit-vector-programming-in-fortran) occurs almost precisely half way through the article, in the section entitled "Calling SIMD-enabled procedures with array arguments". In there, the example given shows an elemental function with !dir$ attributes vector declared,  with a comment underneath saying that !$omp declare simd could be used instead.

I appreciate your help with this.

[Kevin_D_Intel]

Ok, thank you. I saw that and checked the syntax before replying earlier and it did not trigger an error w/16.0 so I'll follow-up with Development and the article author too.

[jimdempseyatthecove]

eos pengwern wrote:

Thank you Jim. That does work, but it's not standard Fortran 2003 any more. That seems like a step backward.

However note that, with respect to Keven's non-pure elemental response, in order to get vectoization of that subroutine you must then compile with -openmp enabled. That routine should be declarable as vectorizable with or without OpenMP enabled. The !DIR$ ATTRIBUTES does this.

Note 2: Technically !$OMP... is not Fortran 2003 either.

Jim Dempsey

[Kevin_D_Intel]

An update. Following additional discussion with Development we now understand the 16.0 compiler’s current handling and issuing error #8807 as shown in the original post will be changed in a future release, and the acceptance within IMPURE ELEMENTAL was by accident.

We learned it is the intent to permit use of !$OMP DECLARE SIMD and !$OMP SIMD in ELEMENTAL subprograms, including PURE and IMPURE ELEMENTAL; however, ratification within the OpenMP 4.1 specification is required first and then the compiler can be changed to permit these uses.

I opened a feature enhancement request to enable tracking the changes to the compiler and updating this forum thread when the support is available in a future release.

(Internal tracking id: DPD200376030)

[eos_pengwern]

Thank you Kevin; I appreciate it; I'll stick with the impure elemental declaration for now, but I'll put a comment in my code to the effect that I should try dropping the 'impure' when I upgrade my compiler.

[jimdempseyatthecove]

Keven,

How does your compiler team suggest addressing this issue when the program is compiled with OpenMP disabled?

Getting good SIMD code should not be dependent on OpenMP (when OpenMP is not used).

IMHO !$OMP SIMD should only affect how the iteration space is partitioned (i.e. into vector size multiples).

Jim Dempsey