Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Black Belt
351 Views

Possible bug in CPU dispatch code

One of my users is running into a problem with the Intel 18.0.2 compiler and multiple-target dispatch.

The code was compiled with "-O2 -xCORE-AVX2 -axCORE-AVX512,MIC-AVX512".  When run on either SKX or KNL, the version with multiple-target dispatch incorrectly processes a ternary assignment of the form "a += (a > b ? -1 : +1)*b" (where "a" is an "int" variable and "b" is a constant defined by a macro.  When the initial value of "a" is "b+1", the code incorrect updates "a" to "2*b+1" instead of to "1".

Compiling with either "-xCORE-AVX512" or "-xMIC-AVX512" results in the correct behavior.

I have not been able to reproduce this in isolation, but the incorrect result is clear in the output of the user code.

Does this look like a known issue?

 

0 Kudos
15 Replies
Highlighted
Moderator
336 Views

Hi John,


We have tried to reproduce using the latest version(19.1)of ICC. 

We got the correct result.

We have tried with 18.0.2 also and got the correct result.

Could you ask him to upgrade to the latest version and see if the error persists?

Do let us know.


Regards

Prasanth


0 Kudos
Highlighted
331 Views

John,

Do you get the error, on the specific machine, when you singly target the architecture?

Also, I thought MIC-AVX512 is KNC not KNL

Jim Dempsey

0 Kudos
Highlighted
Black Belt
327 Views

Sorry I was not clear -- the same error occurs when running on either SKX or KNL using the multi-target binary, and does not occur on either system with the appropriate directly-targeted binary.

The error occurs in the midst of a larger code, where it is used to implement periodic boundary conditions in three dimensions.  When the error occurs, the computation is incorrect in two of the three indices.

"MIC-AVX512" is KNL -- KNC is no longer supported as a target.

0 Kudos
Highlighted
Black Belt
326 Views

I have asked the user to try different compiler versions -- still waiting on a response.

The workaround is trivial (keep two executables and use the correct one!), but this is the first bug I have seen in the multi-dispatch code and was mostly curious about whether it looked familiar.

The specific value for which this case fails is "b=524288" and "a=b+1".  The correct interpretation of the ternary is a+=(-b), so the initial value of "a=b+1" turns into "a=1".  With the directly target binaries, the code delivers this result for all three indices.  With the multi-target binary, the first two indices are incorrectly updated as a+=b (the other branch of the ternary), so the initial value of "a=b+1" turns into "a=2*b+1".   This specific behavior makes it seem less likely that the trouble is with an out-of-bounds memory reference or the like, and the occurrence with the initial value of 2^19 seems like it might be a clue....

I will run "objdump -d" on the three executables and see if I can find the exact spot where this ternary is evaluated.  Probably not, but sometimes I get lucky....

0 Kudos
Highlighted
Moderator
313 Views

We've addressed a number of issues wrt AVX-512 in later compiler versions, but I couldn't find your specific case. It's good that the User is willing to try the newer version. Please let us know how that goes.

Thanks,

Viet


0 Kudos
Highlighted
304 Views

>>and does not occur on either system with the appropriate directly-targeted binary.

Sorry I wasn't clear too. Seeing that each of the non-desired instruction sets should function (at least for the conditional move), my suggestion was to see if by eliminating the multi-target (tests) you observe the same/similar incorrect results. This would be an indicator as to if the error is in (caused by/side effect of) the dispatcher .OR. the "correct" target code for the designated architecture.

While this doesn't help you directly, it should aid the Intel support programmers in where to look.

>>does not occur on either system with the appropriate directly-targeted binary

This leaves you the opportunity to use your own dispatcher to call one of 2, 3, 4 routines, each compiled with a single targeted architecture. While the code will be larger, the single executable should run just as fast.

If the application is C++, you could use a separate namespace for each architecture.

For Fortran, you may have to name mangle (place suffix on procedure name).

Jim Dempsey

 

0 Kudos
Highlighted
Black Belt
272 Views

Unfortunately the user is not able to switch to the Intel 19 compilers at this time.

I have made considerable progress in isolating the incorrect code, but don't yet have a standalone reproducer.  

I was surprised at how different the generated code was for the various compiler options.  Considering four cases that I have looked at in detail (all compiled with "-O2"):

  1. Compiled for KNL only
  2. Compiled for SKX only
  3. Original multi-target: -xCORE-AVX2 -axCORE-AVX512,MIC-AVX512
  4. New multi-target: -xCORE-AVX2 -axCOMMON-AVX512

 

For this code (embedded in a larger routine, which is embedded in a larger application):

// LEN is a constant value of 524288 (0x80000)
// q is an int32 vector with 3 elements
for (u_char dim = 0; dim < 3; ++dim)
   q[dim] += (q[dim] > LEN ? -1 : +1)*LEN;

Examination of the output of "objdump -d" shows that

  • Versions (1) and (2) are implemented with purely scalar code.  
    • For each of the thee dimensions,  compiler places the values "LEN" and "-LEN" in two GPRs, compares q[dim] with LEN, uses a conditional move to define an update value, adds the selected update to the original, and stores the result.
  • Versions (3) and (4) are implemented with a combination of vector and scalar code.
    • The first two dimensions are combined into a single piece of vector code.
    • The third dimension is implemented with (effectively) the same scalar code that was used in variants (1) and (2).
    • The vector code used by (3) and (4) is similar (but only (4) is correct).
  • Versions (1) (2) and (4) give the correct results for all three elements.  
  • Version (3) gives the correct result only for the third dimension (computed using scalar arithmetic), and gives the same incorrect result for the first two dimensions.  
    • The incorrect result is the same as one would expect to see if the wrong branch of the ternary was chosen.

In the incorrect version of the vector code, it looks like the compiler "forgot" to load the data into an xmm register before starting the computations!

Extracting just the vector code from (3) and (4):

Correct version Incorrect version Comments
vmovq  (%rsp),%xmm3   64-bit load = 1st 2 32-bit elements
vpcmpgtd 0x11cf73(%rip),%xmm3,%xmm1   seems important?
vpandn 0x11cf7b(%rip),%xmm1,%xmm0 vpandn 0x12ffe8(%rip),%xmm1,%xmm0 undefined xmm1 in wrong version
vpor   %xmm1,%xmm0,%xmm2 vpor   %xmm1,%xmm0,%xmm2  
vpslld $0x13,%xmm2,%xmm4 vpslld $0x13,%xmm2,%xmm4  
  vmovq  (%rsp),%xmm3 aha, now we load the data
vpaddd %xmm4,%xmm3,%xmm5 vpaddd %xmm4,%xmm3,%xmm5  
vmovq  %xmm5,(%rsp) vmovq  %xmm5,(%rsp)  

 

Could the value in %xmm1 have been hoisted upstream?  Certainly possible, but there are no other SIMD register references in this function, or in the two functions that call this function.  There are uses of xmm1 starting at two levels up the calling tree, but in all the cases I reviewed those were floating-point computations, while this is an integer calculation.

Perhaps these more detailed characteristics will make it clear whether this problem is related to any known bugs....

0 Kudos
Highlighted
Moderator
264 Views

Hi "Dr. Bandwidth",


I guess you were able to reproduce this with 18.0 compiler and wonder if you have a chance to try the 19.1 version?


Thanks,

Viet




0 Kudos
Highlighted
262 Views

While this bug should be fixed, does re-expressing the code produce the correct result:

// LEN is a constant value of 524288 (0x80000)
// q is an int32 vector with 3 elements
for (u_char dim = 0; dim < 3; ++dim)
   q[dim] += q[dim] > LEN ? -LEN : LEN;

 

Also, if q is a triplet, within an array of triplets, why then is the code not taking advantage of the AVX512 by treating the array of triplets as an array of scalars? IOW use all 16 lanes of AVX512 instead of 2 lanes + 1 lane.

Jim Dempsey

0 Kudos
Highlighted
Black Belt
257 Views

This bug was found with 18u2 (our default on the TACC Stampede2 system.

The error occurs when compiling the 3-target binary with either -O2 or -O3, but not with -O0 or -O1.  There is no indication that the compiler generated multiple versions of the routine, so it is not surprising that the error occurs when running on either KNL or SKX.

With the 2-target binary (-xCORE-AVX2 -axCOMMON-AVX512) the code is correct at all four optimization levels. (Again, there is no indication that the compiler created multiple versions of the code.)

The KNL and SKX binaries generated purely scalar code here, while the 3-target and 2-target versions generated a combination of vector and scalar code.  All of the SIMD instructions generated in both cases were AVX instructions (which are obviously supported by the "base" architecture of all four compilation targets).  Weird.....

The application includes a bunch of infrastructure (including PetSC, Hypre, and p4est), so going to Intel 19 is a risk. (18u2 is also the latest compiler that is fully integrated into our module system and infrastructure on Stampede2.) The user reports having trouble with memory leaks with PetSC and Intel 19 in another application, making them a bit cautious.  

0 Kudos
Highlighted
Moderator
251 Views

The reason I asked to try with 19.1 is that I dont think we will release any more 18.0 update.

Is it possible to do a quick check by just compile that 1 file with 19.1?


Thanks,

Viet


0 Kudos
Highlighted
Black Belt
226 Views

It took a while to work around all the dependencies, but from examination of the assembly listing it is clear that the bug exists in Intel 18.0.2 and Intel 18.0.3, but is fixed in 18.0.4.  The bug remains fixed in Intel 19.0.0, 19.0.1, 19.0.2, 19.0.3, and 19.0.5.

We have lots of workaround options, so I think we are done with this issue.   

I remain curious about whether it "looks like" any known fixes in 18.0.4

0 Kudos
Highlighted
Black Belt
217 Views

More bounding details....

For this code, the error requires all of the following:

  1. Intel compiler revision (18.0.2 OR 18.0.3), AND
  2. Optimization level ("-O2" OR "-O3"), AND
  3. Base target architecture of ("-xAVX" OR "-xCORE-AVX2" OR "-xCOMMON-AVX512"), AND
  4. One or more optional target architectures that include "-axCORE-AVX512"

All of these cases show exactly the behavior described above -- the compiler somehow "forgets" that it needs to load the data and perform the compare operation before starting to manipulate the mask resulting from the compare.

0 Kudos
Highlighted
Moderator
212 Views

Thanks for your investigation and verification. It looks like a known issue and has been fixed in the later versions. I just looked into our bug database again, but haven't been able to pinpoint to your case.


0 Kudos
Highlighted
Moderator
189 Views

Hi John,

Since your issue has been resolved we are closing this thread.

If you require additional assistance please start a new thread. 

Regards

Prasanth


0 Kudos