topic Sorry for the late reply, in Intel® ISA Extensions

Compiler decides to use SSE instead of AVX

alzamos — Thu, 16 Aug 2012 17:06:14 GMT

Hello!

I'm compiling some code in fortran with the -xAVX option (ifort version 12.1.0 20111011), and depending on whether or not the code is standalone or part of a subroutine, the compiler vectorizes differently and I was wondering why.The code is as follows:

[fortran]program example

implicit none

double precision, dimension(4, 180000, 4) :: qold, q, res

double precision, dimension(4, 180000) :: adt

double precision, dimension(4) :: adti, del

integer :: i, j, diff, acc

diff = 180000

do j = 1, diff

adti(:) = 1.0 / adt(:,j) # line 10

do i = 1, 4

del(:) = adti(:) * res(:,j,i) # line 12

q(:,j,i) = qold(:,j,i) - del(:) # line 13

res(:,j,i) = 0.0 # line 14

end do

acc = sum(q)

print *, acc

end program [/fortran]

Using -vec-report it says that the loops were indeed vectorized.Furthermore, looking at the assembly, all the relevant instructions were using the ymm registers.However, if I were to encompass the code within a subroutine which takes qold, q, res, adt and diff as arguments and include it in a module, things go differently;This time -vec-report tells me that only lines 10 and 12 could be vectorized, whereas 13 and 14 the vectorization is "possible but seems inefficient".If I decide to force the vectorization to happen using the !DIR$ SIMD directive before lines 13 and 14, it then tells me that the SIMD loop was vectorized.However, looking at the assembly, instead of using AVX instructions it seems to be using SSE instructions.

For line 13,
- vsubpd generated_module_mp_update_kernel_caller_$DEL.0.2(%rip), %xmm6, %xmm7
- vsubpd 16+generated_module_mp_update_kernel_caller_$DEL.0.2(%rip), %xmm8, %xmm9
For line 14,
- vxorpd %xmm0, %xmm0, %xmm0
- vmovupd %xmm0, (%rbx,%r9)
- vmovupd %xmm0, 16(%rbx,%r9)

Because it does two similar instructions for both, with an offset of 16 bytes (128 bits) which is the size of an SSE register, it seems to be doing two SSE vector instructions instead of one AVX instruction.
Does anyone know why this happens and how to change this?

Kind regards,

Alex

Compiler decides to use SSE instead of AVX

TimP — Sat, 18 Aug 2012 23:58:42 GMT

Those are AVX-128 instructions. With AVX-128, the compiler can take advantage of hardware handling misaligned data well enough that there is no need to peel iterations for alignment. This strategy may be evaluated as superior if there isn't sufficient information for the compiler to make other than a default guess about loop count or alignment, as a very long loop may be required to establish a preference for AVX-256 with peeling. If you are interested, you could try loop count and alignment directives to see if they influence a choice between AVX-128 and AVX-256.
If you are interested in further esoterica, you might try to understand why CEAN notation in icc uses AVX-128 more frequently (if that happens in your cases).
Perhaps needless to say, gnu compilers frequently prefer AVX-128 in order to avoid extra code to deal with misalignment.

Compiler decides to use SSE instead of AVX

Martyn_C_Intel — Tue, 21 Aug 2012 22:52:25 GMT

Hi Alex,
            What the compiler does depends on the information available to it, which in turn depends on how the arguments are passed, whether caller and callee are in the same source file, etc. The compiler has to make conservative choices (wrt performance as well as correctness) if it doesn't have full information.
            You haven't given us all the details, please could you include the source version with the module subroutine that does not vectorize? That will allow us to figure out whether the compiler could do better, or whether there's a more effective way to write the code. I think it will be more useful than me constructing an example that doesn't match yours.

Regards,
Martyn

Sorry for the late reply,

alzamos — Mon, 17 Sep 2012 14:47:49 GMT

Sorry for the late reply,

Thank you for both of your answers,

@TimP: Thanks for pointing me to the loop count and alignment directives, I'll take a look at them now and let you know if they work! @Martyn: I'm basically fiddling with some very ugly generated code produced by a source to source compiler to see how to change it in such a way that ifort will vectorize the instructions, below is the subroutine in question and the subroutine which calls the 'problematic' subroutine (both within the same source file): (BS is a macro defined at the beginning of the file, usually set to 4) [fortran] subroutine update_kernel_caller(qold, q, res, adt, rms, diff) implicit none integer :: diff real(kind=8), dimension(BS,diff/BS,4) :: qold, q, res real(kind=8), dimension(BS,*) :: adt real(kind=8), dimension(1) :: rms real(kind=8), dimension(BS) :: adti, del real(kind=8), dimension(BS) :: acc integer :: i,j do j=1,diff/BS adti(:) = 1.0 / adt(:,j) do i=1,4 del(:) = adti(:) * res(:,j,i) q(:,j,i) = qold(:,j,i) - del(:) res(:,j,i) = 0.0 acc(:) = del(:) * del(:) rms(1) = rms(1) + sum(acc) end do end do end subroutine [/fortran] This is called by : (The opDats originally contain data in AoS form, the mkl method turns it into SoA and stores the result into the topDats, the system clocks are just for benchmarking purposes) [fortran] subroutine update_kernel(opDat1,opDat2,opDat3,opDat4,opDat5,sliceStart,sliceEnd) implicit none real(kind=8), dimension(0:*) :: opDat1 real(kind=8), dimension(0:*) :: opDat2 real(kind=8), dimension(0:*) :: opDat3 real(kind=8), dimension(0:*) :: opDat4 real(kind=8), dimension(0:*) :: opDat5 integer(kind=4) :: sliceStart integer(kind=4) :: sliceEnd integer(kind=4) :: i1, diff integer(8) :: time1, time2, count_rate, count_max real(kind=8), allocatable, dimension(:) :: topDat1, topDat2, topDat3 diff = sliceEnd - sliceStart call system_clock(time1, count_rate, count_max) allocate (topDat1(diff*4)) allocate (topDat2(diff*4)) allocate (topDat3(diff*4)) call mkl_domatcopy('r', 't', diff, 4, 1.d0, opDat1(4*sliceStart:4*sliceEnd-1), 4, topDat1, diff) call mkl_domatcopy('r', 't', diff, 4, 1.d0, opDat2(4*sliceStart:4*sliceEnd-1), 4, topDat2, diff) call mkl_domatcopy('r', 't', diff, 4, 1.d0, opDat3(4*sliceStart:4*sliceEnd-1), 4, topDat3, diff) call system_clock(time2, count_rate, count_max) print *, "### not counted", time2 - time1 call update_kernel_caller(topDat1, topDat2, topDat3, opDat4(sliceStart:sliceStart+diff-1), opDat5, diff) call system_clock(time1, count_rate, count_max) call mkl_domatcopy('r', 't', 4, diff, 1.d0, topDat1, diff, opDat1(4*sliceStart:4*sliceEnd-1), 4) call mkl_domatcopy('r', 't', 4, diff, 1.d0, topDat2, diff, opDat2(4*sliceStart:4*sliceEnd-1), 4) call mkl_domatcopy('r', 't', 4, diff, 1.d0, topDat3, diff, opDat3(4*sliceStart:4*sliceEnd-1), 4) deallocate (topDat1) deallocate (topDat2) deallocate (topDat3) call system_clock(time2, count_rate, count_max) print *, "### not counted", time2 - time1 end subroutine [/fortran] I apologize for the horrible spacing... in the forum editor it appears nicely, but somehow it becomes double spaced when posting... I'll see if I can do something about it... Regards, Alex