Different inter-core latency measured on two identical Skylake Xeon Gold 6154 systems

AATLI3 · ‎08-28-2019

We have been using two same Skylake servers with completely the same softwares, Centos 7 OS and BIOS settings. Everything is the same, except the latency performance. Our software is using AVX512.

In tests, I noticed that something slows down performance (increasing latency) in the one of the systems each time. There is a significant performance difference. I checked everything, all are the same. In the attached code, I'm trying to measure inter-cpu latency and even this shows me a proof of slowness, but for the real code, I'm using SIMD/AVX512 for math operations as well and in the field working application, there is an incredible difference between the same systems. How can this happen?

What should I do to solve this problem? Which tool can help?

Thanks in advance..

sudo lshw -class cpu
  *-cpu:0                   
       description: CPU
       product: Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz
       vendor: Intel Corp.
       vendor_id: GenuineIntel
       physical id: 400
       bus info: cpu@0
       version: Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz
       slot: CPU1
       size: 3GHz
       capacity: 4GHz
       width: 64 bits
       clock: 1010MHz
       capabilities: lm fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear spec_ctrl intel_stibp flush_l1d
       configuration: cores=18 enabledcores=18 threads=18
  *-cpu:1 DISABLED
       description: CPU [empty]
       physical id: 401
       slot: CPU2

I also attached the following sample code as an example to measure latency for different mesh network paths on the same die, different cpus have different inter-cpu latency. But still one of the same systems is 25% slower than the other in all case.

Also I don't know if it affects, but the slow CPU has extra md_clear CPU feature flag.

Both system's output as follows: (2-17 are isolated cpus, and the attached code ignores the first iteration for cold cache, thread startup, cache miss etc..)

            slow cpu    fast cpu
------------------------------------                

(M=2-W=3)   mean=580    mean=374
(M=2-W=4)   mean=463    mean=365
(M=2-W=5)   mean=449    mean=391
(M=2-W=6)   mean=484    mean=345
(M=2-W=7)   mean=430    mean=386
(M=2-W=8)   mean=439    mean=369
(M=2-W=9)   mean=445    mean=376
(M=2-W=10)  mean=480    mean=354
(M=2-W=11)  mean=440    mean=392
(M=2-W=12)  mean=475    mean=324
(M=2-W=13)  mean=453    mean=373
(M=2-W=14)  mean=474    mean=344
(M=2-W=15)  mean=445    mean=384
(M=2-W=16)  mean=468    mean=372
(M=2-W=17)  mean=462    mean=373
(M=3-W=2)   mean=447    mean=392
(M=3-W=4)   mean=556    mean=386
(M=3-W=5)   mean=418    mean=409
(M=3-W=6)   mean=473    mean=372
(M=3-W=7)   mean=397    mean=400
(M=3-W=8)   mean=408    mean=403
(M=3-W=9)   mean=412    mean=413
(M=3-W=10)  mean=447    mean=389
(M=3-W=11)  mean=412    mean=423
(M=3-W=12)  mean=446    mean=399
(M=3-W=13)  mean=427    mean=407
(M=3-W=14)  mean=445    mean=390
(M=3-W=15)  mean=417    mean=448
(M=3-W=16)  mean=438    mean=386
(M=3-W=17)  mean=435    mean=396
(M=4-W=2)   mean=463    mean=368
(M=4-W=3)   mean=433    mean=401
(M=4-W=5)   mean=561    mean=406
(M=4-W=6)   mean=468    mean=378
(M=4-W=7)   mean=416    mean=387
(M=4-W=8)   mean=425    mean=386
(M=4-W=9)   mean=425    mean=415
(M=4-W=10)  mean=464    mean=379
(M=4-W=11)  mean=424    mean=404
(M=4-W=12)  mean=456    mean=369
(M=4-W=13)  mean=441    mean=395
(M=4-W=14)  mean=460    mean=378
(M=4-W=15)  mean=427    mean=405
(M=4-W=16)  mean=446    mean=369
(M=4-W=17)  mean=448    mean=391
(M=5-W=2)   mean=447    mean=382
(M=5-W=3)   mean=418    mean=406
(M=5-W=4)   mean=430    mean=397
(M=5-W=6)   mean=584    mean=386
(M=5-W=7)   mean=399    mean=399
(M=5-W=8)   mean=404    mean=386
(M=5-W=9)   mean=408    mean=408
(M=5-W=10)  mean=446    mean=378
(M=5-W=11)  mean=411    mean=407
(M=5-W=12)  mean=440    mean=385
(M=5-W=13)  mean=424    mean=402
(M=5-W=14)  mean=442    mean=381
(M=5-W=15)  mean=411    mean=411
(M=5-W=16)  mean=433    mean=398
(M=5-W=17)  mean=429    mean=395
(M=6-W=2)   mean=486    mean=356
(M=6-W=3)   mean=453    mean=388
(M=6-W=4)   mean=471    mean=353
(M=6-W=5)   mean=452    mean=388
(M=6-W=7)   mean=570    mean=360
(M=6-W=8)   mean=444    mean=377
(M=6-W=9)   mean=450    mean=376
(M=6-W=10)  mean=485    mean=335
(M=6-W=11)  mean=451    mean=410
(M=6-W=12)  mean=479    mean=353
(M=6-W=13)  mean=463    mean=363
(M=6-W=14)  mean=479    mean=359
(M=6-W=15)  mean=450    mean=394
(M=6-W=16)  mean=473    mean=364
(M=6-W=17)  mean=469    mean=373
(M=7-W=2)   mean=454    mean=365
(M=7-W=3)   mean=418    mean=410
(M=7-W=4)   mean=443    mean=370
(M=7-W=5)   mean=421    mean=407
(M=7-W=6)   mean=456    mean=363
(M=7-W=8)   mean=527    mean=380
(M=7-W=9)   mean=417    mean=392
(M=7-W=10)  mean=460    mean=361
(M=7-W=11)  mean=421    mean=402
(M=7-W=12)  mean=447    mean=354
(M=7-W=13)  mean=430    mean=381
(M=7-W=14)  mean=449    mean=375
(M=7-W=15)  mean=420    mean=393
(M=7-W=16)  mean=442    mean=352
(M=7-W=17)  mean=438    mean=367
(M=8-W=2)   mean=463    mean=382
(M=8-W=3)   mean=434    mean=411
(M=8-W=4)   mean=452    mean=372
(M=8-W=5)   mean=429    mean=402
(M=8-W=6)   mean=469    mean=368
(M=8-W=7)   mean=416    mean=418
(M=8-W=9)   mean=560    mean=418
(M=8-W=10)  mean=468    mean=385
(M=8-W=11)  mean=429    mean=394
(M=8-W=12)  mean=460    mean=378
(M=8-W=13)  mean=439    mean=392
(M=8-W=14)  mean=459    mean=373
(M=8-W=15)  mean=429    mean=383
(M=8-W=16)  mean=452    mean=376
(M=8-W=17)  mean=449    mean=401
(M=9-W=2)   mean=440    mean=368
(M=9-W=3)   mean=410    mean=398
(M=9-W=4)   mean=426    mean=385
(M=9-W=5)   mean=406    mean=403
(M=9-W=6)   mean=447    mean=378
(M=9-W=7)   mean=393    mean=427
(M=9-W=8)   mean=408    mean=368
(M=9-W=10)  mean=580    mean=392
(M=9-W=11)  mean=408    mean=387
(M=9-W=12)  mean=433    mean=381
(M=9-W=13)  mean=418    mean=444
(M=9-W=14)  mean=441    mean=407
(M=9-W=15)  mean=408    mean=401
(M=9-W=16)  mean=427    mean=376
(M=9-W=17)  mean=426    mean=383
(M=10-W=2)  mean=478    mean=361
(M=10-W=3)  mean=446    mean=379
(M=10-W=4)  mean=461    mean=350
(M=10-W=5)  mean=445    mean=373
(M=10-W=6)  mean=483    mean=354
(M=10-W=7)  mean=428    mean=370
(M=10-W=8)  mean=436    mean=355
(M=10-W=9)  mean=448    mean=390
(M=10-W=11) mean=569    mean=350
(M=10-W=12) mean=473    mean=337
(M=10-W=13) mean=454    mean=370
(M=10-W=14) mean=474    mean=360
(M=10-W=15) mean=441    mean=370
(M=10-W=16) mean=463    mean=354
(M=10-W=17) mean=462    mean=358
(M=11-W=2)  mean=447    mean=384
(M=11-W=3)  mean=411    mean=408
(M=11-W=4)  mean=433    mean=394
(M=11-W=5)  mean=413    mean=428
(M=11-W=6)  mean=455    mean=383
(M=11-W=7)  mean=402    mean=395
(M=11-W=8)  mean=407    mean=418
(M=11-W=9)  mean=417    mean=424
(M=11-W=10) mean=452    mean=395
(M=11-W=12) mean=577    mean=406
(M=11-W=13) mean=426    mean=402
(M=11-W=14) mean=442    mean=412
(M=11-W=15) mean=408    mean=411
(M=11-W=16) mean=435    mean=400
(M=11-W=17) mean=431    mean=415
(M=12-W=2)  mean=473    mean=352
(M=12-W=3)  mean=447    mean=381
(M=12-W=4)  mean=461    mean=361
(M=12-W=5)  mean=445    mean=366
(M=12-W=6)  mean=483    mean=322
(M=12-W=7)  mean=431    mean=358
(M=12-W=8)  mean=438    mean=340
(M=12-W=9)  mean=448    mean=409
(M=12-W=10) mean=481    mean=334
(M=12-W=11) mean=447    mean=351
(M=12-W=13) mean=580    mean=383
(M=12-W=14) mean=473    mean=359
(M=12-W=15) mean=441    mean=385
(M=12-W=16) mean=463    mean=355
(M=12-W=17) mean=462    mean=358
(M=13-W=2)  mean=450    mean=385
(M=13-W=3)  mean=420    mean=410
(M=13-W=4)  mean=440    mean=396
(M=13-W=5)  mean=418    mean=402
(M=13-W=6)  mean=461    mean=385
(M=13-W=7)  mean=406    mean=391
(M=13-W=8)  mean=415    mean=382
(M=13-W=9)  mean=421    mean=402
(M=13-W=10) mean=457    mean=376
(M=13-W=11) mean=422    mean=409
(M=13-W=12) mean=451    mean=381
(M=13-W=14) mean=579    mean=375
(M=13-W=15) mean=430    mean=402
(M=13-W=16) mean=440    mean=408
(M=13-W=17) mean=439    mean=394
(M=14-W=2)  mean=477    mean=330
(M=14-W=3)  mean=449    mean=406
(M=14-W=4)  mean=464    mean=355
(M=14-W=5)  mean=450    mean=389
(M=14-W=6)  mean=487    mean=342
(M=14-W=7)  mean=432    mean=380
(M=14-W=8)  mean=439    mean=360
(M=14-W=9)  mean=451    mean=405
(M=14-W=10) mean=485    mean=356
(M=14-W=11) mean=447    mean=398
(M=14-W=12) mean=479    mean=338
(M=14-W=13) mean=455    mean=382
(M=14-W=15) mean=564    mean=383
(M=14-W=16) mean=481    mean=361
(M=14-W=17) mean=465    mean=351
(M=15-W=2)  mean=426    mean=409
(M=15-W=3)  mean=395    mean=424
(M=15-W=4)  mean=412    mean=427
(M=15-W=5)  mean=395    mean=425
(M=15-W=6)  mean=435    mean=391
(M=15-W=7)  mean=379    mean=405
(M=15-W=8)  mean=388    mean=412
(M=15-W=9)  mean=399    mean=432
(M=15-W=10) mean=432    mean=389
(M=15-W=11) mean=397    mean=432
(M=15-W=12) mean=426    mean=393
(M=15-W=13) mean=404    mean=407
(M=15-W=14) mean=429    mean=412
(M=15-W=16) mean=539    mean=391
(M=15-W=17) mean=414    mean=397
(M=16-W=2)  mean=456    mean=368
(M=16-W=3)  mean=422    mean=406
(M=16-W=4)  mean=445    mean=384
(M=16-W=5)  mean=427    mean=397
(M=16-W=6)  mean=462    mean=348
(M=16-W=7)  mean=413    mean=408
(M=16-W=8)  mean=419    mean=361
(M=16-W=9)  mean=429    mean=385
(M=16-W=10) mean=463    mean=369
(M=16-W=11) mean=426    mean=404
(M=16-W=12) mean=454    mean=391
(M=16-W=13) mean=434    mean=378
(M=16-W=14) mean=454    mean=412
(M=16-W=15) mean=424    mean=416
(M=16-W=17) mean=578    mean=378
(M=17-W=2)  mean=460    mean=402
(M=17-W=3)  mean=419    mean=381
(M=17-W=4)  mean=446    mean=394
(M=17-W=5)  mean=424    mean=422
(M=17-W=6)  mean=468    mean=369
(M=17-W=7)  mean=409    mean=401
(M=17-W=8)  mean=418    mean=405
(M=17-W=9)  mean=428    mean=414
(M=17-W=10) mean=459    mean=369
(M=17-W=11) mean=424    mean=387
(M=17-W=12) mean=451    mean=372
(M=17-W=13) mean=435    mean=382
(M=17-W=14) mean=459    mean=369
(M=17-W=15) mean=426    mean=401
(M=17-W=16) mean=446    mean=371

McCalpinJohn · ‎08-28-2019

I don't understand your code, but these 18-core processors each have 6 disabled cores, so the positions of the logical processors are likely to be different on many chips.

There is a bit map of the enabled L3 slices in a PCI configuration space register known as CAPID6. On a typical 2-socket system, these can be found with:

# setpci -s 17:1e.3 0x9c.l
0ffbfdff
# setpci -s 85:1e.3 0x9c.l
0fff9fff

These examples are from two 26-core processors in a 2-socket node. The low-order 28 bits of this 32-bit word are a bit map of the enabled L3 slices. Every system that I have checked has the same number of L3 slices disabled in the upper 14 bits and the lower 14 bits (probably to better support SNC mode). In this example, the processor in socket 0 has L3 slices disabled at positions 9 and 18, while the processor in socket 1 has L3 slices disabled at positions 13 and 14. Some discussion about how the bit positions map to locations on the die is in my presentation at https://www.ixpug.org/documents/1524216121knl_skx_topology_coherence_2018-03-23.pptx

The position of the cores is not the only issue, unfortunately. The latency depends on that, but it is also strongly dependent on the location of the L3 slice that is responsible for handling the cacheline address you are using. Cores that are adjacent can have very low cache-to-cache intervention latency if they are exchanging an address mapped to one of the co-located L3 slices, or they can have a very high cache-to-cache intervention latency if they are exchanging an address mapped to a distant L3 slice. Cores that are further apart have less variation by address (but higher best-case values). You might want to try repeat your test using each of the 64 cache lines in a 4KiB page as the shared variable. (I recommend using "mlock()" on the data page to prevent the OS from changing the virtual to physical translation during the run. Alternately, I use the /proc/self/pagemaps interface to look up the physical address of the base of the 4KiB page before the tests and again after the tests.)

AATLI3 · ‎08-29-2019

The attached code was to check the aligned 64-byte loads/stores atomicity using AVX512. Also scalar loads/stores produced the similar results on both systems, so that makes me suspicious of AVX512, not the bitmap..

In order to find the disabled cores, I tried to read CAPID6 to check the bitmap of the enabled L3 slices in a PCI configuration space register as below, and it seems all 28 L3/CHA slices are enabled, but I have only 18 cores. What's wrong with it?

$ lspci | grep :1e.3
16:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
$ setpci -s 16:1e.3 0x9c.l
ffffffff

McCalpinJohn · ‎08-30-2019

The response "ffffffff" (all 32 bits) is used to indicate an error -- in this case it is probably a simple permissions error. User processes are allowed to read the first 64 Bytes of each PCI devices configuration space, but root permission is needed to read past that point. My output above was run as root. On a 28-slice processor, the output is 0ffffffff.

I am not drawing any conclusions about the source of the latency difference yet -- there are several possibilities, and unless all the variables are properly controlled, it is easy to get misled by the results.....