Same binary yields 3.1 IPC on Skylake but 2.3 IPC (Frontend BW Bound) on Cascade Lake

[Mark_D_9]

We've built a multithreaded C++ application using GCC 10.2 and the Gold Linker that achieves an IPC of 3.1 on both our Xeon Gold 6144 and our i9-7980XE (both Skylake CPUs). Yet, that same binary yields an IPC of only 2.3 when run on an i9-10980XE (Cascade Lake). Further analysis with PMU-Tools toplev.py reveals that the application is heavily Front End Bandwidth Bound when running on Cascade Lake.

So, we rebuilt the application with 32-byte alignment and that brought IPC up to ~2.8 (still shy of the 3.1 IPC reached on our Skylake systems):

-falign-functions=32 -falign-jumps=32 -falign-labels=32  -falign-loops=32

How is this so when there supposedly isn't any core microarchitectural changes between Skylake and Cascade Lake to warrant such a difference in the Front End operation? Have you seen anything like this before?

[SergioS_Intel]

Hello Mark_D_9,

We are currently working on your question, can you please confirm the model of the server board that you are using?

Best regards,

Sergio S.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios

[Mark_D_9]

Here is the model and server board information for both systems as provided by dmidecode:

SKYLAKE SYSTEM

System Information
        Manufacturer: Dell Inc.
        Product Name: PowerEdge R640
        Version: Not Specified
        Serial Number: 8M73YM2
        UUID: 4C4C4544-004D-3710-8033-B8C04F594D32
        Wake-up Type: Power Switch
        SKU Number: SKU=NotProvided;ModelName=PowerEdge R640
        Family: PowerEdge

Handle 0x0200, DMI type 2, 8 bytes
Base Board Information
        Manufacturer: Dell Inc.
        Product Name: 0W23H8
        Version: A00
        Serial Number: .8M73YM2.CNIVC0081B1026.

CASCADE LAKE SYSTEM:

System Information
        Manufacturer: Ciara Technologies
        Product Name: ORION HF310-G4
        Version: 0100
        Serial Number: 421030101380
        UUID: db938000-fd74-11e7-8000-b42e99f6cb22
        Wake-up Type: Power Switch
        SKU Number: HF310-G4
        Family: Server

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
        Manufacturer: GIGABYTE
        Product Name: MH11-OC2-Y9
        Version: 00000001
        Serial Number: JGCNC300008
        Asset Tag: 01234567890123456789AB
        Features:
                Board is a hosting board
                Board is replaceable
        Location In Chassis: 01234567890123456789AB
        Chassis Handle: 0x0003
        Type: Motherboard
        Contained Object Handles: 0

[Mark_D_9]

Also, we are NOT using "-march=native" on our builds. We only use "-O3" and LTO.

[SergioS_Intel]

Hello Mark_D_9,

Thank you for the detailed information, please allow us to investigate on this and we will get back to you.

Best regards,

Sergio S.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios

[DeividA_Intel]

Hello Mark_D_9,  

Thank you for posting on the Intel® communities.   

I will send you an e-mail to the e-mail address associated with your community profile. Please reply to the email in order to provide us with the details needed for the investigation. This is to avoid exposing sensitive information in the communities.  

Regards,  

Deivid A. 

Intel Customer Support Technician 

[Mark_D_9]

I never received your email. Did you send it to my WH Trading LLC work email at (deleted for security reasons) Our company already has a mutual NDA signed with you for technical support issues like this.

[DeividA_Intel]

Hello Mark_D_9, 

For the moment we don't have much information on this issue.

As for our findings for the key difference between the 2 CPUs you are testing is the number of memory channels and the per-core cache. We are still trying to get more insights on this issue. However, it may take some time but we cannot make any particular commitments right now.

Please let me know if you have any additional questions.

Regards,  

Deivid A. 

Intel Customer Support Technician