Xeon W-1270P: PCIe read access to I350 register is slow and jitters more than on Xeon E3-1285 v6

[Martin_Moe]

Hello,

the Xeon W-1270P (and also the W-1290P) takes more time for PCIe read accesses, e.g. to the Interrupt Cause Register (E1000_ICR) of an I350-T4 Ethernet card. It mostly is in the range of 0 to 20 microseconds and sometimes shoots up to 65 or 70 microseconds for a simple 32-Bit read, resulting in long times for interrupt processing.

We previously had Xeon E3-1285 v6 CPUs wich C236 Chipset, which performed much better (Always less than 10 microseconds for the ICR read).

Hyperthreading is disabled. Chipset is W-480. I350-T4 are attached to the CPU PCIe (configured to 1 time x8 and 2 times x4 supporting 3 slots). 6 Interfaces of 2 I350-T4 cards are utilized with 12500 frames per second (transmit and receive), 1400 bytes each.

Why is the performance of the new CPU worse than on the old CPU?

Kind Regards

Martin Moe.

[Victor_G_Intel]

Hello Martin_Moe,

Thank you for posting on the Intel® communities.

In this case, I need to inform you that we don’t do benchmarks between our products; however, depending on the software you used to complete your tests and the methods used along with the full environment of the systems in question, performance, in general, is something that can always vary.

Regards,

Victor G.

Intel Technical Support Technician  

[Martin_Moe]

Hello again,

of course measurements can vary, but by a factor of 10?

If have read about several features affecting PCIe transactions (DDIO, DCA, IIO Snoop Response Hold Off, home snoop, source snoop).

I found

https://community.intel.com/t5/Software-Tuning-Performance/PCIe-perfomance-drops-on-Haswell-Broadwell-CPUs/td-p/1125831

and

https://community.intel.com/t5/Software-Tuning-Performance/temporary-pcie-bandwidth-drops-on-Haswell-v3/td-p/1081298

My BIOS does not provide any settins for "IIO Snoop Response Hold Off" and the CPU does not support the MSR_IIO_LLC_WAYS (0xC8B).

Now I am wondering how can I workaround the issue of the Xeon W-1270P sporadically stalling for "tens of microseconds" on an operation which normally completes in a microsecond or less.

• The PCIe Device Status of root port and I350 do not indicate any errors on the physical PCIe links.
• Using old Intel PRO1000 Quad Low Profile cards in place of the I350-T4 results in the exact same behaviour.
• The core p-state is pinned (no frequency changes)
• The ring ratio (uncore frequencey) is pinned (set min to max)
• The network card interrupts are all routed to core 7, no (!) other threads are scheduled on core 7.
• The 32-bit software is a monolith with an integrated proprietary real time OS. Tests on "old" and "new" CPU have been done with the same piece of software.

So please, give any hint on what to further observe.

Martin Mö.

[Victor_G_Intel]

Hello Martin_Moe,

Thank you so much for your response.

In order for us to look forward to this please provide the following information:

1. Are you a developer?
2. Is this request of yours been made on behalf of a company? If yes, please provide as many details as possible about the company that you made the request for.
3. Can you share with us the reason you started this whole testing process, is this investigation for some sort of project?
4. Please provide the exact model of the motherboard that you have this processor installed on.
5. Can you share with us some screenshots of the tests that you made and maybe share with us the software/tool that you used to measure the differences in PCIe read accesses?

Best regards,

Victor G.

Intel Technical Support Technician  

[Martin_Moe]

Hello Victor,

I have provided the requested information on the supportticket 05648536.

Best regards,

Martin Mö.

[Victor_G_Intel]

Hello Martin_Moe,

Thank you for posting on the Intel® communities.

Please let me review this information internally, and kindly wait for an update.

Once we have more information to share, we will post it on this thread.

Regards,

Victor G.

Intel Technical Support Technician  

[Victor_G_Intel]

Hello Martin_Moe,

Thank you for your patience.

We have reviewed your request and unfortunately, we cannot provide any details or replicate anything since you are using your own software for these tests.

Additionally, both the processors you have are from different families and have different features and on our end, we don’t compare our own products based on their performance.

Since the only issue is the one experienced with the software and the PCIe read accesses, we would recommend you address this situation with the rest of the team of developers involved in your project.

Regards,

Victor G.

Intel Technical Support Technician