Ice Lake (Xeon Gold 6346) PCIe TLP using random RequesterIDs

[EricPilmore]

We recently acquired a Supermicro (SYS-220U-TNR, BIOS 1.1a) box containing an Xeon Gold 6346 (Ice Lake) with PCIe Gen4. In doing PCIe TLP captures of I/O Read transactions originating from a CPU core to I/O devices, I am seeing that the RequesterID (RID) within the MemRD TLPs seems to vary regardless of CPU Core that is being utilized. In addition, the RID does not represent any enumerated device in the PCIe tree (as reported by "lspci"). On previous platforms/processors that I have used, the RID of such I/O Read transactions is typically either 0x0000 or the RID of the Root Port into which the respective I/O device is plugged into.

Can somebody point me to any material that might explain this behavior and why (I have some speculations as to why, but would like confirmation)? I would like to understand how the RIDs are selected. Also, is this a feature that can be managed by software, i.e. disabled by either OS or BIOS code? Was not clear if this is necessarily a behavior of the processor itself or the platform (I'm assuming processor).

Thanks in advance!

[JoseH_Intel]

Hello EricPilmore,

Thank you for joining the community.

Please allow me a bit of time to research on your question. I will get back to you soon.

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[JoseH_Intel]

Hello EricPilmore,

We are still looking into your inquiry and will provide an update by end of week. We will let you know soon. Thanks for your patience.

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[EricPilmore]

Hi Jose,

Much appreciated! Anxiously awaiting your findings.

Eric

[JoseH_Intel]

Hello EricPilmore,

Due to the complexity of the issue, a deeper research is been held. For this reason an update will be provided next week. We definitely appreciate your patience.

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[EricPilmore]

Hi Jose,

I do appreciate the complexity and your time digging into it. 

Thanks for the update!

Eric

[EricPilmore]

Hi Jose,

Through our Intel contact we were able to find the information I was looking for. I won't disclose the details here, but for those that have a CNDA with Intel, the info can be found in the following document:

• RDC#574451 - “3rd Generation Intel® Xeon® Scalable Processors, Codename Ice Lake-SP External Design Specification (EDS), Volume One: Architecture” section 4.3.5

There does not appear to be a means of "disabling" this behavior or restoring to the "traditional" behavior of using the BDF of the owning Root Port. The above document also does not explicitly state the reason behind this change, although it does allude to the fact that the number of possible RIDs, and thus number of simultaneously outstanding CPU IO Reads, does greatly increase beyond the typical PCIe limitation of 32 (or 256 using Extended Tags).

[JoseH_Intel]

Hello EricPilmore,

Thank you very much for your update. Through our internal research, we arrived to the same conclusion found in Section 4.3.5 of the 3rd Gen Xeon Scalable Processors EDS Volume 1. Since the information is found there, then, we will proceed to mark this thread as resolved. If you have further issues or questions just go ahead and submit a new topic.

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios