Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4811 Discussions

Intel XL710 operation without PE_RST_N signal

MStar1
Beginner
810 Views

Hi!

I would like to report an issue I've been observed with a XL710-based board.

There are system architectures where there's no physical RESET# pin distributed across the backplane to reset downstream devices, MicroTCA is one of them.

 

There's a problem when pairing a Xeon D-1500 PrAMC with an AMC NIC card based on XL710 controller. When the boards are powered up, the PCIe link comes up and everything works until the CPU needs to be reset (either during the BIOS code execution or via a reboot command from the OS). After the reset the PCIe link is dropped on the CPU side, and XL710's LTSSM gets stuck at that moment and doesn't respond to the link training that occurs again from the CPU side in a while. As a result the NIC card disappears from the PCIe device list.

 

I tried older AMC NIC cards with different controllers - i350, X540 and X599, and all of them successfully pass through such reset scheme without issues, but XL710 fails.

 

Also tried different NVM firmware images for XL710, from 4.42 to 6.01, but nothing changes.

 

Does anybody know how to fix the issue, maybe via some modifications/patches to the NVM image?

Thanks.

 

Best regards,

Mikhail.

0 Kudos
5 Replies
Mike_Intel
Moderator
567 Views

Hello MStar1,

 

Thank you for posting in Intel Ethernet Communities. 

 

Before we proceed and for us to further understand the issue, Please provide the following details.

 

  1. Can you provide any data about the issue?
  2. Are you designing a server system and trying to embed the NIC?
  3. What is the purpose of the system?

 

If you have questions, please let us know.

 

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

0 Kudos
MStar1
Beginner
567 Views

Hi Michael,

Thanks for your response.

 

The MicroTCA system is a telecom class hardware, built around AdvancedMC modules.

Here's the datasheet for the chassis: https://bit.ly/2JHBfXw

Here's the Short Form AMC specification (for board pinout and general system understanding): https://www.picmg.org/wp-content/uploads/AMC.0_R2.0_Short_Form.pdf

The CPU is not a serious server (with BMC, remote management and other features), but it's a light, small form-factor server, based on a Xeon D-1500 CPU with DDR4 ECC RAM, with a discrete dual-port 10GbE NIC card based on a XL710 silicon.

 

In short, the CPU module in one slot (AMC4) is connected to the NIC module in another slot (AMC2) via the PCI Express x4 interface (Ports 4-7 on the diagram), but the link only includes TX and RX lanes and doesn't contain the PERST# signal that is generally used in PCI Express systems.

After initial power-up, the XL710 NIC card correctly passes the RX detect state, and then the link is established in a while (the Xeon D-1500 doesn't bring up the PCIe links right after the power is applied, but after memory init and some other BIOS actions).

When the reset event on the CPU module occurs, the NIC card doesn't receive the PERST# signal assertion, it only "sees" a link down condition, and here something goes wrong with the XL710 Link training state machine and it doesn't respond for further attempts of link negotiation.

 

I have an idea about how you can easily reproduce the problem - you need to take a generic XL710-based NIC add-in card and simply isolate the PERST# signal on the PCIe edge card connector (with a paper slice, a piece of tape, of somehow).

Then the modified card can be inserted into a motherboard, and you can check how the chip handles the reset events and PCIe fall-down and re-negotiation withour being reset via the PERST# signal.

 

Best regards,

Mikhail.

0 Kudos
Mike_Intel
Moderator
567 Views

Hello MStar1,

 

By the way, when you say  XL710-based board, are you referring using an Xl710 Ethernet controller (chipset)? 

 

If you have questions, please let us know.

 

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

0 Kudos
Mike_Intel
Moderator
567 Views

Hello MStar1,

 

Just making a follow up to clarify the information below.

 

when you say XL710-based board, are you referring using an Xl710 Ethernet controller (chipset)? 

 

If you have questions, please let us know.

 

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

 

0 Kudos
Mike_Intel
Moderator
567 Views

Hello MStar1,

 

Just sending another follow up on the information that I asked for me to further assist you.

 If you have questions, please let us know.

 

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

0 Kudos
Reply