Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4810 Discussions

Intel NIC I350 AAP failing to enter PCIe L2 state

APate19
Beginner
3,016 Views

Hi,

We are observing that when Intel NIC I350 AAP (PCIE Endpoint) is used with Nvidia PCIE Root port, PCIE link fails to enter into L2 states in some cases.

Failure Scenario:

The application software initiates lower power sequence to bring the PCIE link to enter into L2 state by first bringing the endpoint device into D3hot state followed by PME_Turn_Off request from Nvidia Root port to the endpoint device(NIC).

It is observed that the endpoint device(NIC) does bring the link into L1 and also responds with PME_TO_ACK as expected but it does not initiate L2 entry request. So the PCIE Link (Root port) fails to enter into L2 state.

Details of the failure sequence and observations:

There seems to be some issue with the NIC I350 AAP during the L2 entry using Sequence-1 below, whereas same sequence works for the NIC - I350AM4. The Sequence-1 followed for the L2 entry and device details are described below.

Sequence-1 (Failing Sequence): L0 → L1 → L0 → L2/L3 Ready: Intel NIC I350 AAP Failing while NIC - I350AM4 works fine.

 

1. Device is in L0 and System software directs all Functions of a Downstream component to D3hot.

 

2. The Downstream component then initiates the transition of the Link to L1 as required.

 

3. System software then causes the Root Complex to broadcast the PME_Turn_Off Message in preparation for removing the main power source.

 

4. This Message causes the subject Link to transition back to L0 in order to send it and to enable the Downstream component to respond with PME_TO_Ack.

 

5. After sending the PME_TO_Ack, the Downstream component initiates the L2/L3 Ready transition protocol. (Intel NIC I350 AAP fails in this step)

Failing Device Details :

I210T1BLK - Single Port NIC ( I350 AAP)

Date of Manufacture : 05/2014

Device 8086:1533 (rev 03)

Working Device Details :

Dual Port NIC - I350AM4

Date of Manufacture : 06/2013

Device 8086:1521 (rev 01)

It is also observed that both Intel NIC I350 AAP and NIC I350AM4 enter into L2 without any issue while following sequence-2 below.

Sequence-2 (passing sequence):

 

1. Device is in L0 and system software causes the Root Complex to broadcast the PME_Turn_Off Message in preparation for removing the main power source.

 

2. The Downstream components respond with PME_TO_Ack.

 

3. After sending the PME_TO_Ack, the Downstream component initiates the L2/L3 Ready transition protocol.

I am looking for help from your team if there is any known issue with Intel NIC I350 AAP related to Link entering into L2 while following Sequence-1 describe above.

Please also suggest how can we proceed further to root cause the issue and avoid the failure for the L2 entry while following the sequence-1.

Thanks & Regards,

Alap

Baltazar

0 Kudos
16 Replies
idata
Employee
1,590 Views

Hi AlapPatel,

 

 

Thank you for posting in Wired Communities. Please provide information below for clarification:

 

1) Are you referring to I210T1 or I350 onboard NIC ,as you mentioned in the failing device detail "I210-T1blk" but with I350-AAP

 

2) What is the application software used to initiate the power sequence?

 

3) What is the operating system?

 

4) What is the server or system these NIC are installed?

 

 

Looking forward to your reply.

 

 

Thanks,

 

Sharon T
0 Kudos
vsaga
Beginner
1,590 Views

>> 1) Are you referring to I210T1 or I350 onboard NIC ,as you mentioned in the failing device detail "I210-T1blk" but with I350-AAP

 

This is I210T1

Vendor-ID: Device-ID -> 8086:1533

Subsystem Vendor-ID : Device-ID -> 8086:0002

>> 2) What is the application software used to initiate the power sequence?

If you are referring to the power sequence followed to power-up Intel NIC card, then, there is no SW involvement as such there. Powering up Tegra platform to which Intel NIC is connected takes care of powering up Intel NIC as well. Are you interested in any specific sequence here?

 

>> 3) What is the operating system?

Linux

 

>> 4) What is the server or system these NIC are installed?

It's an Nvidia Tegra based custom platform

0 Kudos
idata
Employee
1,590 Views

Hi VidyaSagar,

 

 

Thank you for the information. Just to double check:

 

1) As mentioned "The application software initiates lower power sequence to bring the PCIE link to enter into L2 state by first bringing the endpoint device into D3hot state followed by PME_Turn_Off request from Nvidia Root port to the endpoint device(NIC)" so how do you observe the power is not entering L2 state?

 

 

2) So it means you are using an Intel I210-T1 retail unit installed on NVIdia Tegra platform?

 

 

3) What is the i350 APP refers to?

 

 

4) What is the driver used for the I210-T1 NIC? Can you provide the link where you download the driver

 

 

5) How did you perform all these sequence? can you provide the steps?

 

 

Thanks,

 

Sharon T

 

0 Kudos
APate19
Beginner
1,590 Views

Hi Sharon

Please find responses below -

>>1) As mentioned "The application software initiates lower power sequence to bring the PCIE link to enter into L2 state by first bringing the endpoint device into D3hot state followed by PME_Turn_Off request from Nvidia Root port to the endpoint device(NIC)" so how do you observe the power is not entering L2 state?

We verified with PCIe logic analyzer traces, following are our observations,

"When Intel NIC is in D0, it responds with PME_TurnOff_Ack for PME_TurnOff broadcast message and then starts sending Enter_L2/3 message .. and finally link enters into L2.

Whereas when Intel NIC is in D3_Hot it responds with PME_TurnOff_Ack for PME_TurnOff broadcast message, but after that it starts sending L1 entry requests."

 

>>2) So it means you are using an Intel I210-T1 retail unit installed on NVIdia Tegra platform?

Yes

 

 

>>3) What is the i350 APP refers to?

On Intel chip "1350AAP" and "WGI210AT" serial number are present. These number are hard to read, so we got confused "1" with "I".

 

 

>>4) What is the driver used for the I210-T1 NIC? Can you provide the link where you download the driver

We are using linux-4.9.61 upstream igb driver.

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/tree/drivers/net/ethernet/intel/igb?h=v4.9.82&id=bc438831606acd445bad9f7df20ac6ce87d5b490 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/tree/drivers/net/ethernet/intel/igb?h=v4.9.82&id=bc438831606acd445bad9f7df20ac6ce87d5b490

 

>>5) How did you perform all these sequence? can you provide the steps?

Sequence:

  1. igb_suspend() function will put device in D3hot
  2. The Downstream component then initiates the transition of the Link to L1 as required
  3. Tegra root complex broadcasts PME_Turn_Off
  4. Tegra root complex driver will poll for link state L2.

Step-4 is failing because of the issue mentioned in point-1.

Please let us know if more information is required from our side.

Thanks & Regards,

Alap

0 Kudos
idata
Employee
1,590 Views

Hi AlapPatel,

 

 

Thank you for the additional information provided. I will have to check further.

 

 

Regards,

 

Sharon T
0 Kudos
idata
Employee
1,590 Views

Hi AlapPatel,

Can you clarify when you said on Intel chips "1350AAP" where is this Intel chip? Are you referring this 1350AAP is found on the chipset of I210-T1 network adapter (which is a standalone NIC) you can plug in and out? What you referring here is I210-T1 is the failing device (not the 1350AAp?)? Thanks.

Thanks,

 

Sharon

 

0 Kudos
vsaga
Beginner
1,590 Views

I think you can ignore "1350AAP" for now. What we are using is a I210-T1 card. Link for the same is : https://ark.intel.com/products/68668/Intel-Ethernet-Server-Adapter-I210-T1 Intel® Ethernet Server Adapter I210-T1 Product Specifications

0 Kudos
idata
Employee
1,590 Views

Hi AlapPatel,

 

 

Thank you for the confirmation. That is what I want to clarify. Much thanks.

 

 

Regards,

 

Sharon T
0 Kudos
APate19
Beginner
1,590 Views

Hi Sharon T,

Do you have any further update on this issue?

Thanks & Regards,

Alap

0 Kudos
idata
Employee
1,590 Views

Hi AlapPatel,

 

 

We are still checking on this, will update you if there is any thing to share.

 

 

Regards,

 

Sharon T
0 Kudos
APate19
Beginner
1,590 Views

Hi Sharon

For this NIC,

MM # 921434

Please find attached images showing further details.

Thanks & Regards,

Alap

0 Kudos
idata
Employee
1,590 Views

Hi AlapPatel,

 

 

Thank you for providing the NIC picture. I hope you don't mind to provide below information for better investigation:

 

 

1) What is the loaded driver of I210-T1?

 

 

The one you provided linux-4.9.61 is the kernel, can you run ethtool -i command to output the version for both I210-T1 and I350-AM4?

 

 

2. Can you provide the exact name and version of the PCIE logic analyzer initiates the L2 state

 

 

3. Please run ethtool -k to provide the output of I210-T1 and I350-AM4 to compare the settings

 

 

Looking forward to your reply.

 

 

Thanks,

 

Sharon T
0 Kudos
APate19
Beginner
1,590 Views

Hi Sharon,

Please find attached files for output of "ethtool -i " and "ethtool -k".

L2 is initiated by Nvidia PCIe root port, not logic analyzer.

Thanks & Regards,

Alap

0 Kudos
idata
Employee
1,590 Views

Hi AlapPatel,

 

 

Thank you for the information. I will further check on this.

 

 

Regards,

 

Sharon T
0 Kudos
idata
Employee
1,590 Views

HI AlapPatel,

 

 

Just to double check if you have NDA with us since there is newer NVM image for this NIC which required NDA access.

 

 

 

Thanks,

 

Sharon T
0 Kudos
idata
Employee
1,590 Views

HI AlapPatel,

 

 

Please feel free to update us about the NDA so that we can further assist.

 

 

Thanks,

 

Sharon T
0 Kudos
Reply