Embedded Connectivity
Intel network controllers, Firmware, and drivers support systems
845 Discussions

i210 PCIe read random hang

EDano
Beginner
2,423 Views

Hi,

We are validating a custom board embedding Intel i210 product, which connect the processor through PCIe.

Our custom board system is linux 4.1.15 which contains i210 driver natively.

i210 is connected to a serial flash, which we already programmed through SPI with a SPI programmer.

We programmed our serial flash with Dev_Start_I210_Copper_NOMNG_4Mb_A2_3.25_0.03.bin (provided by one of your distributor).

During linux boot up, i210 driver is correctly initializing itself (Device ID 0x1533).

It performs a lot of read through PCIe, but it hangs at a random PCIe read : "value = readl(&hw_addr[reg]);" in "igb_rd32(struct e1000_hw *hw, u32 reg)" in "igb_main.c".

In attached files :

- log of i210 initialization process,

- log of lspci -vv

Can you give us some hint ?

In advance, thank you for your support.

0 Kudos
7 Replies
CarlosAM_INTEL
Moderator
1,108 Views

Hello, Axurit

Thank you for contacting the Intel Embedded Community.

In order to have a better idea of this situation, we would like to address the following questions:

Could you please tell us if the affected design is a third party one or it has been developed by you? In case that it is your project, could you please tell us if it has been developed based on the suggestions stated in the Ihttps://edc.intel.com/Link.aspx%3Fid%3D8393 ntel(R) Ethernet Controller I210: Design Guide document # 513305, https://edc.intel.com/Link.aspx%3Fid%3D8390 Intel(R) Ethernet Controller I210-AT/IT: Layout Review Checklist document # 495298, and https://edc.intel.com/Link.aspx%3Fid%3D8391 Intel(R) Ethernet Controller I210-IS: Layout Review Checklist document # 495299? Also, could you please confirm us if it has been verified by Intel?

Thanks in advance for your collaboration to solve this inconvenience.

Best regards,

Carlos_A.

0 Kudos
EDano
Beginner
1,108 Views

Hello Carlos

Thanks for your quick answer.

The design was not reviewed by Intel but the schematic was reviewed by the AVNET specialist.

We review the schematic using the check list you sent and the main differences are that :

  • we have 220 nf capacitors on the i210 Receive lines these capacitors are near the iMX6 processor
  • we also have 100 nF capacitors on the Clock lines (the capacitors are recommended in the iMX ref design).

We don't review for the moment the Ethernet side because our issue is the PCIe interface.

We make some changes :

  • replacing the 220 nF capacitors on the RX line with 100 nF capacitors but the issue remains the same.
  • Replacing the 100 nF capacitors with 0 W

The last thing we try is to remove the 100 nF (replaced with 0W resistor) on the TX lines of the i210 with the 100 nF capacitors on the RX and clock lines (these is the way the PCIe interface is implemented on the iMX ref design).

The first time we launch the driver all seems good but we can't ping our CPU using this Ethernet channel.

The other times the driver stops but seem to go further.

We have added traces in the igb driver and the drivers always hangs on read instruction but never at the same address.

We think we have some timing issues on the PCIe interface but we cannot find how to correct that.

The imx6 has some capabilities modifying timing at the PCIe phy level but it seem it mainly affect the transmit line of the processor.

The i210 and the iM6 are very close in our design which should minimize the timming issue.

Do you have any idea?

You will find here attached the 3 schematic sheets of our design which concern the iMX6 and i210 PCIe interface if you want to review them.

One other issue we have was with EEPROMARMTOOL that couldn't run and stops with bus error.

We have a look on several discussion on embedded community that have the same problem but we can't find the way they solve this issue (it seems that the problem is due to memory mapping).

So we use an AARDVARK USB to SPI interface to program the i210 external Flash and we found we have a bug on the AARDVARK interface with the flash (you can see on the schematic that the pin 5

of the connector is connected to Flash DO0 instead of flash DO1) this issue was corrected on the board so we can program the flash but the flash read hangs using the external probe.

Is there any link with the driver issue?

Thank for your help, if you need some more information on our design or the whole schematic don't hesitate to ask me.

Best regards

0 Kudos
CarlosAM_INTEL
Moderator
1,108 Views

Hello, Axurit:

Thanks for your detailed reply.

Based on your previous communication, we want to address the following questions:

Could you please verify that your implementation fulfills with the layout requirements stated at the documents listed on our previous communication? In case that there are discrepancies please correct them, try to reproduce the problem, and let us know the consequences of these changes.

Could you please confirm if the system hangs, or just the part? in case that it is the part when it hangs, does it come back after a reset or is a power cycle needed? or does it hang during boot up, or after the system has been up for a bit? please give us a detailed explanation.

Thanks again for your cooperation.

Best regards,

Carlos_A.

0 Kudos
EDano
Beginner
1,108 Views

Hello Carlos

We will review the lauout next week with our specialist and if there is somme issues on the layout we will modify the board and make a new pcb.

In fact it's not the system that hangs its the igb driver.

If we launch the baord without loading the driver all works fine and the i210 is detected using the lspci (you have the lspci -vv result attached withthe first message).

When we launch the driver, rhe driver hangs and it's why the whole system hangs.

We add traces in the driver abd the last action before the driver hangs is a register read but never at the same address (you also have a trace of these prints attached with the first message).

It seems that there is someting wrong (perhaps a timing issue or someting like that) whent the imX6 tries to read the i210 registers.

Modifying the Ac coupling capacitor on these lines (replacing them by 0 Hom resistor) seems to make some diffrences.

The first time we start the driver after this modification all seems to work. The driver launch doesn't hang and we see the the eth link and can configure it using ifconfig but we cant ping the board.

so we restart the baord and the driver hangs again at each launch.

We realy think tere is some issu on the read lines or in the driver.

Best regards

0 Kudos
CarlosAM_INTEL
Moderator
1,108 Views

Hello, Axurit:

Thanks for your update.

In order to help you, could you please try to reproduce this situation using the latest version of the driver and let us know the results?

We really appreciate your collaboration.

Best regards,

Carlos_A.

0 Kudos
EDano
Beginner
1,108 Views

Hello Carlos A,

Thank you for suggestion.

We tried the latest igb driver retrieved from intel support website : igb-5.3.5.4.tar.gz, but it fails the same way.

Attach file : the log of the initialization process until it hangs.

Like with the igb mainline driver of linux-4.1.15, hangs appear randomly.

Were you able to check board schematics of PCIe connection ?

Your ideas are very appreciated.

Thank you for your support.

Best regards.

0 Kudos
CarlosAM_INTEL
Moderator
1,108 Views

Hello Axurit:

Thanks for your reply.

In case that you need to review your design, please follow the steps stated at the https://edc.intel.com/Tools/Design-Review/Default.aspx?language=en Design Review Services website.

We hope that this information may help you.

Best regards,

Carlos_A.

0 Kudos
Reply