82599EN repeats Link Up and Down

[TSuzu6]

Hi all

We use ixgbe driver for 82599EN on linux. driver version is 4.1.2

diriver is downloaded at https://downloadcenter.intel.com/download/25463/Network-Adapter-Driver-for-PCI-E-Intel-10-Gigabit-Ethernet-Network-Connections-under-Linux https://downloadcenter.intel.com/download/25463/Network-Adapter-Driver-for-PCI-E-Intel-10-Gigabit-Ethernet-Network-Connections-under-Linux-.

[test condition]:

OS:Linux kernel 3.10.31-ltsi

driver:Network Adapter Driver for PCI-E* Intel® 10 Gigabit Ethernet Network Connections under Linux* ver4.1.2

device:Intel 82599EN

PHY I/F:SFP module(FINISAR FTLX8573D3BTL) + MMF Fiber cable

EEPROM data:82599EN_SFI_NO_MNG_4.40.bin(modiy MAC address)

[steps]:

1.insmod mdio.ko

2.insmod ixgbe.ko

3.ifconfig eth2 up

4.ifconfig eth2 192.168.2.100

5.connect MMF cable to 10GbE Tester(SPIRENT C1).

[result]

------------------------------------------------------------------------------

mailto:root@socfpga:/usr/fsim root@socfpga:/usr/fsim# insmod ixgbe.ko

ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.13.10-kixgbe: Copyright (c) 1999-2013 Intel Corporation.ixgbe_probePCI: enabling device 0000:01:00.0 (0140 -> 0142)PCI resource c0000000 + 00020000ixgbe 0000:01:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1ixgbe 0000:01:00.0: (PCI Express:5.0GT/s:Width x4) 00:a0:c9:12:34:56ixgbe 0000:01:00.0: MAC: 2, PHY: 12, SFP+: 5, PBA No: FFFFFF-0FFixgbe 0000:01:00.0: PCI-Express bandwidth available for this card is not sufficient for optimal performance.ixgbe 0000:01:00.0: For optimal performance a x8 PCI-Express slot is required.ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connectionplatform leds.7: Driver leds-gpio requests probe deferral

---------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------

mailto:root@socfpga:/usr/fsim root@socfpga:/usr/fsim# ifconfig eth2

eth2 Link encap:Ethernet HWaddr 00:a0:c9:12:34:56 inet addr:192.168.2.100 Bcast:192.168.2.255 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

mailto:root@socfpga:/usr/fsim root@socfpga:/usr/fsim# ixgbe 0000:01:00.0 eth2: NIC Link is Up 10 Gbps, Flow Control: RX/TX

ixgbe 0000:01:00.0 eth2: NIC Link is Downixgbe 0000:01:00.0 eth2: NIC Link is Up 10 Gbps, Flow Control: RX/TXixgbe 0000:01:00.0 eth2: NIC Link is Downixgbe 0000:01:00.0 eth2: NIC Link is Up 10 Gbps, Flow Control: RX/TXixgbe 0000:01:00.0 eth2: NIC Link is Downixgbe 0000:01:00.0 eth2: NIC Link is Up 10 Gbps, Flow Control: RX/TXixgbe 0000:01:00.0 eth2: NIC Link is Downixgbe 0000:01:00.0 eth2: NIC Link is Up 10 Gbps, Flow Control: RX/TXixgbe 0000:01:00.0 eth2: NIC Link is Down

--------------------------------------------------------------------------------------------------

Our issue is that link up and down is repeated. How should we resolve this issue ?

BR,

Taira Suzuki

[idata]

Hi Taira,

Thank you for the post. Have you tried using the newer version 4.3.15 https://downloadcenter.intel.com/download/14687/Network-Adapter-Driver-for-PCI-E-Intel-10-Gigabit-Ethernet-Network-Connections-under-Linux-?v=trgds,wb

[TSuzu6]

Hi wb

I tried to use ixgbe ver4.3.15. But the issue is occured again.

I can not resolve this issue. eth2(target eth) status is as below.

[staus result]

mailto:root@socfpga:/usr/fsim root@socfpga:/usr/fsim# ethtool eth2

Settings for eth2:

Supported ports: [ FIBRE ]

Supported link modes: 10000baseT/Full

Supported pause frame use: No

Supports auto-negotiation: No

Advertised link modes: 10000baseT/Full

Advertised pause frame use: Symmetric

Advertised auto-negotiation: No

Speed: Unknown!

Duplex: Unknown! (255)

Port: FIBRE

PHYAD: 0

Transceiver: external

Auto-negotiation: off

Supports Wake-on: d

Wake-on: d

Current message level: 0x00000007 (7)

drv probe link

Link detected: no

mailto:root@socfpga:/usr/fsim root@socfpga:/usr/fsim# ethtool -i eth2

driver: ixgbe

version: 4.3.15

firmware-version: 0x80000707

bus-info: 0000:01:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: no

mailto:root@socfpga:/usr/fsim root@socfpga:/usr/fsim#

I doubt that " Supports auto-negotiation: No" is a reason for repeating link up and down.

Could you tell a next step to resolve ?

BR,

Taira

[idata]

Hi Taira,

Thank you for the update. Let me further check on this issue.rgds,wb

[idata]

Hi Taira,

Checking on the fiber module used, FINISAR FTLX8573D3BTL is not a supported fiber module for Intel X520 network adapter. Here is the website for reference: http://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000005688.html If you are using Intel X520 server adapter, you need to check on the supported fiber module. If not, then please confirm the following information:1) Are you using an onboard NIC? If yes, what is the brand and model of your system?2) If it is onboard NIC, have you contacted the system vendor to check with them?3) With regards to the autonegotiation, as stated on our README.txt https://downloadmirror.intel.com/13663/eng/README.txt"An Intel(R) Network Adapter using fiber-based connections, however, will notattempt to auto-negotiate with its link partner since those adapters operateonly in full duplex and only at their native speed."4) Have you try testing on another switch to isolate the issue? Please feel free to update me.Thanks,wb

[TSuzu6]

Hi wb

I answer your questions.

1) Are you using an onboard NIC? If yes, what is the brand and model of your system?

No, I use 82599EN for our PCIe exntended board. This PCI extended board is connected to mother board via PCI slot.

2) If it is onboard NIC, have you contacted the system vendor to check with them?

It is not onboard NIC.

3) With regards to the autonegotiation, as stated on our README.txt

I understand that fiber-based connections can not be auto-negotiated.

4) Have you try testing on another switch to isolate the issue?

No, but I used a tester(SPIRENT C1). The SFP modules and cable which I use can be linked between tester port A and tester port B.

Regurding to SFP module, I refer the site which you direct.

According to the site, FTLX8571D3BCV-IT works with 82599. So I order FTLX8571D3BCV without -IT.

When I will get FTLX8571D3BCV, I will try and update soon.

BR,

Taira

[idata]

Hi Taira,

Thank you for the clarification. Please update me the test result of using FTLX8571D3BCV fiber module. Looking forward to your update.rgds,wb

[TSuzu6]

Hi! wb

I tried FTLX8571D3BCV SFP module. But the issue can not be solved.

Same messages appear on linux console.

If you have another method for effecive debug, Please tell.

BR,

Taira

[idata]

Hi Taira,

Thank you for the update, can you provide me the serial number of your X520 server adapter? You can locate the serial number on the network adapter's white sticker (NOT on the box).Please provide me the number that has the format as follow:12 alphanumeric + 5 digits + 6 digits -3 digits Thanks,wb

[TSuzu6]

Hi! wb

No, We use 82599EN on the board which is designed by ourself.

This board is connected to mother board via PCI-e.

ixgbe driver runs on Linux with ALTERA FPGA(ArriaV SoC).

We performed ethtool -t eth2. The result is below.

The test result is FAIL

The test extra info:

Register test (offline) 0

Eeprom test (offline) 0

Interrupt test (offline) 0

Loopback test (offline) 0

Link test (on/offline) 1

Link test is failed. What should we do for preventing to fail link test ?

BR,

Taira

[idata]

Hi Taira,

Thank you for the clarification. As this is an ethernet controller used on the board designed by your company, which is a setup involves third party board. Please contact your Intel Field Application engineer for the support or contact the point of purchase .Thanks,wb

[TSuzu6]

Hi wb

Sorry for generating another thread.

I contacted to Inetl Japan and local distributor.

But they can not supprt my issue. Because they do not have experience to support a device which is implemented on non-Intel board.

So I wish you to help my issue.

I constructed test enviroment for 1Gb ether link as below,

*[82599EN]-->[FNISAR FTLX8571D3BCV(1G/10G)]-->[Fiber cable]-->[FNISAR FTLX8571D3BCV(1G/10G)]-->[Tester(SPIRENT)]

82599EN can link up on this enviroment.

And next,I constructed test enviroment for 10Gb ether link as below,

*[82599EN]-->[FNISAR FTLX8573D3BTL(10G)]-->[Fiber cable]-->[FNISAR FTLX8573D3BTL(10G)]-->[Tester(SPIRENT)]

The issue(link up/down) is occured on this enviroment.

[Detail]

When I run an ixgbe dirver and enter "ifconfig eth2 up", link up and down are displaied repeatly in linux console.

Then, I read the Link Status Register — LINKS (0x042A4; RO).

read value is 0x79081113=0111_1001_0000_1000_0001_0001_0001_0011.

I organize above value to the register field of LINKS.

[11:8]KX4_SIG_DET 0001b

Signal Detect of 10 GbE Parallel (KX4, CX4 or XAUI).Bit[11, 10, 9, 8] shows lane <3,2,1,0> status, respectively.

For each bit:0b = A signal is not present (failed). 1b = A signal is present (good).

[12]KR_SIG_DET 1b

Signal Detect of 10 GbE serial (KR or SFI) success.

[17]10G Align Status 0b

10 GbE Align_status failed (deskew process not complete).

I change SFP module to FINISAR FTLX8571D3BCV. FTLX8571D3BCV can work for 1GbE or 10GbE.

When I connect to 1GbE tester via this SFP module, 82599EN can keep a link up status.

If I should confirm the some registers, please tell.

Could you estimate causes which occur link up/down repeatly ?

BR,

Taira

[idata]

HiTaira,

Thank you for the update and taking time to check. Let me further check for you.rgds,wb

[TSuzu6]

Hi wb

Thank you for your support continuously.

I notice that the register "LINKS" has strange value.

bit 30 "Link up" is 1b which means "Link is up."

bit 7 "Link Status is 0b which means "Link is currently down or link was down since last time read."

I read register value as below, when the issue was occured.

reg[00004200] = 80000008

reg[00004208] = 0204000e

reg[0000420c] = 0000000e

reg[00004218] = 000001a0

reg[0000421c] = 00000000

reg[00004220] = 00000000

reg[00004224] = 00000000

reg[00004240] = 08002fff

reg[00004244] = 00000381

reg[00004248] = 0000ffff

reg[0000425c] = 10000000

reg[00004260] = 00000000

reg[00004268] = 05ee0000

reg[00004288] = 00000080

reg[0000428c] = 00008400

reg[00004290] = 04000000

reg[00004298] = 1b1b0000

reg[0000429c] = 461f02f0

reg[000042a0] = c09c6084

reg[000042a4] = 79081113

reg[00004324] = 00000012

reg[000042a8] = 000a0000

reg[000042b0] = 00090000

reg[000042b4] = 00000000

reg[000042d0] = 00000000

reg[000042d4] = 00000000

reg[000042d8] = 00000000

reg[000042e0] = 10010000

reg[000042e4] = 822ffff8

reg[000042e8] = 00000000

reg[000042ec] = 00000000

reg[00014f00] = 0000006d

reg[00014f10] = 00000000

reg[00004294] = 0000000a

reg[00004314] = 00020000

reg[00004000] = 00000000

reg[00004004] = 00000000

reg[00004008] = 00000000

reg[00004034] = 00000013

reg[00004038] = 0010e891

reg[00004040] = 00000000

reg[00008780] = 00000000

reg[000041a4] = 00000000

reg[000041a8] = 00000000

reg[00004140] = 00000000

reg[00004144] = 00000000

reg[00004148] = 00000000

reg[0000414c] = 00000000

reg[00004150] = 00000000

reg[00004154] = 00000000

reg[00004158] = 00000000

reg[0000415c] = 00000000

reg[00004160] = 00000000

reg[00004164] = 00000000

reg[00004168] = 00000000

reg[0000416c] = 00000000

reg[00004170] = 00000000

reg[00004174] = 00000000

reg[00004178] = 00000000

reg[0000417c] = 00000000

reg[0000405c] = 00000000

reg[00004060] = 00000000

reg[00004064] = 00000000

reg[00004068] = 00000000

reg[0000406c] = 00000000

reg[00004070] = 00000000

reg[00004078] = 00000000

reg[0000407c] = 00000000

reg[00004074] = 00000000

reg[00004088] = 00000000

reg[0000408c] = 00000000

reg[000041b0] = 00000000

reg[000041b4] = 00000000

reg[000041b8] = 00000000

reg[00002f50] = 00000000

reg[00002f54] = 00000000

reg[00002f58] = 00000000

reg[00002f5c] = 00000000

reg[00002f60] = 00000000

reg[00002f64] = 00000000

reg[00002f68] = 00000000

reg[00002f6c] = 00000000

reg[00002f70] = 00000000

reg[00002f74] = 00000000

reg[00002f78] = 00000000

reg[00002f7c] = 00000000

reg[00004080] = 00000000

reg[00004090] = 00000000

reg[00004094] = 00000000

reg[000087a0] = 00000000

reg[000087a4] = 00000000

reg[000087a8] = 00000000

reg[000040a4] = 00000000

reg[000040a8] = 00000000

reg[000040ac] = 00000000

reg[000040b0] = 00000000

reg[000040b4] = 00000000

reg[000040b8] = 00000000

reg[000040c0] = 00000000

reg[000040c4] = 00000000

reg[000040d0] = 00000000

reg[000040d4] = 00000000

reg[000040d8] = 00000000

reg[000040dc] = 00000000

reg[000040e0] = 00000000

reg[000040e4] = 00000000

reg[000040e8] = 00000000

reg[000040ec] = 00000000

reg[000040f0] = 00000000

reg[000040f4] = 00000000

reg[00004010] = 00000000

reg[00004120] = 00000000

reg[00002300] = 00000000

reg[00002304] = 00000000

reg[00002308] = 00000000

reg[0000230c] = 00000000

reg[00002310] = 00000000

reg[00002314] = 00000000

reg[00002318] = 00000000

reg[0000231c] = 00000000

reg[00002320] = 00000000

reg[00002324] = 00000000

reg[00002328] = 00000000

reg[0000232c] = 00000000

reg[00002330] = 00000000

reg[00002334] = 00000000

reg[00002338] = 00000000

reg[0000233c] = 00000000

reg[00002340] = 00000000

reg[00002344] = 00000000

reg[00002348] = 00000000

reg[0000234c] = 00000000

reg[00002350] = 00000000

reg[00002354] = 00000000

reg[00002358] = 00000000

reg[0000235c] = 00000000

reg[00002360] = 00000000

reg[00002364] = 00000000

reg[00002368] = 00000000

reg[0000236c] = 00000000

reg[00002f40] = 00000000

reg[00005118] = 00000000

reg[0000241c] = 00000000

reg[00002424] = 00000000

reg[00002428] = 00000000

reg[0000242c] = 00000000

reg[00008784] = 00000000

reg[00008788] = 00000000

I hope that above result is useful for yur analysis.

BR,

Taira

[idata]

Hi Taira ,

Just want to clarify, I noticed that you mentioend. "I change SFP module to FINISAR FTLX8571D3BCV. FTLX8571D3BCV can work for 1GbE or 10GbE. When I connect to 1GbE tester via this SFP module, 82599EN can keep a link up status. " Are you saying that you are able to keep the link up status with FINISAR FTLX8271D38CV? If yes, then please continue used the supported fiber modules. You may also tried newer driver package 4.4.6 available at sourcefore https://sourceforge.net/projects/e1000/files/ixgbe%20stable/ Thanks,wb

[TSuzu6]

Hi wb

Thank you for your response.

Yes, I keep the link up status with FINISAR FTLX8271D38CV on 1Gbps speed.

I suspected the reference clock because the link up status could be kept on1Gbps.

Then, I measured a reference clock and found the cause for this issue.

The reference clock had a noise. So I replaced clock device near a 82599EN.

I connect to SPIRENT C1 tester after this. And I confirmed that the link up status is kept on 10Gbps.

The issue is solved.The cause is noisy reference clock.

Thank you for your analysis and a lot of advice.

BR,

Taira.

[idata]

Hi Taira,

Thank you for sharing your test result. I am glad the issue is fixed and we will take note of this useful information.rgds,wb