Unable to establish link with Intel 82599ES 10-Gigabit SFI/SFP+ Network Connection

JMars10 · ‎02-17-2015

Hi,

I'm looking for help establishing (and troubleshooting) a link to a SPAN in promiscuous mode on Ubuntu Server 14.04.

This link was up and running and seeing packets for a few weeks. To help with an application issue, we installed the PF_RING kernel module, and after a server reboot, we have not been able to establish a link. We've rolled back our changes and removed PF_RING, but we are still not able to establish a link, and I'm looking for some help.

We've looked at our Gigamon SPAN and it appears to be sending data to our device but we have not been able to see any traffic ('tcpdump -i em1 -vv' does not see any traffic).

Any help would be greatly appreciated.

Jeff

$ifconfig em1

em1 Link encap:Ethernet HWaddr ec:f4:bb:**:**:**

UP BROADCAST NOARP PROMISC MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:7180 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

$ sudo ethtool em1

Settings for em1:

Supported ports: [ FIBRE ]

Supported link modes: 1000baseT/Full

10000baseT/Full

Supported pause frame use: No

Supports auto-negotiation: Yes

Advertised link modes: 1000baseT/Full

10000baseT/Full

Advertised pause frame use: No

Advertised auto-negotiation: Yes

Speed: Unknown!

Duplex: Unknown! (255)

Port: FIBRE

PHYAD: 0

Transceiver: external

Auto-negotiation: on

Supports Wake-on: umbg

Wake-on: g

Current message level: 0x00000007 (7)

drv probe link

Link detected: no

$ cat /var/log/syslog | grep ixgbe

Feb 17 15:16:46 servername kernel: [ 7.371724] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.15.1-k

$ cat /var/log/syslog | grep em1

Feb 17 15:16:46 servername kernel: [ 11.789614] systemd-udevd[572]: renamed network interface eth0 to em1

Feb 17 15:16:46 servername kernel: [ 14.071614] IPv6: ADDRCONF(NETDEV_UP): em1: link is not ready

Feb 17 15:16:46 servername kernel: [ 15.404429] ixgbe 0000:01:00.0: registered PHC device on em1

Feb 17 15:16:46 servername kernel: [ 15.509280] IPv6: ADDRCONF(NETDEV_UP): em1: link is not ready

Feb 17 15:16:47 servername kernel: [ 15.580413] ixgbe 0000:01:00.0 em1: detected SFP+: 6

Feb 17 15:16:48 servername kernel: [ 16.713425] device em1 entered promiscuous mode

Feb 17 15:16:48 servername kernel: [ 17.498677] ixgbe 0000:01:00.0 em1: detected SFP+: 6

$sudo ethtool -m em1

Identifier : 0x03 (SFP)

Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)

Connector : 0x07 (LC)<...

st4 · ‎03-01-2015

HI Jmarshall80,

What is the exact 10 Gigabit NIC used here? Is it a X520series ? and what is the exact model?

This will enable us to further assist on this.

thanks,

wb

JMars10 · ‎03-02-2015

Hi,

This is a "Intel X520 DP 10Gb DA/SFP+, + I350 DP 1Gb Ethernet, Network Daughter Card"

Thanks,

Jeff

st4 · ‎03-02-2015

Hi jmarshall80,

thanks for the info, will check on this.

rgds,

wb

st4 · ‎03-24-2015

Hi jmarshall80,

Please help share the ff info:

1) output of ethtool -t

2. The setup script for further investigation on why the link fails after installing PF_ring

3) Use 10Gbps connection instead of 1Gbps

4) Based on your log info, you are using Finisar ftlx8571d3bcv which is not a compatible SFP+ module

(http://www.intel.com/support/network/adapter/pro100/sb/CS-030612.htm http://www.intel.com/support/network/adapter/pro100/sb/CS-030612.htm)

Thanks,

wb

PSami · ‎12-22-2016

Hi,

We are experiencing similar issue when deploying a production system. We have installed PF_RING with ZC prior to cabling and ethtool says "No link detected" after connecting fiber. Uninstalling PF_RING+ZC and updating ixgbe driver didn't help to resolve the issue.

Linux 4.4.0-57-generic x86_64

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=16.04

05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter

Kernel driver in use: ixgbe

Kernel modules: ixgbe, ixgbe_zc

05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter

Kernel driver in use: ixgbe

Kernel modules: ixgbe, ixgbe_zc

08:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter

Kernel driver in use: ixgbe

Kernel modules: ixgbe, ixgbe_zc

08:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter

Kernel driver in use: ixgbe

Kernel modules: ixgbe, ixgbe_zc

root@host1:~# ethtool -i ens1f1

driver: ixgbe

version: 4.2.1-k

firmware-version: 0x80000897

expansion-rom-version:

bus-info: 0000:05:00.1

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: no

root@host1:~# ethtool ens1f1

Settings for ens1f1:

Supported ports: [ FIBRE ]

Supported link modes: 10000baseT/Full

Supported pause frame use: No

Supports auto-negotiation: No

Advertised link modes: 10000baseT/Full

Advertised pause frame use: No

Advertised auto-negotiation: No

Speed: Unknown!

Duplex: Unknown! (255)

Port: FIBRE

PHYAD: 0

Transceiver: external

Auto-negotiation: off

Supports Wake-on: d

Wake-on: d

Current message level: 0x00000007 (7)

drv probe link

Link detected: no

root@host1:~# ethtool -t ens1f1

The test result is FAIL

The test extra info:

Register test (offline) 0

Eeprom test (offline) 0

Interrupt test (offline) 0

Loopback test (offline) 0

Link test (on/offline) 1

Dec 22 17:14:03 kernel: [ 13.404410] IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready

Dec 22 17:14:03 kernel: [ 13.741596] cgroup: new mount options do not match the existing superblock, will be ignored

Dec 22 17:14:06 kernel: [ 16.917021] tg3 0000:02:00.0 eno1: Link is up at 1000 Mbps, full duplex

Dec 22 17:14:06 kernel: [ 16.917035] tg3 0000:02:00.0 eno1: Flow control is off for TX and off for RX

Dec 22 17:14:06 kernel: [ 16.917037] tg3 0000:02:00.0 eno1: EEE is enabled

Dec 22 17:14:06 kernel: [ 16.917073] IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready

Dec 22 17:21:09 kernel: [ 439.915859] ixgbe 0000:05:00.0: registered PHC device on ens1f0

Dec 22 17:21:09 kernel: [ 440.019386] IPv6: ADDRCONF(NETDEV_UP): ens1f0: link is not ready

Dec 22 17:21:09 kernel: [ 440.083870] ixgbe 0000:05:00.0 ens1f0: detected SFP+: 6

Dec 22 17:22:04 kernel: [ 494.852207] ixgbe 0000:05:00.1: registered PHC device on ens1f1

Dec 22 17:22:04 kernel: [ 494.953737] IPv6: ADDRCONF(NETDEV_UP): ens1f1: link is not ready

Dec 22 17:22:04 kernel: [ 495.018225] ixgbe 0000:05:00.1 ens1f1: detected SFP+: 5

Dec 22 17:24:32 kernel: [ 642.245720] ixgbe 0000:08:00.0: registered PHC device on ens3f0

Dec 22 17:24:32 kernel: [ 642.348922] IPv6: ADDRCONF(NETDEV_UP): ens3f0: link is not ready

Dec 22 17:24:32 kernel: [ 642.413442] ixgbe 0000:08:00.0 ens3f0: detected SFP+: 6

Dec 22 17:26:18 kernel: [ 748.661562] ixgbe 0000:08:00.1: registered PHC device on ens3f1

Dec 22 17:26:18 kernel: [ 748.766107] IPv6: ADDRCONF(NETDEV_UP): ens3f1: link is not ready

Dec 22 17:26:18 kernel: [ 748.830408] ixgbe 0000:08:00.1 ens3f1: detected SFP+: 5

Dec 22 22:19:08 kernel: [18319.317635] ixgbe 0000:05:00.1: removed PHC on ens1f1

Dec 22 22:19:12 kernel: [18323.695150] ixgbe 0000:05:00.1: registered PHC device on ens1f1

Dec 22 22:19:12 kernel: [18323.798353] IPv6: ADDRCONF(NETDEV_UP): ens1f1: link is not ready

Dec 22 22:19:12 kernel: [18323.864169] ixgbe 0000:05:00.1 ens1f1: detected SFP+: 5

Dec 22 22:20:24 kernel: [18395.349270] ixgbe 0000:05:00.1: removed PHC on ens1f1

Dec 22 22:20:28 kernel: [18399.726247] ixgbe 0000:05:00.1: registered PHC device on ens1f1

Dec 22 22:20:28 kernel: [18399.830017] IPv6: ADDRCONF(NETDEV_UP): ens1f1: link is not ready

Dec 22 22:20:28 kernel: [18399.894342] ixgbe 0000:05:00.1 ens1f1: detected SFP+: 5

Can you please assist to resolve this issue?

Thanks.

PSami · ‎12-22-2016

ethtool -m ens1f1

Identifier : 0x03 (SFP)

Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)

Connector : 0x07 (LC)

Transceiver codes : 0x10 0x00 0x00 0x00 0x00 0x00 0x00 0x00

Transceiver type : 10G Ethernet: 10G Base-SR

Encoding : 0x06 (64B/66B)

BR, Nominal : 10300MBd

Rate identifier : 0x00 (unspecified)

Length (SMF,km) : 0km

Length (SMF) : 0m

Length (50um) : 80m

Length (62.5um) : 30m

Length (Copper) : 0m

Length (OM3) : 300m

Laser wavelength : 850nm

Vendor name : FINISAR CORP.

Vendor OUI : 00:90:65

Vendor PN : FTLX8571D3BCL-HP

Vendor rev : A

Option values : 0x00 0x1a

Option : RX_LOS implemented

Option : TX_FAULT implemented

Option : TX_DISABLE implemented

BR margin, max : 0%

BR margin, min : 0%

Vendor SN : MW10G56

Date code : 16070928

Optical diagnostics support : Yes

Laser bias current : 8.268 mA

Laser output power : 0.6088 mW / -2.16 dBm

Receiver signal average optical power : 0.0008 mW / -30.97 dBm

Module temperature : 43.75 degrees C / 110.76 degrees F

Module voltage : 3.3046 V

Alarm/warning flags implemented : Yes

Laser bias current high alarm : Off

Laser bias current low alarm : Off

Laser bias current high warning : Off

Laser bias current low warning : Off

Laser output power high alarm : Off

Laser output power low alarm : Off

Laser output power high warning : Off

Laser output power low warning : Off

Module temperature high alarm : Off

Module temperature low alarm : Off

Module temperature high warning : Off

Module temperature low warning : Off

Module voltage high alarm : Off

Module voltage low alarm : Off

Module voltage high warning : Off

Module voltage low warning : Off

Laser rx power high alarm : Off

Laser rx power low alarm : On

Laser rx power high warning : Off

Laser rx power low warning : On

Laser bias current high alarm threshold : 13.200 mA

Laser bias current low alarm threshold : 4.000 mA

Laser bias current high warning threshold : 12.600 mA

Laser bias current low warning threshold : 5.000 mA

Laser output power high alarm threshold : 1.0000 mW / 0.00 dBm

Laser output power low alarm threshold : 0.2512 mW / -6.00 dBm

Laser output power high warning threshold : 0.7943 mW / -1.00 dBm

Laser output power low warning threshold : 0.3162 mW / -5.00 dBm

Module temperature high alarm threshold : 78.00 degrees C / 172.40 degrees F

Module temperature low alarm threshold : -13.00 degrees C / 8.60 degrees F

Module temperature high warning threshold : 73.00 degrees C / 163.40 degrees F

Module temperature low warning threshold : -8.00 degrees C / 17.60 degrees F

Module voltage high alarm threshold : 3.7000 V

Module voltage low alarm threshold : 2.9000 V

Module voltage high warning threshold : 3.6000 V

Module voltage low warning threshold : 3.0000 V

Laser rx power high alarm threshold : 1.0000 mW / 0.00 dBm

Laser rx power low alarm threshold : 0.0100 mW / -20.00 dBm

Laser rx power high warning threshold : 0.7943 mW / -1.00 dBm

Laser rx power low warning threshold : 0.0158 mW / -18.01 dBm