x520-da2 not working with SFP+ twinax cable.

[DSloa2]

Hi all,

I have 2 servers Windows 2008 R2 x64 that I have just installed X520-DA2 cards. The moment I attach the cable the network adapter disappears from the network connections screen and in the device manager it displays a message that 'Windows has stopped this device because it has reported problems.'

The System event log records an error "The driver was unable to load due to an unsupported SFP+ module installed in the adapter." which seems to indicate that I am using an incompatable cable. However I have reviewed the information at http://www.intel.com/support/network/adapter/pro100/sb/CS-030612.htm http://www.intel.com/support/network/adapter/pro100/sb/CS-030612.htm and it indicates that :

Which direct attach cables are compatible with the Intel Ethernet Server Adapter X520 Series?  

The Intel Ethernet Server Adapter X520 Series and Intel® 82599 10 Gigabit Ethernet Controller-based connections support any SFP+ passive or active limiting direct attach copper cable that complies with the SFF-8431 v4.1 and SFF-8472 v10.4 specifications.

The cable I am using is a Tripp-Lite N280-05M which they list on their website as being SFF-8431 http://www.tripplite.com/en/products/model.cfm?txtModelID=4904 http://www.tripplite.com/en/products/model.cfm?txtModelID=4904

I have found several other posts here that appear to be similar or identical; however, the posters never posted their solution or perhaps they simply gave up.

The other posts are here:

http://www.intel.com/content/www/us/en/search.html?keyword=The+driver+was+unable+to+load+due+to+an+unsupported+SFP%2B+module+installed+in+the+adapter http://www.intel.com/content/www/us/en/search.html?keyword=The+driver+was+unable+to+load+due+to+an+unsupported+SFP%2B+module+installed+in+the+adapter.

I hope I am not alone on this one! TIA for any help.

Dave

[idata]

Hi,

have now a second box with only a X520-da2 card and the mentioned cable, this works without

any drop. Even with the newst driver, this x520-da2 card seams to be incompatbile with a Intel® 10 Gigabit CX4 Dual Port Server Adapter (82598EB)

in the same box.

Any idee to solve it?

Henri

[Mark_H_Intel]

I do not know a solution, but I have some thoughts that I hope will be helpful.

I doubt that the issue is an incompatibility between the two adapters. The first thought I had was that some lack of resources in the server is causing the drops. What messages do you get in the system log? Are there any complaints about lack of resources? Do you see any messages of any type that happen when the connection drops?

An interresting experiment would be to remove the two CX4 adapters and place the DA adapters in the same slots in the same system where you are getting the failure.If they work without a drop, then the original issue is more likely a resource issue rather than incompatibility between the adapters. I understand if you cannot do the above experiment.

How much system memory is installed in the system with the four adapters?

You could try disabling I/O map mode, but I am not sure if this will help you. See http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=4237 http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=4237 for a download of the software. You will need to extract the files to get to the IOUTIL tool. The download includes tools for Windows, Linux, and EFI, but you might need to put this download on a Windows machine to extract the files. You can use the same utility to enable I/O map mode if disabling it causes something else not to work.

Mark H

[idata]

Hi Mark,

after a lot trial and errors I got it now working by replacing the x520-da2 card with a x520-sr2 card. This works as expected.

Can you please explain it?

Thanks

Henri

[Mark_H_Intel]

Are you using optical modules and cables or DA cables with the SR2 adapter? The DA2 and SR2 adapters are the same except for the SR optics that are pre-installed. If you are using optical modules and cables, then I would make a guess that some difference in the communication with the switch using optical cables is responsible for keeping the connection for dropping. If you are still using the same Direct Attach cables, then I do not even have a guess.

Mark H

[idata]

Hi Mark,

have now replaced the X520-DA2 card with a X520-SR2 card, Port1:Intel GBIC, Port2: SFP+ DA cable.

I have now 2 boxes with same configuration (motherboard and all PCI-e cards).

For any reason I get now a lot of "port A2 is now off-line/online" messages. With the X520-DA2 card I was unable to get the interface online.

To make sure that it's not cable/switch port related,I exchanged the cable a couple of times between both boxes.

Whereever I put in the working SR2 card, it works without any issues/messages (since 3-4 weeks).

But here is a difference, even OS and driver are at the same level.

Can you explain where to set the RxQ and TxQ values?

Thanks a lot in advance

Henri

S200:

[ 34.439526] Intel(R) 10 Gigabit PCI Express Network Driver - version 3.7.14-NAPI

[ 34.613628] ixgbe 0000:04:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18

[ 34.613709] ixgbe 0000:04:00.0: setting latency timer to 64

[ 34.804487] ixgbe 0000:04:00.0: (PCI Express:5.0GT/s:Width x8) 00:1b:21:d8:40:ec

[ 34.804569] ixgbe 0000:04:00.0: eth2: MAC: 2, PHY: 15, SFP+: 5, PBA No: E68785-005

[ 34.804572] ixgbe 0000:04:00.0: eth2: Enabled Features: RxQ: 8 TxQ: 8 FdirHash RSS RSC

[ 34.804590] ixgbe 0000:04:00.0: eth2: Intel(R) 10 Gigabit Network Connection

[ 34.804666] ixgbe 0000:04:00.1: PCI INT B -> GSI 19 (level, low) -> IRQ 19

[ 34.804779] ixgbe 0000:04:00.1: setting latency timer to 64

[ 36.024317] ixgbe 0000:04:00.1: (PCI Express:5.0GT/s:Width x8) 00:1b:21:d8:40:ed

[ 36.024412] ixgbe 0000:04:00.1: eth3: MAC: 2, PHY: 1, PBA No: E68785-005

[ 36.024414] ixgbe 0000:04:00.1: eth3: Enabled Features: RxQ: 8 TxQ: 8 FdirHash RSS RSC

[ 36.024433] ixgbe 0000:04:00.1: eth3: Intel(R) 10 Gigabit Network Connection

[ 36.024470] ixgbe 0000:05:00.0: PCI INT B -> GSI 17 (level, low) -> IRQ 17

[ 36.024579] ixgbe 0000:05:00.0: setting latency timer to 64

[ 36.110945] ixgbe 0000:05:00.0: (PCI Express:2.5GT/s:Width x8) 00:1b:21:a4:aa:4d

[ 36.111028] ixgbe 0000:05:00.0: eth4: MAC: 1, PHY: 0, PBA No: E37623-004

[ 36.111030] ixgbe 0000:05:00.0: eth4: Enabled Features: RxQ: 8 TxQ: 8 RSS LRO

[ 36.111032] ixgbe 0000:05:00.0: eth4: Intel(R) 10 Gigabit Network Connection

[ 36.111067] ixgbe 0000:05:00.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16

[ 36.111175] ixgbe 0000:05:00.1: setting latency timer to 64

[ 36.216595] ixgbe 0000:05:00.1: (PCI Express:2.5GT/s:Width x8) 00:1b:21:a4:aa:4c

[ 36.216678] ixgbe 0000:05:00.1: eth5: MAC: 1, PHY: 0, PBA No: E37623-004

[ 36.216680] ixgbe 0000:05:00.1: eth5: Enabled Features: RxQ: 8 TxQ: 8 RSS LRO

[ 36.216682] ixgbe 0000:05:00.1: eth5: Intel(R) 10 Gigabit Network Connection

200:

[ 32.518449] Intel(R) 10 Gigabit PCI Express Network Driver - version 3.7.14-NAPI

32.586487] ixgbe 0000:04:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18

[ 32.586551] ixgbe 0000:04:00.0: setting latency timer to 64

[ 32.804121] ixgbe 0000:04:00.0: (PCI Express:5.0GT/s:Width x8) 00:1b:21:d8:3d:c0

[ 32.804201] ixgbe 0000:04:00.0: eth2: MAC: 2, PHY: 15, SFP+: 5, PBA No: E68785-005

[ 32.804203] ixgbe 0000:04:00.0: eth2: Enabled Features: RxQ: 4 TxQ: 4 FdirHash RSS RSC

[ 32.804221] ixgbe 0000:04:00.0: eth2: Intel(R) 10 Gigabit Network Connection

32.804295] ixgbe 0000:04:00.1: PCI INT B -> GSI 19 (level, low) -> IRQ 19

[ 32.804390] ixgbe 0000:04:00.1: setting latency timer to 64

[ 34.024001] ixgbe 0000:04:00.1: (PCI Express:5.0GT/s:Width x8) 00:1b:21:d8:3d:c1

[ 34.024081] ixgbe 0000:04:00.1: eth3: MAC: 2, PHY: 1, PBA No: E68785-005

[ 34.024083] ixgbe 0000:04:00.1: eth3: Enabled Features: RxQ: 4 TxQ: 4 FdirHash RSS RSC

[ 34.024101] ixgbe 0000:04:00.1: eth3: Intel(R) 10 Gigabit Network Connection

[ 34.024128] ixgbe 0000:05:00.0: PCI INT B -> GSI 17 (level, low) -> IRQ 17

[ 34.024224] ixgbe 0000:05:00.0: setting latency timer to 64

[ 34.116392] ixgbe 0000:05:00.0: (PCI Express:2.5GT/s:Width x8) 00:1b:21:9d:20:c1

[ 34.116473] ixgbe 0000:05:00.0: eth4: MAC: 1, PHY: 0, PBA No: E37623-004

[ 34.116475] ixgbe 0000:05:00.0: eth4: Enabled Features: RxQ: 4 TxQ: 4 RSS LRO

[ 34.116476] ixgbe 0000:05:00.0: eth4: Intel(R) 10 Gigabit Network Connection

[ 34.116504] ixgbe 0000:05:00.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16

[ 34.116603] ixgbe 0000:05:00.1: setting latency timer to 64

[ 34.226401] ixgbe 0000:05:00.1: (PCI Express:2.5GT/s:Width x8) 00:1b:21:9d:20:c0

[ 34.226483] ixgbe 0000:05:00.1: eth5: MAC: 1, PHY: 0, PBA No: E37623-004

[ 34.226485] ixgbe 0000:05:00.1: eth5: Enabled Features: RxQ: 4 TxQ: 4 RSS LRO

[ 34.226486] ixgbe 0000:05:00.1: eth5: Intel(R) 10 Gigabit Network Connection

I 01/13/12 17:29:59 00435 ports: port A2 is Blocked by STPI 01/13/12 17:29:59 00076 ports: port A2 is now on-lineI 01/13/12 17:30:04 00077 ports: port A2 is now off-lineI 01/13/12 17:30:05 00435 ports: port A2 is Blocked by STPI 01/13/12 17:30:05 00076 ports: port A2 is now on-lineI 01/13/12 17:30:10 00077 ports: port A2 is now off-lineI 01/13/12 17:30:10 00435 ports: port A2 is Blocked by STPI 01/13/12 17:30:10 00076 ports: port A2 is now on-lineI 01/13/12 17:30:13 00077 ports: port A2 is now off-line

[Mark_H_Intel]

Hi Henri,

I was looking at your messages concerning port A2. I think this one message might explain the issue. I 01/13/12 17:29:59 00435 ports: port A2 is Blocked by STP

The messages you included here show that STP is enabled on the switch and is blocking traffic on the port so that you don't have a network loop. If you are using channel bonding on the ports of you server, make sure you have the appropriate configuration to match on the switch.

I got a little help from one of our Linux developers for the answer on the transmit and receive queues. Thanks Don.

We automatically allocate queues depending on system and OS capabilities. Is the processor different between the two systems? Do we have a different number of processor cores available between the two systems? Does one system support Hyper-Threading Technology, but the other one does not? Configuration differences can also change the number of queues allocated. For example if Data Center Bridging or Flow Director is enabled, then the number of queues might change.

I hope this helps.

Mark H

[idata]

Hi Mark,

I also created a record at HP, STP for this ports is switched off. The message occures in any case when links are terminated (e.g. shutdown server).

And yes the CPUs are different. After I set RSS=8 I got this messages, here are the values a bit inconsistent!

Is there any way to check the firmware/revision to be sure that the cards running with the same firmware?

It's curious, no X520-DA2 card works, one X520-SR2 card works partly and one X520-SR2 card works perfect, regardsless of the

box where I put it in.

Both SR2 cards are label with: E10G42BFSR, 900137, CPU E69818(B).

Thanks

Henri

[ 32.495283] ixgbe 0000:04:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18[ 32.495342] ixgbe 0000:04:00.0: setting latency timer to 64[ 32.690460] ixgbe: Receive-Side Scaling (RSS) set to 8[ 32.690462] ixgbe: Flow Director filtering mode set to 0[ 32.690463] ixgbe: 0000:04:00.0: ixgbe_check_options: Flow Director disabled[ 32.714140] ixgbe 0000:04:00.0: (PCI Express:5.0GT/s:Width x8) 00:1b:21:d8:3d:c0[ 32.714220] ixgbe 0000:04:00.0: eth2: MAC: 2, PHY: 15, SFP+: 5, PBA No: E68785-005[ 32.714222] ixgbe 0000:04:00.0: eth2: Enabled Features: RxQ: 8 TxQ: 8 RSS RSC [ 32.714239] ixgbe 0000:04:00.0: eth2: Intel(R) 10 Gigabit Network Connection[ 32.714311] ixgbe 0000:04:00.1: PCI INT B -> GSI 19 (level, low) -> IRQ 19[ 32.714395] ixgbe 0000:04:00.1: setting latency timer to 64[ 33.933881] ixgbe 0000:04:00.1: (PCI Express:5.0GT/s:Width x8) 00:1b:21:d8:3d:c1[ 33.933961] ixgbe 0000:04:00.1: eth3: MAC: 2, PHY: 1, PBA No: E68785-005[ 33.933963] ixgbe 0000:04:00.1: eth3: Enabled Features: RxQ: 4 TxQ: 4 FdirHash RSS RSC [ 33.933980] ixgbe 0000:04:00.1: eth3: Intel(R) 10 Gigabit Network Connection[ 33.934009] ixgbe 0000:05:00.0: PCI INT B -> GSI 17 (level, low) -> IRQ 17[ 33.934106] ixgbe 0000:05:00.0: setting latency timer to 64[ 34.016449] ixgbe 0000:05:00.0: (PCI Express:2.5GT/s:Width x8) 00:1b:21:9d:20:c1[ 34.016530] ixgbe 0000:05:00.0: eth4: MAC: 1, PHY: 0, PBA No: E37623-004[ 34.016532] ixgbe 0000:05:00.0: eth4: Enabled Features: RxQ: 4 TxQ: 4 RSS LRO [ 34.016533] ixgbe 0000:05:00.0: eth4: Intel(R) 10 Gigabit Network Connection[ 34.016560] ixgbe 0000:05:00.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16[ 34.016656] ixgbe 0000:05:00.1: setting latency timer to 64[ 34.126411] ixgbe 0000:05:00.1: (PCI Express:2.5GT/s:Width x8) 00:1b:21:9d:20:c0[ 34.126493] ixgbe 0000:05:00.1: eth5: MAC: 1, PHY: 0, PBA No: E37623-004[ 34.126494] ixgbe 0000:05:00.1: eth5: Enabled Features: RxQ: 4 TxQ: 4 RSS LRO [ 34.126496] ixgbe 0000:05:00.1: eth5: Intel(R) 10 Gigabit Network Connection

here are the spanning tree informations:

Multiple Spanning Tree (MST) Information STP Enabled : Yes Force Version : MSTP-operation IST Mapped VLANs : 1-4094 Switch MAC Address : Switch Priority : 4096 Max Age : 20 Max Hops : 20 Forward Delay : 15 Topology Change Count : 1 Time Since Last Change : 47 hours CST Root MAC Address : CST Root Priority : 4096 CST Root Path Cost : 0 CST Root Port : This switch is root IST Regional Root MAC Address : IST Regional Root Priority : 4096 IST Regional Root Path Cost : 0 IST Remaining Hops : 20 Root Guard Ports : Loop Guard Ports : TCN Guard Ports : BPDU Protected Ports : A1-A2 BPDU Filtered Ports : A1-A2 PVST Protected Ports : PVST Filtered Ports : A1-A2 | Prio | Designated Hello Port Type | Cost rity State | Bridge Time PtP Edge ------ --------- + --------- ---- ------------ + ------------- ---- --- ---- A1 SFP+DA15 | 2000 128 Forwarding | 2 Yes Yes A2 SFP+DA15 | 2000 128 Forwarding | 2 Yes Yes A3 10GbE-CX4 | 2000 128 Forwarding | 2 Yes Yes A4 10GbE-CX4 | 2000 128 Forwarding | 2 Yes Yes

[idata]

Hi Mark,

here are some more informations:

as you can see we loose very 3-4 packet.

The switch reports a couple of "M 01/14/12 08:43:32 02672 FFI: port A2-Excessive link state transitions" errors ("Link-flap detected").

In the log of the Linux system I can see a lot of "NIC Link is Up/NIC Link is Down" messages, that's all.

There is a HP support document (http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=pt&cc=br&taskId=110&prodSeriesId=1827663&prodTypeId=12883&objectID=c02124172 http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=pt&cc=br&taskId=110&prodSeriesId=1827663&prodTypeId=12883&objectID=c02124172).

Enhancement (PR_0000017201) — The switch Fault Finder function has been extended to cover an improperly behaving fiber transceiver, or other condition which results in a link "flapping" rapidly between link-up and link-down states. A new fault event "link-flap" has been created to detect these events. Additionally, a new action, "warn-and-disable," has been created to report and disable the events. Together, these enhancements allow the errant condition to be detected, and the port in question optionally disabled.

Is there as verbose switch somewhere to report the reasons of the link UP/DOWNs of your adapter?

Please keep in mind that I already cross changed the cables (this means the transceivers) a couple of times.

Thanks

Henri

64 bytes from 172.20.10.22: icmp_seq=5570 ttl=64 time=0.135 ms64 bytes from 172.20.10.22: icmp_seq=5571 ttl=64 time=0.181 ms64 bytes from 172.20.10.22: icmp_seq=5572 ttl=64 time=0.194 ms64 bytes from 172.20.10.22: icmp_seq=5576 ttl=64 time=0.209 ms64 bytes from 172.20.10.22: icmp_seq=5577 ttl=64 time=0.211 ms64 bytes from 172.20.10.22: icmp_seq=5584 ttl=64 time=0.204 ms64 bytes from 172.20.10.22: icmp_seq=5585 ttl=64 time=0.258 ms64 bytes from 172.20.10.22: icmp_seq=5587 ttl=64 time=0.154 ms64 bytes from 172.20.10.22: icmp_seq=5591 ttl=64 time=0.244 ms64 bytes from 172.20.10.22: icmp_seq=5592 ttl=64 time=0.250 ms64 bytes from 172.20.10.22: icmp_seq=5596 ttl=64 time=0.255 ms64 bytes from 172.20.10.22: icmp_seq=5599 ttl=64 time=0.251 ms64 bytes from 172.20.10.22: icmp_seq=5600 ttl=64 time=0.213 ms64 bytes from 172.20.10.22: icmp_seq=5604 ttl=64 time=0.207 ms64 bytes from 172.20.10.22: icmp_seq=5605 ttl=64 time=0.221 ms64 bytes from 172.20.10.22: icmp_seq=5611 ttl=64 time=0.235 ms64 bytes from 172.20.10.22: icmp_seq=5612 ttl=64 time=0.209 ms64 bytes from 172.20.10.22: icmp_seq=5613 ttl=64 time=0.260 ms64 bytes from 172.20.10.22: icmp_seq=5616 ttl=64 time=0.233 ms64 bytes from 172.20.10.22: icmp_seq=5618 ttl=64 time=0.243 ms64 bytes from 172.20.10.22: icmp_seq=5619 ttl=64 time=0.245 ms64 bytes from 172.20.10.22: icmp_seq=5620 ttl=64 time=4.41 ms64 bytes from 172.20.10.22: icmp_seq=5621 ttl=64 time=0.173 ms64 bytes from 172.20.10.22: icmp_seq=5628 ttl=64 time=0.357 ms64 bytes from 172.20.10.22: icmp_seq=5630 ttl=64 time=0.288 ms64 bytes from 172.20.10.22: icmp_seq=5631 ttl=64 time=0.202 ms64 bytes from 172.20.10.22: icmp_seq=5632 ttl=64 time=0.168 ms64 bytes from 172.20.10.22: icmp_seq=5636 ttl=64 time=0.239 ms64 bytes from 172.20.10.22: icmp_seq=5637 ttl=64 time=0.249 ms64 bytes from 172.20.10.22: icmp_seq=5641 ttl=64 time=0.225 ms64 bytes from 172.20.10.22: icmp_seq=5643 ttl=64 time=0.258 ms64 bytes from 172.20.10.22: icmp_seq=5645 ttl=64 time=3.66 ms64 bytes from 172.20.10.22: icmp_seq=5650 ttl=64 time=0.237 ms64 bytes from 172.20.10.22: icmp_seq=5652 ttl=64 time=0.229 ms64 bytes from 172.20.10.22: icmp_seq=5659 ttl=64 time=0.377 ms64 bytes from 172.20.10.22: icmp_seq=5661 ttl=64 time=102 ms

[ 61.101372] ixgbe 0000:04:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 61.420117] ixgbe 0000:04:00.1: eth3: NIC Link is Down

[ 63.490197] ixgbe 0000:04:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 66.250741] ixgbe 0000:04:00.1: eth3: NIC Link is Down

[ 67.690199] ixgbe 0000:04:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 67.932606] ixgbe 0000:04:00.1: eth3: NIC Link is Down

[ 69.890199] ixgbe 0000:04:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 70.732108] ixgbe 0000:04:00.1: eth3: NIC Link is Down

[ 72.101383] ixgbe 0000:04:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 72.444681] ixgbe 0000:04:00.1: eth3: NIC Link is Down

[ 74.290199] ixgbe 0000:04:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 74.951676] ixgbe 0000:04:00.1: eth3: NIC Link is Down

[ 76.490195] ixgbe 0000:04:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 77.313971] ixgbe 0000:04:00.1: eth3: NIC Link is Down

[ 78.690195] ixgbe 0000:04:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 78.790193] ixgbe 0000:04:00.1: eth3: NIC Link is Down

[ 78.890194] ixgbe 0000:04:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 79.669477] ixgbe 0000:04:00.1: eth3: NIC Link is Down

[ 81.090195] ixgbe 0000:04:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 82.255178] ixgbe 0000:04:00.1: eth3: NIC Link is Down

[ 83.601370] ixgbe 0000:04:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 83.891204] ixgbe 0000:04:00.1: eth3: NIC Link is Down

[ 85.801341] ixgbe 0000:04:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 86.335694] ixgbe 0000:04:00.1: eth3: NIC Link is Down

[ 87.991415] ixgbe 0000:04:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 90.545692] ixgbe 0000:04:00.1: eth3: NIC Link is Down

[ 92.190203] ixgbe 0000:04:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 92.449841] ixgbe 0000:04:00.1: eth3: NIC Link is Down

I 01/14/12 08:43:04 00076 ports: port A2 is now on-line

I 01/14/12 08:43:26 00077 ports: port A2 is now off-line

I 01/14/12 08:43:26 00435 ports: port A2 is Blocked by STP

I 01/14/12 08:43:26 00076 ports: port A2 is now on-line

I 01/14/12 08:43:27 00077 ports: port A2 is now off-line

I 01/14/12 08:43:27 00435 ports: port A2 is Blocked by STP

I 01/14/12 08:43:27 00076 ports: port A2 is now on-line

I 01/14/12 08:43:29 00077 ports: port A2 is now off-line

I 01/14/12 08:43:30 00435 ports: port A2 is Blocked b...

[idata]

Hi Mark,

have just tripple checked that the cable is not the problem, after exchanging the DA cables, get the same UP/DOWN problem with the working one, so the switch and the cable are okay!

To summarise:

1.) 2 different X520-DA2 adapter, in 2 boxes -> none worked, almost got a "no cable" message

2.) 1 X520-SR2 adapter works well in all boxes

3.) 1 X520-SR2 adapter UP/DOWN problem, regardless of the used DA cable and/or switch port

How to find the reason of this very strage behavoir?

Thanks

Henri

[Mark_H_Intel]

Hi Henri,

Regarding "Is there any way to check the firmware/revision to be sure that the cards running with the same firmware?"

The firmware is not field upgradable, but that is not usually an issue, because updates to any firmware is rare. The logs show the PBA numbers for your adapters with newer adapters having a higher dash level. Quite often the dash level changes if for something trivial such as a label change or a new OUI prefix in the MAC address. I will see if I can tell what differences you might have in your X520 adapters..

Here are the PBA numbers I see in your posts. I do not have the PBA numbers for some of your adapters. Please provide me the missing PBA numbers so that I can check for differences. Do both SR2 adapters have the same PBA number? What is the PBA number for the DA adapter?

E68785-005 is for an Intel® Ethernet Server Adapter X520-SR2. E37623-004 is for a 10 Gigabit CX4 Dual Port Server Adapter.

I will compare differences in adapters after I receive your PBA numbers. I do not think I will likely find any major differences when I compare the PBA numbers. Nevertheless, I want to compare the numbers to be sure.

You said, "have now replaced the X520-DA2 card with a X520-SR2 card, Port1:Intel GBIC, Port2: SFP+ DA cable." What do you mean by Intel GBIC? Do you mean the Intel SR optical module? I think that is what you mean, but I want to make sure that I understand.

Thank you for the additional information on the switch messages. I understand that the STP messages were because of the link flap.

I agree that the DA cables are likely good cables, but the length is longer than the supported length. I am very concerned that the link flap is related to the DA cable length. One adapter might work only because of a lucky manufacturing coincidence that caused this card to work with the long cables when the average adapter will not come up or will have link flap with the long DA cables. I am not confident that you will get any X520 adapter to work reliably with the unsupported cable length. However, I will make the PBA comparisons to check for any difference in manufacturing that might apply.

Mark H

[idata]

Hi Mark,

corrent, both SR2 card are cross connected with a Intel SR optical GBIC and connected to the switch a SFP+ DA cable.

Will send you the PBA numbers later, I have stored a picture of all card on my iPhone.

Henri

[idata]

Hi Mark,

I forgot to ask, to which specification the cable is not compatble. Before I ordered the X520, I found at your page that this card supports any specification conform cable, so I need to know what HP did wrong.

Thanks

Henri

[idata]

Hi Mark,

in order to analyze the problem a bit more deeper, I can confirm that a second HP E3500yl switch, attached to the (link-flapping) port A2 works as fine as

the X520-SR2 card on port A1 (you know the working one). I have now attached a CX4 cable to the 200 SAN box via a Intel 82598EB 10-Gigabit AT CX4 card (so the HP switch works currently as SFP+ to CX4 converter), this works also fine.

By the way, the servers are attached since a couple of months to the first E3500yl switch, using 15m Cisco CX4 cables, to some 82598EB 10-Gigabit AT CX4 adapters, without any issues.

Here are the PBA numbers:

working one: X520-SR2 E69818 (B) 11/2011 made in Malaysia - E10G42BFSR - 900137

no cable: X520-DA2 ??? E10G42BTDA - 900139

link-flap: X520-SR2 E69818 (B) 11/2011 made in Malaysia - E10G42BFSR - 900137

I made some photos, if you like, I could send you the files.

Thanks

Henri

[Mark_H_Intel]

Hi Henri,

Thank you for posting the information on your adapters.

The PBA numbers i need are recorded in the output like you posted earlier. Here is an example:

[ 34.439526] Intel(R) 10 Gigabit PCI Express Network Driver - version 3.7.14-NAPI

[ 34.613628] ixgbe 0000:04:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18

[ 34.613709] ixgbe 0000:04:00.0: setting latency timer to 64

[ 34.804487] ixgbe 0000:04:00.0: (PCI Express:5.0GT/s:Width x8) 00:1b:21:d8:40:ec

[ 34.804569] ixgbe 0000:04:00.0: eth2: MAC: 2, PHY: 15, SFP+: 5, PBA No: E68785-005

[ 34.804572] ixgbe 0000:04:00.0: eth2: Enabled Features: RxQ: 8 TxQ: 8 FdirHash RSS RSC

[ 34.804590] ixgbe 0000:04:00.0: eth2: Intel(R) 10 Gigabit Network Connection

I need the PBA number as shown in the example for the comparison. I am especially interested in comparing the PBA numbers for the two different SR2 adapters. I also did not see a PBA number for the DA adapter.

I probably do not need to see the photos, but I will take a look if you send them to me. I will send you my email address in a private message.

I cannot say which specification the cable might not meet except that I am concerned that the lengthy is over 7 meters. We only support up to 7 meters for passive DA cables. I suspect that the issue you are having is because the cable is passive and is over 7 meters.

Mark H

[idata]

Hi Mark,

here is the output for the working one. I already asked the HP support, if this is a active or passive cable, but up to now

they were unable to tell me which one of it it is.

[ 34.506770] ixgbe 0000:04:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18[ 34.506842] ixgbe 0000:04:00.0: setting latency timer to 64[ 34.684388] ixgbe 0000:04:00.0: (PCI Express:5.0GT/s:Width x8) 00:1b:21:d8:40:ec[ 34.684470] ixgbe 0000:04:00.0: eth2: MAC: 2, PHY: 15, SFP+: 5, PBA No: E68785-005[ 34.684473] ixgbe 0000:04:00.0: eth2: Enabled Features: RxQ: 8 TxQ: 8 FdirHash RSS RSC [ 34.684491] ixgbe 0000:04:00.0: eth2: Intel(R) 10 Gigabit Network Connection[ 34.684566] ixgbe 0000:04:00.1: PCI INT B -> GSI 19 (level, low) -> IRQ 19[ 34.684679] ixgbe 0000:04:00.1: setting latency timer to 64[ 35.906760] ixgbe 0000:04:00.1: (PCI Express:5.0GT/s:Width x8) 00:1b:21:d8:40:ed[ 35.906842] ixgbe 0000:04:00.1: eth3: MAC: 2, PHY: 1, PBA No: E68785-005[ 35.906844] ixgbe 0000:04:00.1: eth3: Enabled Features: RxQ: 8 TxQ: 8 FdirHash RSS RSC [ 35.906862] ixgbe 0000:04:00.1: eth3: Intel(R) 10 Gigabit Network Connection[ 35.906898] ixgbe 0000:05:00.0: PCI INT B -> GSI 17 (level, low) -> IRQ 17[ 35.907012] ixgbe 0000:05:00.0: setting latency timer to 64[ 36.016545] ixgbe 0000:05:00.0: (PCI Express:2.5GT/s:Width x8) 00:1b:21:a4:aa:4d[ 36.016628] ixgbe 0000:05:00.0: eth4: MAC: 1, PHY: 0, PBA No: E37623-004[ 36.016630] ixgbe 0000:05:00.0: eth4: Enabled Features: RxQ: 8 TxQ: 8 RSS LRO [ 36.016632] ixgbe 0000:05:00.0: eth4: Intel(R) 10 Gigabit Network Connection[ 36.016666] ixgbe 0000:05:00.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16[ 36.016778] ixgbe 0000:05:00.1: setting latency timer to 64[ 36.116618] ixgbe 0000:05:00.1: (PCI Express:2.5GT/s:Width x8) 00:1b:21:a4:aa:4c[ 36.116701] ixgbe 0000:05:00.1: eth5: MAC: 1, PHY: 0, PBA No: E37623-004[ 36.116703] ixgbe 0000:05:00.1: eth5: Enabled Features: RxQ: 8 TxQ: 8 RSS LRO [ 36.116705] ixgbe 0000:05:00.1: eth5: Intel(R) 10 Gigabit Network Connection Here is the specification of the cable, no word if this is a active or passive one. The 10m and 15m cables are new, just a few months on the market. The cables up to 7m are longer on the market, so I assume the 10m and 15m cables are active, not just longer passive cables. But up to now I can not find any confirmation. ThanksHenri http://h18004.www1.hp.com/products/quickspecs/13286_div/13286_div.html http://h18004.www1.hp.com/products/quickspecs/13286_div/13286_div.html ModelsHP X242 SFP+ SFP+ 1 m Direct Attach CableJ9281BHP X242 SFP+ SFP+ 3 m Direct Attach CableJ9283BHP X242 SFP+ SFP+ 7 m Direct Attach CableJ9285BHP X242 SFP+ to SFP+ 10m Direct Attach Copper CableJ9286BHP X242 SFP+ to SFP+ 15m Direct Attach Copper CableJ9287B Key Features

• SFF-8431 compliant
• Supports 10-GbE data rate operation
• Serial data transmission

HP X242 SFP+ to SFP+ 15m Direct Attach Copper Cable (J9287B)

Connectivity

Length

49.20 ft. (15 m)Physical characteristicsWeight1.75 lb. (0.79 kg), Fully loaded the cable with an SFP+ transceiver at each end of the cableEnvironmentOperating temperature23°F to 185°F (-5°C to 85°C)Operating relative humidity5% to 95%, noncondensingNonoperating/Storage temperature14ºF to 185ºF (-10ºC to 85ºC)Nonoperating/Storage relative humidity5% to 95%, noncondensingAltitude</...

[idata]

Hi Mark,

comparing the electrical specifications, all cables up to 7m are specified as listed here, the 10m and 15m cable specification is different as you can see.

Is this the indication of a "active" cable? Sorry I'am not a cable specialist, I just using cables to connect things together.

Thanks

Henri

HP X242 SFP+ SFP+ 7 m Direct Attach Cable(J9285B)

Electrical characteristicsNotes0.04 watts maximum per transceiver end

HP X242 SFP+ to SFP+ 10m Direct Attach Copper Cable (J9286B)

Electrical characteristicsMaximum power rating1.2 WNotesMaximum power rating and maximum heat dissipation are the worst-case theoretical maximum numbers provided for planning the infrastructure with fully loaded PoE (if equipped), 100% traffic, all ports plugged in, and all modules populated. 0.6 watts maximum per transceiver end

http://h18004.www1.hp.com/products/quickspecs/13286_div/13286_div.html http://h18004.www1.hp.com/products/quickspecs/13286_div/13286_div.html

[Mark_H_Intel]

Hi Henri,

I will investigate further how this cable works with our adapters. I do not know if we have tested with the longer HP cables or not. As you have pointed out, the longer cables appear to be new.

Mark H

Added Information:

I thought I should add some clarification along with a link to the http://www.ethernetalliance.org/wp-content/uploads/2011/10/document_files_SFP_Plugfest_White_Paper_formattedv2.pdf SFP+ Direct Attach Copper Interoperability Demonstration White Paper to show some of the testing that has been done with Direct Attach cables. The testing included some cables as long as 8.5 meters, but no 10 or 15 meter cables. As a member of the http://www.ethernetalliance.org/ Ethernet Alliance, Intel participates in interoperability testing events such as these. We do our best to make interopreability easy, but in this case we are dealing with a longer cable that we are not familiar with.

Longer cables face big challenges in trying to fully meet the specifications, which is why I am concerned about the 15 meter cable length. I am not saying that the cable does not meet the specifications. Maybe there are active components that allow the cable to meet the specifications. I do not know if the cable does or does not meet the specifications, but I will investigate.

[idata]

Hi Mark,

superb, thanks you very much, this will help a lot.

Henri

[idata]

Hi Mark,

to keep you informed, I tested in the meantime also a HP 3m passive cable, this works without issues with both X520-SR2 adapters.

The SFP+DA15 cable works still fine with the (working) one X520-SR2 card, even I moved the card to the second server again.

Any news regarding your tests?

Thanks

Henri

Product Serial Part Port Type Number Number Number ------- ----------- ------------ ------------------ ---------- A1 SFP+DA15 J9287B US10F 8121-1245 A2 SFP+DA3 J9283B CN1295 8121-1152

[Mark_H_Intel]

Hi Henri,

Thank you for the update.

Your tests imply an issue with compatibility between the Intel(R) Ethernet Server Adapter X520 series and the 15 meter cables from HP. Unfortunately, I do not know anything that can be done at this time to make the 15 meter cable work. We will investigate the incompatiblity. I am sorry that the longer cable length is not working when connected to most of the adatpers. I will let you know anything I find out, but I do not expect any results soon.

Whenever I find out anything more, I will post the information here.

Mark H

[idata]

good info