X520-DA2, FreeBSD, rec_len_errs

[GAlex4]

Hello,

We have a FreeBSD server in one of our data centers that is accumulating rec_len_errs. Our network team says everything looks okay on the Omni switch to which it is attached. However, something is causing these errors and I have exhausted my bucket of expertise trying to track it down. We have replaced everything my department owns after the switch - Intel SPF+ modules (both sides), fiber jumper (twice), X520-DA2 Network Adapter (once). This server is configured exactly the same as an identical server that is in a different data center. However, that other server is not showing these errors. Here is some relevant information for anyone who wants it:

FreeBSD 11.2-RELEASE-p2

X520-DA2 Network Adapter

Intel FTLX1471D3BCV-IT 10Gb SFP+ Module

Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k

# sysctl dev.ix.1 | grep rec_len_errs

dev.ix.1.mac_stats.rec_len_errs: 528161

Let me know if something else needs to be added. I believe this rec_len_errs field is for a receive length error. Is that correct? Does anyone have information on how I can diagnose this on the server side? I have tried tcpdump with various switches but am not seeing anything that suggests whence cometh this error. Thank you.

Gene

[idata]

Hi Gene,

Thank you for posting in Wired Ethernet Communities.You've mentioned there's identical server in a different data center that is not showing the error. Please share the configured settings on the X520-DA2. You can share the ethtool -k output of the server w/ error vs server w/out error.We have driver version 3.2.18 posted in this https://downloadcenter.intel.com/download/14688/Ethernet-Intel-Network-Adapters-Driver-for-PCIe-10-Gigabit-Network-Connections-Under-FreeBSD- link. You may want to check it out.Best Regards,Vince T.Intel Customer Support

[GAlex4]

Vince,

Thank you for the reply. There is no 'ethtool' on FreeBSD, that is a Linux tool. Here is the information I can get for you from FreeBSD:

Server with errors:

# sysctl hw.ix

hw.ix.enable_rss: 1

hw.ix.enable_legacy_tx: 0

hw.ix.enable_fdir: 0

hw.ix.unsupported_sfp: 0

hw.ix.rxd: 2048

hw.ix.txd: 2048

hw.ix.num_queues: 8

hw.ix.enable_msix: 1

hw.ix.advertise_speed: 0

hw.ix.flow_control: 3

hw.ix.tx_process_limit: 256

hw.ix.rx_process_limit: 256

hw.ix.max_interrupt_rate: 31250

hw.ix.enable_aim: 1

# sysctl dev.ix.1

dev.ix.1.mac_stats.tx_frames_1024_1522: 212217345

dev.ix.1.mac_stats.tx_frames_512_1023: 27883088

dev.ix.1.mac_stats.tx_frames_256_511: 1019347

dev.ix.1.mac_stats.tx_frames_128_255: 158582

dev.ix.1.mac_stats.tx_frames_65_127: 84329344

dev.ix.1.mac_stats.tx_frames_64: 62395

dev.ix.1.mac_stats.management_pkts_txd: 0

dev.ix.1.mac_stats.mcast_pkts_txd: 2

dev.ix.1.mac_stats.bcast_pkts_txd: 2

dev.ix.1.mac_stats.good_pkts_txd: 325670101

dev.ix.1.mac_stats.total_pkts_txd: 325670101

dev.ix.1.mac_stats.good_octets_txd: 356914457228

dev.ix.1.mac_stats.checksum_errs: 1030

dev.ix.1.mac_stats.management_pkts_drpd: 0

dev.ix.1.mac_stats.management_pkts_rcvd: 0

dev.ix.1.mac_stats.recv_jabberd: 0

dev.ix.1.mac_stats.recv_oversized: 4

dev.ix.1.mac_stats.recv_fragmented: 0

dev.ix.1.mac_stats.recv_undersized: 0

dev.ix.1.mac_stats.rx_frames_1024_1522: 57388071

dev.ix.1.mac_stats.rx_frames_512_1023: 25136

dev.ix.1.mac_stats.rx_frames_256_511: 39087

dev.ix.1.mac_stats.rx_frames_128_255: 15812

dev.ix.1.mac_stats.rx_frames_65_127: 28078923

dev.ix.1.mac_stats.rx_frames_64: 1259559

dev.ix.1.mac_stats.bcast_pkts_rcvd: 633245

dev.ix.1.mac_stats.mcast_pkts_rcvd: 1374

dev.ix.1.mac_stats.good_pkts_rcvd: 86806588

dev.ix.1.mac_stats.total_pkts_rcvd: 87481597

dev.ix.1.mac_stats.good_octets_rcvd: 89185700980

dev.ix.1.mac_stats.total_octets_rcvd: 89231162404

dev.ix.1.mac_stats.xoff_recvd: 0

dev.ix.1.mac_stats.xoff_txd: 0

dev.ix.1.mac_stats.xon_recvd: 0

dev.ix.1.mac_stats.xon_txd: 0

dev.ix.1.mac_stats.rx_missed_packets: 0

dev.ix.1.mac_stats.rec_len_errs: 563723

dev.ix.1.mac_stats.remote_faults: 3

dev.ix.1.mac_stats.local_faults: 6

dev.ix.1.mac_stats.short_discards: 0

dev.ix.1.mac_stats.byte_errs: 0

dev.ix.1.mac_stats.ill_errs: 0

dev.ix.1.mac_stats.crc_errs: 0

dev.ix.1.queue7.lro_flushed: 0

dev.ix.1.queue7.lro_queued: 0

dev.ix.1.queue7.rx_discarded: 0

dev.ix.1.queue7.rx_copies: 3571848

dev.ix.1.queue7.rx_bytes: 10195751974

dev.ix.1.queue7.rx_packets: 10157527

dev.ix.1.queue7.rxd_tail: 1494

dev.ix.1.queue7.rxd_head: 1495

dev.ix.1.queue7.br_drops: 0

dev.ix.1.queue7.tx_packets: 39588079

dev.ix.1.queue7.no_desc_avail: 1835985

dev.ix.1.queue7.no_tx_dma_setup: 0

dev.ix.1.queue7.tso_tx: 0

dev.ix.1.queue7.txd_tail: 506

dev.ix.1.queue7.txd_head: 506

dev.ix.1.queue7.irqs: 7268775

dev.ix.1.queue7.interrupt_rate: 35714

dev.ix.1.queue6.lro_flushed: 0

dev.ix.1.queue6.lro_queued: 0

dev.ix.1.queue6.rx_discarded: 0

dev.ix.1.queue6.rx_copies: 2870611

dev.ix.1.queue6.rx_bytes: 10435029610

dev.ix.1.queue6.rx_packets: 9644821

dev.ix.1.queue6.rxd_tail: 788

dev.ix.1.queue6.rxd_head: 789

dev.ix.1.queue6.br_drops: 0

dev.ix.1.queue6.tx_packets: 41895750

dev.ix.1.queue6.no_desc_avail: 0

dev.ix.1.queue6.no_tx_dma_setup: 0

dev.ix.1.queue6.tso_tx: 0

dev.ix.1.queue6.txd_tail: 64

dev.ix.1.queue6.txd_head: 64

dev.ix.1.queue6.irqs: 8601254

dev.ix.1.queue6.interrupt_rate: 29411

dev.ix.1.queue5.lro_flushed: 0

dev.ix.1.queue5.lro_queued: 0

dev.ix.1.queue5.rx_discarded: 0

dev.ix.1.queue5.rx_copies: 3417848

dev.ix.1.queue5.rx_bytes: 9888683630

dev.ix.1.queue5.rx_packets: 9806736

dev.ix.1.queue5.rxd_tail: 911

dev.ix.1.queue5.rxd_head: 912

dev.ix.1.queue5.br_drops: 0

dev.ix.1.queue5.tx_packets: 39585350

dev.ix.1.queue5.no_desc_avail: 1149660

dev.ix.1.queue5.no_tx_dma_setup: 0

dev.ix.1.queue5.tso_tx: 0

dev.ix.1.queue5.txd_tail: 1511

dev.ix.1.queue5.txd_head: 1511

dev.ix.1.queue5.irqs: 6931326

dev.ix.1.queue5.interrupt_rate: 500000

dev.ix.1.queue4.lro_flushed: 0

dev.ix.1.queue4.lro_queued: 0

dev.ix.1.queue4.rx_discarded: 0

dev.ix.1.queue4.rx_copies: 4171931

dev.ix.1.queue4.rx_bytes: 11464040752

dev.ix.1.queue4.rx_packets: 11568882

dev.ix.1.queue4.rxd_tail: 1777

dev.ix.1.queue4.rxd_head: 1778

dev.ix.1.queue4.br_drops: 0

dev.ix.1.queue4.tx_packets: 48718706

dev.ix.1.queue4.no_desc_avail: 53113

dev.ix.1.queue4.no_tx_dma_setup: 0

dev.ix.1.queue4.tso_tx: 0

dev.ix.1.queue4.txd_tail: 1732

dev.ix.1.queue4.txd_head: 1732

dev.ix.1.queue4.irqs: 10741290

dev.ix.1.queue4.interrupt_rate: 500000

dev.ix.1.queue3.lro_flushed: 0

dev.ix.1.queue3.lro_queued: 0

dev.ix.1.queue3.rx_discarded: 0

dev.ix.1.queue3.rx_copies: 3488870

dev.ix.1.queue3.rx_bytes: 10782358002

dev.ix.1.queue3.rx_packets: 10466983

dev.ix.1.queue3.rxd_tail: 1702

dev.ix.1.queue3.rxd_head: 1703

dev.ix.1.queue3.br_drops: 0

dev.ix.1.queue3.tx_packets: 37299621

dev.ix.1.queue3.no_desc_avail: 26160

dev.ix.1.queue3.no_tx_dma_setup: 0

dev.ix.1.queue3.tso_tx: 0

dev.ix.1.queue3.txd_tail: 672

dev.ix.1.queue3.txd_head: 672

dev.ix.1.queue3.irqs: 8362710

dev.ix.1.queue3.interrupt_rate: 500000

dev.ix.1.queue2.lro_flushed: 0

dev.ix.1.queue2.lro_queued: 0

dev.ix.1.queue2.rx_discarded: 0

dev.ix.1.queue2.rx_copies: 3314209

dev.ix.1.queue2.rx_bytes: 13476080887

dev.ix.1.queue2.rx_packets: 12083700

dev.ix.1.queue2.rxd_tail: 499

dev.ix.1.queue2.rxd_head: 500

dev.ix.1.queue2.br_drops: 0

dev.ix.1.queue2.tx_packets: 36066665

dev.ix.1.queue2.no_desc_avail: 225275

dev.ix.1.queue2.no_tx_dma_setup: 0

dev.ix.1.queue2.tso_tx: 0

dev.ix.1.queue2.txd_tail: 1813

dev.ix.1.queue2.txd_head: 1813

dev.ix.1.queue2.irqs: 8956553

dev.ix.1.queue2.interrupt_rate: 71428

dev.ix.1.queue1.lro_flushed: 0

dev.ix.1.queue1.lro_queued: 0

dev.ix.1.queue1.rx_discarded: 0

dev.ix.1.queue1.rx_copies: 3575987

dev.ix.1.queue1.rx_bytes: 9279977915

dev.ix.1.queue1.rx_packets: 9557914

dev.ix.1.queue1.rxd_tail: 1945

dev.ix.1.queue1.rxd_head: 1946

dev.ix.1.queue1.br_drops: 0

dev.ix.1.queue1.tx_packets: 40520...

[GAlex4]

Vince, et al,

One of our network engineers seems to have tracked down the problem. Here is a copy of his e-mail about it:

I may have found a lead. I found an email on Intel's ixgbe driver mailing list with a very similar situation to the one we're having. They also state they've seen this issue with Alcatel-Lucent Omniswitch 6850 which is in the same switch family as ours.

ixgbe and rx errors: https://sourceforge.net/p/e1000/mailman/message/30759532/

Based on the post it seems to be caused by STP (likely the BPDUs) where the packet length in the header doesn't match the actual packet length. They also indicate that changing from STP/RSTP to MSTP fixes the error.

The email also seems to indicate that you can't capture the errored packets from the switch unless you disable the HLREG.RXLNGTHERREN bit by recompiling the driver. I'm assuming that the Intel NIC drops the NIC before it sends it to the kernel. So the packet capture probably won't reveal the issue unless we had a passive tap or recompiled drivers.

So with that information I think we could probably turn off STP on for VLAN 1254 and that would probably fix the receive length errors. I don't see a reason for having RSTP on the VLAN for the server since it's only connecting with a LAG and doesn't look like it has a vSwitch.

So, it is not actually a problem on the FreeBSD side or with the Intel hardware / driver. My manager and the engineering team have decided to replace that Omniswitch with a newer Cisco model that matches the one to which our other server is attached in the next quarter.

Gene

[idata]

Hi Gene,

Thank you for sharing the information. Glad to know you've found the solution to your issue.Feel free to post in this thread if you have other assistance needed.Regards,Vince T.Intel Customer Support