X520 10G - UDP packet drops

[idata]

Hello,

I have a strange problem occurring with my new server which has the intel x520 network cards. This server runs Ubuntu KVM with virtual machines that collect netflow data- these device are virtual appliances. I noticed within the virtual machine that netflow datagrams. I first suspected resource constraints within KVM however after disabling all of the virtual appliances (except one) the packet drops were still occurring. I am able to determine the packet drops by simply taking a pcap and running it in a script with tshark. Each UDP datagram that contains the netflow data has a sequence number in the application layer header. I then ran a pcap on the host bridge interface and found that the issue is occurring outside the virtual machine. I then did a packet capture on physical interface to rule out that it was caused by the bridge and the issue is occurring there too. I then mirrored the interface to a separate server that has a similar 10gig network card and this device is able to capture each packet.

I suspect this is a network card offload /drive issue however I have not been successful in finding a solution. I tried upgrading the driver from 4.0.1 to 4.3.13 with no difference.

I am running out of options.

I tried disabling

for i in rx tx sg tso ufo gso gro lro; do sudo ethtool -K enp33s0f1 $i off; done

Tried increasing memory allocation to receive memory buffer:

cat /proc/sys/net/core/rmem_default

212992

cat /proc/sys/net/core/rmem_max

212992

and set the values to:

net.core.rmem_default = 16777216

net.core.rmem_max = 16777216

I noticed packets were incrementing the below counters in ethtool -S so i tried the below modprobe command but it did not help.

hw_rsc_aggregated: 4074

hw_rsc_flushed: 1837

modprobe ixgbe InterruptThrottleRate=off

server with issue:

lspci |grep Ethernet

0c:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

0c:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

21:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

21:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

62:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

62:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

63:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

63:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

I first tried with this below driver version.

version: 4.0.1-k

Then upgraded and tried the below driver version.

version: 4.3.13

server without issue:

[em621d@shdw01vmh ~]$ lspci |grep Ethernet

0e:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)

0e:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)

version: 3.6.7-k

Traffic characteristics:

Under 150Mbps

Rate 10k pps

Packet size 1340 - 1372 bytes

ethtool -S enp33s0f1

NIC statistics:

rx_packets: 728170

tx_packets: 8

rx_bytes: 1007244560

tx_bytes: 648

rx_errors: 0

tx_errors: 0

rx_dropped: 0

tx_dropped: 0

multicast: 254

collisions: 0

rx_over_errors: 0

rx_crc_errors: 0

rx_frame_errors: 0

rx_fifo_errors: 0

rx_missed_errors: 0

tx_aborted_errors: 0

tx_carrier_errors: 0

tx_fifo_errors: 0

tx_heartbeat_errors: 0

rx_pkts_nic: 728170

tx_pkts_nic: 8

rx_bytes_nic: 1010157240

tx_bytes_nic: 680

lsc_int: 1

tx_busy: 0

non_eop_descs: 0

broadcast: 63

rx_no_buffer_count: 0

tx_timeout_count: 0

tx_restart_queue: 0

rx_long_length_errors: 0

rx_short_length_errors: 0

tx_flow_control_xon: 0

rx_flow_control_xon: 0

tx_flow_control_xoff: 0

rx_flow_control_xoff: 0

rx_csum_offload_errors: 0

alloc_rx_page_failed: 0

alloc_rx_buff_failed: 0

lro_aggregated: 0

lro_flushed: 0

rx_no_dma_resources: 0

hw_rsc_aggregated: 0

hw_rsc_flushed: 0

fdir_match: 0

fdir_miss: 727925

fdir_overflow: 0

fcoe_bad_fccrc: 0

fcoe_last_errors: 0

rx_fcoe_dropped: 0

rx_fcoe_packets: 0

rx_fcoe_dwords: 0

fcoe_noddp: 0

fcoe_noddp_ext_buff: 0

tx_fcoe_packets: 0

tx_fcoe_dwords: 0

os2bmc_rx_by_bmc: 0

os2bmc_tx_by_bmc: 0

os2bmc_tx_by_host: 0

os2bmc_rx_by_host: 0

tx_hwtstamp_timeouts: 0

rx_hwtstamp_cleared: 0

tx_queue_0_packets: 0

tx_queue_0_bytes: 0

tx_queue_0_bp_napi_yield: 0

tx_queue_0_bp_misses: 0

tx_queue_0_bp_cleaned: 0

tx_queue_1_packets: 0

tx_queue_1_bytes: 0

tx_queue_1_bp_napi_yield: 0

tx_queue_1_bp_misses: 0

tx_queue_1_bp_cleaned: 0

tx_queue_2_packets: 0

tx_queue_2_bytes: 0

tx_queue_2_bp_napi_yield: 0

tx_queue_2_bp_misses: 0

tx_queue_2_bp_cleaned: 0

tx_queue_3_packets: 0

tx_queue_3_bytes: 0

tx_queue_3_bp_napi_yield: 0

tx_queue_3_bp_misses: 0

tx_queue_3_bp_cleaned: 0

tx_queue_4_packets: 0

tx_queue_4_bytes: 0

tx_queue_4_bp_napi_yield: 0

tx_queue_4_bp_misses: 0

tx_queue_4_bp_cleaned: 0

tx_queue_5_packets: 0

tx_queue_5_bytes: 0

tx_queue_5_bp_napi_yield: 0

tx_queue_5_bp_misses: 0

tx_queue_5_bp_cleaned: 0

tx_queue_6_packets: 0

tx_queue_6_bytes: 0

tx_queue_6_bp_napi_yield: 0

tx_queue_6_bp_misses: 0

tx_queue_6_bp_cleaned: 0

tx_queue_7_packets: 0

tx_queue_7_bytes: 0

tx_queue_7_bp_napi_yield: 0

tx_queue_7_bp_misses: 0

tx_queue_7_bp_cle...

[st4]

Hi emueller,

Thank you for the info. We need to check on this.

rgds,

wb

[MichaelJakobs]

May I ask for any progress regarding this issue?
We have exactly the same issue with X550 nics.