Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4628 Discussions

Use of both ports of a single network card ( X710 or 82599ES ) causes slow thoughtput

fYama
Beginner
‎01-22-2019 06:35 PM
1,689 Views

Hi

I have some servers ( they all have 2 cards ( x710 or 82599ES ) )

If I use both ports ( for a very intensive service ( a lot of small packets )) I get a very low throutput.

If I try the same, but using one port of each card, all work fine.

 

the pcie bus is "PCI-Express: Speed 5.0GT/s Width x8"

 

also, I got the message saying "PCI-Express bandwidth available for this device may be insufficient for optimal performance."

 

but afaik 5*8 is bandwidth enough for a dual 10gb, I'm right?

 

here is the output out lspci -vvvv

43:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 0, Cache Line Size: 64 bytes

Interrupt: pin A routed to IRQ 203

Region 0: Memory at fb300000 (64-bit, non-prefetchable) [size=1M]

Region 2: I/O ports at ece0 [size=32]

Region 4: Memory at fb1fc000 (64-bit, non-prefetchable) [size=16K]

Expansion ROM at d0000000 [disabled] [size=512K]

Capabilities: [40] Power Management version 3

Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)

Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-

Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

Address: 0000000000000000 Data: 0000

Masking: 00000000 Pending: 00000000

Capabilities: [70] MSI-X: Enable+ Count=64 Masked-

Vector table: BAR=4 offset=00000000

PBA: BAR=4 offset=00002000

Capabilities: [a0] Express (v2) Endpoint, MSI 00

DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us

ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W

DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+

RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-

MaxPayload 256 bytes, MaxReadReq 512 bytes

DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-

LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 <8us

ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-

LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported

DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled

LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-

EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

Capabilities: [e0] Vital Product Data

Product Name: HP Ethernet 10Gb 2-port 560SFP+ Adapter

Read-only fields:

[PN] Part number: 665247-001

[EC] Engineering changes: B-5317

[SN] Serial number: MYI33105FB

[V0] Vendor specific: 11W/8W PCIeG2x8 2p 10Gb SFP+ Intel 82599

[V2] Vendor specific: 5331

[V4] Vendor specific: 38EAA7902738

[V5] Vendor specific: 0B

[RV] Reserved: checksum good, 0 byte(s) reserved

Read/write fields:

[V1] Vendor specific: 00.00.00

[V3] Vendor specific: 2.7.110

[V6] Vendor specific: 2.2.02

[YA] Asset tag: N/A

[YB] System specific: xxxxxxxxxxxxxxxx

[YC] System specific: xxxxxxxxxxxxxxxx

End

Capabilities: [100 v1] Advanced Error Reporting

UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-

CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-

CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+

AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-

Capabilities: [140 v1] Device Serial Number 38-ea-a7-ff-ff-90-27-38

Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)

ARICap: MFVC- ACS-, Next Function: 0

ARICtl: MFVC- ACS-, Function Group: 0

Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)

IOVCap: Migration-, Interrupt Message Number: 000

IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-

IOVSta: Migration-

Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 01

VF offset: 128, stride: 2, Device ID: 10ed

Supported Page Size: 00000553, System Page Size: 00000001

Region 0: Memory at 00000000d0280000 (64-bit, prefetchable)

Region 3: Memory at 00000000d0380000 (64-bit, prefetchable)

VF Migration: offset: 00000000, BIR: 0

Kernel driver in use: ixgbe

Kernel modules: ixgbe

 

0 Kudos

Link Copied

5 Replies
Daniel_D_Intel1
Employee
‎01-23-2019 08:19 AM
536 Views
Hello FYama, Thank you for posting in Intel Ethernet Communities. Please provide the markings from the product label so we can identify the adapter. What are you using to measure throughput, and would you be able to get an iperf output using the connections on the servers? Please also provide "ethtool -i <interface>" and "ethtool -k <interface>". Let us know if you have any other questions. Best regards, Daniel D Intel Customer Support Under Contract to Intel Corporation
0 Kudos
Copy link
fYama
Beginner
‎01-24-2019 02:24 AM
536 Views

Hi

 

iperf does not show any problem, but I don't think bandwidth is the problem. but packet per second.

Ou solution sends about 250k pps ( very small packets ) to about 5000k users ( per network card )

 

there is a shared irq or something at hardware level that prevent-us to use both ports of the card?

( if we use 2 cards, and use 1 port of each card, we don't get this problem )

 

here is the info

 

ethtool -i enp67s0f1

driver: i40e

version: 2.7.29

firmware-version: 5.40 0x80002d35 18.0.17

expansion-rom-version: 

bus-info: 0000:43:00.1

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

 

 

 

ethtool -k enp67s0f1

Features for enp67s0f1:

rx-checksumming: on

tx-checksumming: on

tx-checksum-ipv4: on

tx-checksum-ip-generic: off [fixed]

tx-checksum-ipv6: on

tx-checksum-fcoe-crc: off [fixed]

tx-checksum-sctp: on

scatter-gather: on

tx-scatter-gather: on

tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: on

tx-tcp-segmentation: on

tx-tcp-ecn-segmentation: on

tx-tcp-mangleid-segmentation: off

tx-tcp6-segmentation: on

udp-fragmentation-offload: off

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off [fixed]

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: on

receive-hashing: on

highdma: on

rx-vlan-filter: on [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

tx-gre-segmentation: on

tx-gre-csum-segmentation: on

tx-ipxip4-segmentation: on

tx-ipxip6-segmentation: on

tx-udp_tnl-segmentation: on

tx-udp_tnl-csum-segmentation: on

tx-gso-partial: on

tx-sctp-segmentation: off [fixed]

tx-esp-segmentation: off [fixed]

fcoe-mtu: off [fixed]

tx-nocache-copy: off

loopback: off [fixed]

rx-fcs: off [fixed]

rx-all: off [fixed]

tx-vlan-stag-hw-insert: off [fixed]

rx-vlan-stag-hw-parse: off [fixed]

rx-vlan-stag-filter: off [fixed]

l2-fwd-offload: off [fixed]

hw-tc-offload: on

esp-hw-offload: off [fixed]

esp-tx-csum-hw-offload: off [fixed]

rx-udp_tunnel-port-offload: on

 

0 Kudos
Copy link
Daniel_D_Intel1
Employee
‎01-24-2019 03:23 AM
536 Views
Hello FYama, Thank you for the outputs. This does not seem to be an Intel retail adapter from the firmware in ethtool. Please check with the OEM of the adapter. They should be able to inform you if the adapter requires more than x8 PCIe. The 82599ES Ethernet controller requires 5.0GT/s, but the specific hardware may require more lanes. You may also check if there is a recommended firmware update for this adapter. The Intel NVM update will only work on Intel retail X710 adapters. Let us know if you have any other questions after contacting the OEM. Best regards, Daniel D Intel Customer Support Under Contract to Intel Corporation
0 Kudos
Copy link
Daniel_D_Intel1
Employee
‎01-29-2019 08:30 AM
536 Views
Hello FYama, Please let us know if a firmware update from the OEM resolved this issue. If you have any other questions please do not hesitate to ask. Best regards, Daniel D Intel Customer Support Under Contract to Intel Corporation
0 Kudos
Copy link
VincentT_Intel
Moderator
‎02-01-2019 06:37 AM
536 Views
Hi FYama, Please let us know if there's other assistance needed. Best Regards, Vince T. Intel Customer Support Under Contract to Intel Corporation
0 Kudos
Copy link
Reply

Community support is provided during standard business hours (Monday to Friday 7AM - 5PM PST). Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.