Community
cancel
Showing results for 
Search instead for 
Did you mean: 
fYama
Beginner
1,217 Views

Use of both ports of a single network card ( X710 or 82599ES ) causes slow thoughtput

Hi

I have some servers ( they all have 2 cards ( x710 or 82599ES ) )

If I use both ports ( for a very intensive service ( a lot of small packets )) I get a very low throutput.

If I try the same, but using one port of each card, all work fine.

 

the pcie bus is "PCI-Express: Speed 5.0GT/s Width x8"

 

also, I got the message saying "PCI-Express bandwidth available for this device may be insufficient for optimal performance."

 

but afaik 5*8 is bandwidth enough for a dual 10gb, I'm right?

 

here is the output out lspci -vvvv

43:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 0, Cache Line Size: 64 bytes

Interrupt: pin A routed to IRQ 203

Region 0: Memory at fb300000 (64-bit, non-prefetchable) [size=1M]

Region 2: I/O ports at ece0 [size=32]

Region 4: Memory at fb1fc000 (64-bit, non-prefetchable) [size=16K]

Expansion ROM at d0000000 [disabled] [size=512K]

Capabilities: [40] Power Management version 3

Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)

Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-

Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

Address: 0000000000000000 Data: 0000

Masking: 00000000 Pending: 00000000

Capabilities: [70] MSI-X: Enable+ Count=64 Masked-

Vector table: BAR=4 offset=00000000

PBA: BAR=4 offset=00002000

Capabilities: [a0] Express (v2) Endpoint, MSI 00

DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us

ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W

DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+

RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-

MaxPayload 256 bytes, MaxReadReq 512 bytes

DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-

LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 <8us

ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-

LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported

DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled

LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-

EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

Capabilities: [e0] Vital Product Data

Product Name: HP Ethernet 10Gb 2-port 560SFP+ Adapter

Read-only fields:

[PN] Part number: 665247-001

[EC] Engineering changes: B-5317

[SN] Serial number: MYI33105FB

[V0] Vendor specific: 11W/8W PCIeG2x8 2p 10Gb SFP+ Intel 82599

[V2] Vendor specific: 5331

[V4] Vendor specific: 38EAA7902738

[V5] Vendor specific: 0B

[RV] Reserved: checksum good, 0 byte(s) reserved

Read/write fields:

[V1] Vendor specific: 00.00.00

[V3] Vendor specific: 2.7.110

[V6] Vendor specific: 2.2.02

[YA] Asset tag: N/A

[YB] System specific: xxxxxxxxxxxxxxxx

[YC] System specific: xxxxxxxxxxxxxxxx

End

Capabilities: [100 v1] Advanced Error Reporting

UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-

CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-

CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+

AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-

Capabilities: [140 v1] Device Serial Number 38-ea-a7-ff-ff-90-27-38

Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)

ARICap: MFVC- ACS-, Next Function: 0

ARICtl: MFVC- ACS-, Function Group: 0

Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)

IOVCap: Migration-, Interrupt Message Number: 000

IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-

IOVSta: Migration-

Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 01

VF offset: 128, stride: 2, Device ID: 10ed

Supported Page Size: 00000553, System Page Size: 00000001

Region 0: Memory at 00000000d0280000 (64-bit, prefetchable)

Region 3: Memory at 00000000d0380000 (64-bit, prefetchable)

VF Migration: offset: 00000000, BIR: 0

Kernel driver in use: ixgbe

Kernel modules: ixgbe

 

0 Kudos
5 Replies
Daniel_D_Intel1
Employee
64 Views

Hello FYama, Thank you for posting in Intel Ethernet Communities. Please provide the markings from the product label so we can identify the adapter. What are you using to measure throughput, and would you be able to get an iperf output using the connections on the servers? Please also provide "ethtool -i <interface>" and "ethtool -k <interface>". Let us know if you have any other questions. Best regards, Daniel D Intel Customer Support Under Contract to Intel Corporation
fYama
Beginner
64 Views

Hi

 

iperf does not show any problem, but I don't think bandwidth is the problem. but packet per second.

Ou solution sends about 250k pps ( very small packets ) to about 5000k users ( per network card )

 

there is a shared irq or something at hardware level that prevent-us to use both ports of the card?

( if we use 2 cards, and use 1 port of each card, we don't get this problem )

 

here is the info

 

ethtool -i enp67s0f1

driver: i40e

version: 2.7.29

firmware-version: 5.40 0x80002d35 18.0.17

expansion-rom-version: 

bus-info: 0000:43:00.1

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

 

 

 

ethtool -k enp67s0f1

Features for enp67s0f1:

rx-checksumming: on

tx-checksumming: on

tx-checksum-ipv4: on

tx-checksum-ip-generic: off [fixed]

tx-checksum-ipv6: on

tx-checksum-fcoe-crc: off [fixed]

tx-checksum-sctp: on

scatter-gather: on

tx-scatter-gather: on

tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: on

tx-tcp-segmentation: on

tx-tcp-ecn-segmentation: on

tx-tcp-mangleid-segmentation: off

tx-tcp6-segmentation: on

udp-fragmentation-offload: off

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off [fixed]

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: on

receive-hashing: on

highdma: on

rx-vlan-filter: on [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

tx-gre-segmentation: on

tx-gre-csum-segmentation: on

tx-ipxip4-segmentation: on

tx-ipxip6-segmentation: on

tx-udp_tnl-segmentation: on

tx-udp_tnl-csum-segmentation: on

tx-gso-partial: on

tx-sctp-segmentation: off [fixed]

tx-esp-segmentation: off [fixed]

fcoe-mtu: off [fixed]

tx-nocache-copy: off

loopback: off [fixed]

rx-fcs: off [fixed]

rx-all: off [fixed]

tx-vlan-stag-hw-insert: off [fixed]

rx-vlan-stag-hw-parse: off [fixed]

rx-vlan-stag-filter: off [fixed]

l2-fwd-offload: off [fixed]

hw-tc-offload: on

esp-hw-offload: off [fixed]

esp-tx-csum-hw-offload: off [fixed]

rx-udp_tunnel-port-offload: on

 

Daniel_D_Intel1
Employee
64 Views

Hello FYama, Thank you for the outputs. This does not seem to be an Intel retail adapter from the firmware in ethtool. Please check with the OEM of the adapter. They should be able to inform you if the adapter requires more than x8 PCIe. The 82599ES Ethernet controller requires 5.0GT/s, but the specific hardware may require more lanes. You may also check if there is a recommended firmware update for this adapter. The Intel NVM update will only work on Intel retail X710 adapters. Let us know if you have any other questions after contacting the OEM. Best regards, Daniel D Intel Customer Support Under Contract to Intel Corporation
Daniel_D_Intel1
Employee
64 Views

Hello FYama, Please let us know if a firmware update from the OEM resolved this issue. If you have any other questions please do not hesitate to ask. Best regards, Daniel D Intel Customer Support Under Contract to Intel Corporation
VincentT_Intel
Moderator
64 Views

Hi FYama, Please let us know if there's other assistance needed. Best Regards, Vince T. Intel Customer Support Under Contract to Intel Corporation
Reply