Setting the MaxReadReq to 2048 bytes on a 82599ES dual 10Gbit card on Linux

[Kalibro]

Hello,

I have an Intel 82599ES Dual 10G network card, running on a Linux Mint 20.2, kernel version 5.11.0.-34-generic, Intel Linux driver ixgbe ver 5.12.5. 

I am trying to figure out how to set the MaxReadReq parameter to the max value of 2048 bytes...it is currently set at 512 bytes. The Datasheet says that this value can be changed in the 0xA8 register, bits 14:12 to value of 0100b...however, I don't know how to go about doing this. I know that the linux command 'setpci' can change PCIe register values...but I don't know how to do this.

Here is the output from lspci:

04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
	Subsystem: Intel Corporation Ethernet Server Adapter X520-2
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 34
	Region 0: Memory at e0300000 (64-bit, prefetchable) [size=512K]
	Region 2: I/O ports at d000 [disabled] [size=32]
	Region 4: Memory at e0400000 (64-bit, prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
		Vector table: BAR=4 offset=00000000
		PBA: BAR=4 offset=00002000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s (ok), Width x8 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR-
			 10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-, TPHComp-, ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [140 v1] Device Serial Number f8-f2-1e-ff-ff-b1-2d-08
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy-
		IOVSta:	Migration-
		Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 01
		VF offset: 384, stride: 2, Device ID: 10ed
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 00000000e0700000 (64-bit, non-prefetchable)
		Region 3: Memory at 00000000e0800000 (64-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Kernel driver in use: ixgbe
	Kernel modules: ixgbe

And here is the hexadecimal map of the 82599ES:

04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
00: 86 80 fb 10 06 04 10 00 01 00 00 02 10 00 80 00
10: 0c 00 30 e0 00 00 00 00 01 d0 00 00 00 00 00 00
20: 0c 00 40 e0 00 00 00 00 00 00 00 00 86 80 03 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 02 00 00
40: 01 50 23 48 00 20 00 2b 00 00 00 00 00 00 00 00
50: 05 70 80 01 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 11 a0 3f 80 04 00 00 00 04 20 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 10 00 02 00 c2 8c 00 10 3f 28 09 00 82 c4 01 00
b0: 40 00 82 10 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 1f 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Does anyone know how to use setpci command to change the MaxReadReq of the 82599ES card to 2048 bytes ? Is there maybe a Linux GUI utility that can also do this ? 

Thank you kindly for any help!

[Mike_Intel]

Hello Kalibro,

Thank you for posting in Intel Ethernet Communities. 

Before we proceed, let me check if you are designing a board or system with embedded 82599ES? Can you share why do you need to edit the MaxReadReq.

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

Best regards,

Michael L.

Intel® Customer Support Technician

[Kalibro]

Hi,

The 82599ES is being used with an Ettus USRP x300 via a 10G SFP+ connection to stream UDP packets to our hardware RAID 6 system at a rate of 161 MegaBytes per second (the RAID can be written to at 6 GigaBytes per second). We are troubleshooting performance issues -- dropped packets on the Intel 82599ES as shown via 

ethtool -S enp4s0f1

and looking at the  rx_missed_errors stats. Several suggestions was to increase the ring buffers to the max of the 82599ES using the

ethtool -G enp4s0f1 rx 4096 tx 4096

, enable jumbo frames (MTU of 9000 for this particular USRP at 10Gbit connection), slightly increase the txqueuelen from 1000 to 2000,  and to also increase the Linux Kernel buffers i.e. 

sysctl -w net.core.wmem_max=200000000
sysctl -w net.core.rmem_max=200000000

After much research and talking with other USRP users, many have tweaked the PCIe bus / and or the network card by changing the MMRBC, MaxPayloadSize, MaxReadReq, etc. values, which lead to better throughput performance and no dropped packets.

However, I do not know to set change any such values, i.e. MMRBC, MaxPayloadSize, or MaxReadReq values on this Intel 82599ES.  I was wondering if there is a GUI Linux utility that shows these values and allows to easily change them.

Any help much appreciated!

[Mike_Intel]

Hello Kalibro,

Thank you so much for the quick update and more details about your end goal. Let me also gather more details about your system. Are you using a customer system with 82599ES? If no, please provide the brand and model of your system or mother board.

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

Best regards,

Michael L.

Intel® Customer Support Technician

[Kalibro]

Hello, 

Here are the system specs:

Motherboard:  ASUS Pro WS X570-ACE with BIOS v3801 (Aug 12, 2021) 

CPU:  AMD Ryzen 7 5700G

RAM:  16GB DDR4 2400 (one stick)

OS:  Linux Mint 20.2 w/ kernel version 5.11.0.-34-generic, Intel Linux driver ixgbe ver 5.12.5

OS Drive:  1TB SanDisk SSD

RAID:  Areca 1883ix-24, 12x 16TB Seagate Enterprise, configured into RAID 6.  Avg write speed:  6 GB/s. Avg read speed: 900MB/s

The 82599ES is confirmed to be plugged into a PCIe width of 8x --> LnkSta: Speed 5GT/s (ok), Width x8 (ok)

Please let me know how to change 82599ES values for MMRBC, MaxPayloadSize, MaxReadReq. Thank you.

[Mike_Intel]

Hello Kalibro,

Thank you so much for providing the details that we requested. I will have this one checked on how we can edit the MMRBC, MaxPayloadSize and MaxReadReq.

By the way, while doing this, can you also help provide the following as we may also need this to check later.

1. Can you provide some photos of the card 82599ES on both sides focusing on the markings?
2. Please help generate the SSU logs of the system for further checking as well. Kindly download the utility here:

https://www.intel.com/content/www/us/en/download/18895/intel-system-support-utility-for-the-linux-operating-system.html?wapkw=ssu

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

Best regards,

Michael L.

Intel® Customer Support Technician

[Kalibro]

Here you go:

[Mike_Intel]

Hello Kalibro,

Thank you for the update. Please give us 1 to 2 business days to provide an update on this. We will try to check on how you can edit MaxReadReq to 2048 bytes.

Best regards,

Michael L.

Intel® Customer Support Technician

[Mike_Intel]

Hello Kalibro,

By the way, I realize this while collecting the data. When you created the thread, you mentioned that your network card is Intel® 82599ES 10 Gigabit Ethernet Controller but the photos that you sent is for Intel® Ethernet Server Adapter X520-DA2. Kindly confirm which card is installed in your system.

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

Best regards,

Michael L.

Intel® Customer Support Technician

[Kalibro]

The card in the photos is the exact card that is installed in the system (I took the card out for taking the photos on the table). When I stick this card into the motherboard, the Linux Mint OS reports it to be:

04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

After looking at the datasheet for the  Intel® Ethernet Server Adapter X520-DA2, it says that the controller inside it is the 82599. So the answer is YES, the card installed in the motherboard is the Intel X520-DA2, and YES, it is Intel 82599ES as per controller and as per Linux 'lspci' command    

[Mike_Intel]

Hello Kalibro,

Thank you so much for the clarification. We will continue to further check your inquiry and please give us 1 to 2 business days to provide an update.

Best regards,

Michael L.

Intel® Customer Support Technician

[Mike_Intel · Accepted solution]

Hello Kalibro,

Thank you so much for patiently waiting. Here is the update from our Engineering team regarding your inquiry.

According to engineering, the specific register the you want to modify is at offset 08h (the same as 0x08 in my nomenclature). In the case of X520 our PCIe capabilities structure is located at offset 0xA0 from the base of config space. This is where the 0xA8 offset came from that you are referring to. So per our understanding the value at offset 0xA8 in hex for the Device Control Register on their device (based on the dump from lspci they provided) is 0x283F. To change the Max read request size to 2048 bytes the new value needs to be 0x483F.

To get the current value use this command:

setpci -s 04:00.1 a8.w

This should return 283f. If not, then something is not working as expected.

setpci -s 04:00.1 a8.w=483f

Then to verify do:

setpci -s 04:00.1 a8.w

and it should return 483f.

Then do a power off reboot for it to take effect.

We tried this with the X540, port 0. Here are excerpts of the lspci output before (a8=283f)

MaxPayload 256 bytes, MaxReadReq 512 bytes

a0: 10 00 02 00 c2 8c 00 10 3f 28 19 00 82 cc 41 02

An after (a8=483f)

MaxPayload 256 bytes, MaxReadReq 2048 bytes

a0: 10 00 02 00 c2 8c 00 10 3f 48 19 00 82 cc 41 02

Let us also set an expectation that our engineering can’t guarantee the same results they plan to do, however the mechanics as described above should do what they are asking for.

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

Best regards,

Michael L.

Intel® Customer Support Technician

[Kalibro]

Excellent! Thank you so much! 

[Mike_Intel]

Hello Kalibro,

Thank you so much for the confirmation and we are happy that our recommendation was able to help you. As for the thread, we will close this one now and if you need assistance again, please post a new question.

Thank you and stay safe.

Best regards,

Michael L.

Intel® Customer Support Technician

[Mike_Intel]

Hello Kalibro,

By the way, as we close this thread, we would like to thank you and also to apologize for the delay in providing the update. During that time, we had to coordinated this request to our higher level of engineers to provide an accurate update and steps. Thank you again for your kind patience and cooperation for the thread.  

Thank you and stay safe.

Best regards,

Michael L.

Intel® Customer Support Technician