- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
I have an Intel 82599ES Dual 10G network card, running on a Linux Mint 20.2, kernel version 5.11.0.-34-generic, Intel Linux driver ixgbe ver 5.12.5.
I am trying to figure out how to set the MaxReadReq parameter to the max value of 2048 bytes...it is currently set at 512 bytes. The Datasheet says that this value can be changed in the 0xA8 register, bits 14:12 to value of 0100b...however, I don't know how to go about doing this. I know that the linux command 'setpci' can change PCIe register values...but I don't know how to do this.
Here is the output from lspci:
04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 34
Region 0: Memory at e0300000 (64-bit, prefetchable) [size=512K]
Region 2: I/O ports at d000 [disabled] [size=32]
Region 4: Memory at e0400000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s (ok), Width x8 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR-
10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-, TPHComp-, ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [140 v1] Device Serial Number f8-f2-1e-ff-ff-b1-2d-08
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 01
VF offset: 384, stride: 2, Device ID: 10ed
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000e0700000 (64-bit, non-prefetchable)
Region 3: Memory at 00000000e0800000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: ixgbe
Kernel modules: ixgbe
And here is the hexadecimal map of the 82599ES:
04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
00: 86 80 fb 10 06 04 10 00 01 00 00 02 10 00 80 00
10: 0c 00 30 e0 00 00 00 00 01 d0 00 00 00 00 00 00
20: 0c 00 40 e0 00 00 00 00 00 00 00 00 86 80 03 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 02 00 00
40: 01 50 23 48 00 20 00 2b 00 00 00 00 00 00 00 00
50: 05 70 80 01 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 11 a0 3f 80 04 00 00 00 04 20 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 10 00 02 00 c2 8c 00 10 3f 28 09 00 82 c4 01 00
b0: 40 00 82 10 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 1f 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Does anyone know how to use setpci command to change the MaxReadReq of the 82599ES card to 2048 bytes ? Is there maybe a Linux GUI utility that can also do this ?
Thank you kindly for any help!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Kalibro,
Thank you so much for patiently waiting. Here is the update from our Engineering team regarding your inquiry.
According to engineering, the specific register the you want to modify is at offset 08h (the same as 0x08 in my nomenclature). In the case of X520 our PCIe capabilities structure is located at offset 0xA0 from the base of config space. This is where the 0xA8 offset came from that you are referring to. So per our understanding the value at offset 0xA8 in hex for the Device Control Register on their device (based on the dump from lspci they provided) is 0x283F. To change the Max read request size to 2048 bytes the new value needs to be 0x483F.
To get the current value use this command:
setpci -s 04:00.1 a8.w
This should return 283f. If not, then something is not working as expected.
setpci -s 04:00.1 a8.w=483f
Then to verify do:
setpci -s 04:00.1 a8.w
and it should return 483f.
Then do a power off reboot for it to take effect.
We tried this with the X540, port 0. Here are excerpts of the lspci output before (a8=283f)
MaxPayload 256 bytes, MaxReadReq 512 bytes
a0: 10 00 02 00 c2 8c 00 10 3f 28 19 00 82 cc 41 02
An after (a8=483f)
MaxPayload 256 bytes, MaxReadReq 2048 bytes
a0: 10 00 02 00 c2 8c 00 10 3f 48 19 00 82 cc 41 02
Let us also set an expectation that our engineering can’t guarantee the same results they plan to do, however the mechanics as described above should do what they are asking for.
If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.
Best regards,
Michael L.
Intel® Customer Support Technician
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Kalibro,
Thank you for posting in Intel Ethernet Communities.
Before we proceed, let me check if you are designing a board or system with embedded 82599ES? Can you share why do you need to edit the MaxReadReq.
If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.
Best regards,
Michael L.
Intel® Customer Support Technician
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
The 82599ES is being used with an Ettus USRP x300 via a 10G SFP+ connection to stream UDP packets to our hardware RAID 6 system at a rate of 161 MegaBytes per second (the RAID can be written to at 6 GigaBytes per second). We are troubleshooting performance issues -- dropped packets on the Intel 82599ES as shown via
ethtool -S enp4s0f1
and looking at the rx_missed_errors stats. Several suggestions was to increase the ring buffers to the max of the 82599ES using the
ethtool -G enp4s0f1 rx 4096 tx 4096
, enable jumbo frames (MTU of 9000 for this particular USRP at 10Gbit connection), slightly increase the txqueuelen from 1000 to 2000, and to also increase the Linux Kernel buffers i.e.
sysctl -w net.core.wmem_max=200000000
sysctl -w net.core.rmem_max=200000000
After much research and talking with other USRP users, many have tweaked the PCIe bus / and or the network card by changing the MMRBC, MaxPayloadSize, MaxReadReq, etc. values, which lead to better throughput performance and no dropped packets.
However, I do not know to set change any such values, i.e. MMRBC, MaxPayloadSize, or MaxReadReq values on this Intel 82599ES. I was wondering if there is a GUI Linux utility that shows these values and allows to easily change them.
Any help much appreciated!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Kalibro,
Thank you so much for the quick update and more details about your end goal. Let me also gather more details about your system. Are you using a customer system with 82599ES? If no, please provide the brand and model of your system or mother board.
If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.
Best regards,
Michael L.
Intel® Customer Support Technician
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
Here are the system specs:
Motherboard: ASUS Pro WS X570-ACE with BIOS v3801 (Aug 12, 2021)
CPU: AMD Ryzen 7 5700G
RAM: 16GB DDR4 2400 (one stick)
OS: Linux Mint 20.2 w/ kernel version 5.11.0.-34-generic, Intel Linux driver ixgbe ver 5.12.5
OS Drive: 1TB SanDisk SSD
RAID: Areca 1883ix-24, 12x 16TB Seagate Enterprise, configured into RAID 6. Avg write speed: 6 GB/s. Avg read speed: 900MB/s
The 82599ES is confirmed to be plugged into a PCIe width of 8x --> LnkSta: Speed 5GT/s (ok), Width x8 (ok)
Please let me know how to change 82599ES values for MMRBC, MaxPayloadSize, MaxReadReq. Thank you.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Kalibro,
Thank you so much for providing the details that we requested. I will have this one checked on how we can edit the MMRBC, MaxPayloadSize and MaxReadReq.
By the way, while doing this, can you also help provide the following as we may also need this to check later.
- Can you provide some photos of the card 82599ES on both sides focusing on the markings?
- Please help generate the SSU logs of the system for further checking as well. Kindly download the utility here:
If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.
Best regards,
Michael L.
Intel® Customer Support Technician
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Kalibro,
Thank you for the update. Please give us 1 to 2 business days to provide an update on this. We will try to check on how you can edit MaxReadReq to 2048 bytes.
Best regards,
Michael L.
Intel® Customer Support Technician
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Kalibro,
By the way, I realize this while collecting the data. When you created the thread, you mentioned that your network card is Intel® 82599ES 10 Gigabit Ethernet Controller but the photos that you sent is for Intel® Ethernet Server Adapter X520-DA2. Kindly confirm which card is installed in your system.
If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.
Best regards,
Michael L.
Intel® Customer Support Technician
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The card in the photos is the exact card that is installed in the system (I took the card out for taking the photos on the table). When I stick this card into the motherboard, the Linux Mint OS reports it to be:
04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
After looking at the datasheet for the Intel® Ethernet Server Adapter X520-DA2, it says that the controller inside it is the 82599. So the answer is YES, the card installed in the motherboard is the Intel X520-DA2, and YES, it is Intel 82599ES as per controller and as per Linux 'lspci' command
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Kalibro,
Thank you so much for the clarification. We will continue to further check your inquiry and please give us 1 to 2 business days to provide an update.
Best regards,
Michael L.
Intel® Customer Support Technician
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Kalibro,
Thank you so much for patiently waiting. Here is the update from our Engineering team regarding your inquiry.
According to engineering, the specific register the you want to modify is at offset 08h (the same as 0x08 in my nomenclature). In the case of X520 our PCIe capabilities structure is located at offset 0xA0 from the base of config space. This is where the 0xA8 offset came from that you are referring to. So per our understanding the value at offset 0xA8 in hex for the Device Control Register on their device (based on the dump from lspci they provided) is 0x283F. To change the Max read request size to 2048 bytes the new value needs to be 0x483F.
To get the current value use this command:
setpci -s 04:00.1 a8.w
This should return 283f. If not, then something is not working as expected.
setpci -s 04:00.1 a8.w=483f
Then to verify do:
setpci -s 04:00.1 a8.w
and it should return 483f.
Then do a power off reboot for it to take effect.
We tried this with the X540, port 0. Here are excerpts of the lspci output before (a8=283f)
MaxPayload 256 bytes, MaxReadReq 512 bytes
a0: 10 00 02 00 c2 8c 00 10 3f 28 19 00 82 cc 41 02
An after (a8=483f)
MaxPayload 256 bytes, MaxReadReq 2048 bytes
a0: 10 00 02 00 c2 8c 00 10 3f 48 19 00 82 cc 41 02
Let us also set an expectation that our engineering can’t guarantee the same results they plan to do, however the mechanics as described above should do what they are asking for.
If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.
Best regards,
Michael L.
Intel® Customer Support Technician
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Kalibro,
Thank you so much for the confirmation and we are happy that our recommendation was able to help you. As for the thread, we will close this one now and if you need assistance again, please post a new question.
Thank you and stay safe.
Best regards,
Michael L.
Intel® Customer Support Technician
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Kalibro,
By the way, as we close this thread, we would like to thank you and also to apologize for the delay in providing the update. During that time, we had to coordinated this request to our higher level of engineers to provide an accurate update and steps. Thank you again for your kind patience and cooperation for the thread.
Thank you and stay safe.
Best regards,
Michael L.
Intel® Customer Support Technician
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page