TCP receive packet latency

[Bombey__Jasper]

Running on a linux system with Intel 82599ES 10-Gigabit device and ixgbe driver.

Using tcpdump I'm intermittently seeing cases where there is more than 5ms between a tcp packet being received and the tcp ack being sent. The application's socket receive call also receives the tcp packet at roughly the same time that the ack is sent. I'm not sure exactly what point tcpdump sniffs the data, but I'm assuming it's in the hardware before it gets passed to the kernel.  I'd like to know what could cause this latency?  Are there monitoring tools I could use to detect when this condition occurs and/or determine where the latency is occurring?

[AlfredoS_Intel]

Hi Bombey__jasper,

Thank you for posting in our Intel® Ethernet Communities Page.

We are sorry to hear about the issue that you are experiencing with your network adapter. 

So we would have a better understanding of your issue, please provide the following information:

1. Please download and run our Intel® System Support Utility from this page, https://downloadcenter.intel.com/download/26735/Intel-System-Support-Utility-for-the-Linux-Operating-System#:~:text=Intel%20SSU%20for%20the%20Linux,and%20shared%20by%20the%20user. After running it, you will be given an option to save the logs to a text file, please do so and attach the file on your reply.

2. Kindly provide us the results of this command: ethtool -i ethx where ethx is the Ethernet port.

We look forward to hearing from you. If we do not get your reply, we will follow up after 3 business days.

Best Regards,

Alfred S

Intel® Customer Support

[Bombey__Jasper]

ethtool -i results
driver: ixgbe
version: 5.1.0-k-rh7.7
firmware-version: 0x800006da, 1.1824.0
expansion-rom-version:
bus-info: 0000:81:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Attached please find the output of ssu.sh. It said my os not compatible so if you need additional info, other than what was captured, please let me know.

[AlfredoS_Intel]

Hi Bombey__jasper,

We are just following up.

It looks like you need more time to assess the answers that we have provided.

We will follow up again after 3 business days. Should we not hear from you, our system may automatically close the thread.

Best Regards,

Alfred S

Intel Customer Support

[AlfredoS_Intel]

Hi Bombey__jasper,

Thank you for your response and for finding the time to gather those logs for us.

We have analyzed the logs and please allow us to clarify some things:

1. The logs doesn’t correctly display the Operating system and this could be connected to the error that you got when ran our tool. May we know the version and type of Linux that you are running on the system?

2. We also noticed that the network adapter that is detected is an i350 network adapter not the Intel® 82599ES 10 Gigabit Ethernet Controller. Are you reporting for the i350 network adapter instead?

3. May we know if your experience the TCP latency what problems does it give you on the applications on the system?

We look forward to your reply. Should we not get your reply, we will follow up after three business days.

Best Regards,

Alfred S

Intel® Customer Support

[Bombey__Jasper]

1. I'm running version: Red Hat Enterprise Linux Server release 7.8 (Maipo)

2.  I'm running with both network cards, but my issue is with the 10G interfaces:

# lspci | egrep -i --color 'network|ethernet'
03:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

3.  The application intermittently takes longer to complete.  for example for some tasks that normally take 1ms to complete, we sometimes see them take  20ms and above.  The tasks are repeating the same work over and over so we'd expect the completion time to be consistent.

[AlfredoS_Intel]

Hi Bombey__jasper,

Thank you for your response and for clarifying the things that we saw from the logs.

After reading the answers that you have provided, please allow us to ask additional questions:

1. Are both i350 and 82599ES built-in or integrated on the server board?

2. If both network adapters are built-in on the board, drivers and firmware that could mitigate the issue are usually available on the server board manufacturers website. May we ask if you have already checked and reported the issue with Supermicro? If yes, may we ask what their suggestion is?

We look forward to your reply. Should we not get your reply, we will follow up after three business days.

Best Regards,

Alfred S

Intel® Customer Support

[AlfredoS_Intel]

Hi Bombey__jasper,

Thank you for your response and for clarifying the things that we saw from the logs.

After reading the answers that you have provided, please allow us to ask additional questions:

1. Are both i350 and 82599ES built-in or integrated on the server board?

2. If both network adapters are built-in on the board, drivers and firmware that could mitigate the issue are usually available on the server board manufacturers website. May we ask if you have already checked and reported the issue with Supermicro? If yes, may we ask what their suggestion is regarding the issue?

We look forward to your reply. Should we not get your reply, we will follow up after three business days.

Best Regards,

Alfred S

Intel® Customer Support

[AlfredoS_Intel]

Hi Bombey__jasper,

We are just following up.

It looks like you need more time to provide the information that we asked.

We will follow up again after 3 business days. Should we not hear from you, our system may automatically close the thread.

Best Regards,

Alfred S

Intel Customer Support

[AlfredoS_Intel]

Hi Bombey__jasper, 

We need to close this thread since we have not gotten a response from you: maybe because you are busy or preoccupied at the moment. We know that this is important for you to get it resolved and it is also equally important for us to give you the right solution; as much as we would like to assist you, we need to close it to attend to other customers. We hope for your consideration and understanding on this one.

If you need any additional information, please submit a new question as this thread will no longer being monitored.

Thank you for contacting Intel® and have a great week!

Best Regards,

Alfred S

Intel® Customer Support