topic Re: I218-LM - Delayed frames in Embedded Connectivity

I218-LM - Delayed frames

ipe_age — Thu, 13 Aug 2020 09:35:22 GMT

Hi,

We noticed some behaviour we cannot explain when using the Intel I218-LM ethernet interface under Linux (Debian Buster, Kernel 5.2.17) with the in-tree e1000e driver, running on an Intel Core i3-4010U CPU.

We have a situation with very high CPU load on all cores (multiple processes using full CPU) while also writing large amounts of data to disk. The ethernet interface receives data at a constant rate of about 1000 frames or 400 kB per second.

In this situation from time to time no ethernet frames are processed for up to 500 ms, equivalent to about 200 kB. However, no data is lost, and all missing frames are processed after the pause.

According to the data sheet there doesn’t seem to be enough memory on the ethernet controller to store the whole 200 kB. Therefore, we suppose that the frames are not buffered on the interface during the pause but are still transferred to host memory.

We created a custom kernel with debug output in the ethernet driver in “drivers/net/ethernet/intel/e1000e/netdev.c” to further analyse this problem.

The “e1000_clean_rx_irq” function is still called regularly (< 50 ms), which seems to imply that the problem isn’t related to interrupt handling. However, the “while (staterr & E1000_RXD_STAT_DD)” loop is not entered for up to 500 ms.

This left us wondering why the ethernet controller might transfer the data into RAM but not set the DD flag. As the problem seems to occur only while writing large amounts of data to disk, we assume it might be caused by lots of DMA access. But why is the DD flag not set if the data is already in RAM?

Thank you for your help,
Armin

Re:I218-LM - Delayed frames

Caguicla_Intel — Mon, 17 Aug 2020 07:39:43 GMT

Hello ipe_age,

Thank you for posting in Intel Ethernet Communities and we sincerely apologize for the delay on our response.

Please allow 1-3 business days for us to check on your request. Rest assured that we will provide an update once there is any findings.

Hoping for your kind patience.

Best regards,

Crisselle C

Intel® Customer Support

Re:I218-LM - Delayed frames

Caguicla_Intel — Wed, 19 Aug 2020 06:23:37 GMT

Hello ipe_age,

Good day!

Before we further check your query with our higher level, can you share the model of the motherboard where the I218-LM was embedded? Please also provide the tool used in monitoring this behavior. This would help us in checking your query.

Hoping to hear from you soon.

Should there be no reply, we will follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

Re: Re:I218-LM - Delayed frames

ipe_age — Fri, 21 Aug 2020 11:55:48 GMT

Hello,

Thank you for your answer.

We use a conga-TC87 motherboard from Congatec. It is part of an embedded measurement-system, that runs our own measurement software.

The problem was detected while using this custom measurement application. Ethernet data was acquired by this application using a UDP-socket. We then noticed, that expected frames were sometimes processed too late.

Triggered by that observation we started debugging this behaviour on kernel level as described in my original post. I have attached the patches we added to the kernel and the generated kernel log. Maybe this is helpful to you. If you need more information, please feel free to ask.

Thank your for your help,
Armin

Re: I218-LM - Delayed frames

Caguicla_Intel — Mon, 24 Aug 2020 03:37:55 GMT

Hello ipe_age,

Thank you for the reply.

Please allow us to continue checking on your request. Rest assured that we will provide an update as soon as possible but no later than 2-3 business days.

Hoping for your kind patience.

Best regards,

Crisselle C

Intel® Customer Support

Re:I218-LM - Delayed frames

Caguicla_Intel — Tue, 25 Aug 2020 06:08:07 GMT

Hello Armin,

Good day!

While we are checking on this, can you extract System Support Utility (SSU) log from your system? This will allow us to check your adapter details and configuration.

Please refer to this link for the instructions on how to extract the SSU logs.

Help Guide for the Intel® System Support Utility

https://www.intel.com/content/www/us/en/support/articles/000008563/software/software-applications.html

Looking forward to your reply.

Should there be no response, I’ll make sure to reach out after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

Re: Re:I218-LM - Delayed frames

ipe_age — Thu, 27 Aug 2020 07:09:36 GMT

Hi,

we used your tool to create the log file. I have added it as an attachment. The "xrandr" command was not installed on our system however, as we do not have any graphical user interface on this machine.

Thanks for your help,
Armin

Re:I218-LM - Delayed frames

Caguicla_Intel — Fri, 28 Aug 2020 01:18:32 GMT

Hello Armin,

Thank you for the reply.

Please allow us to further check on this matter. We will get back to you as soon as possible but no later than 2-3 business days.

Hoping for your kind patience.

Best regards,

Crisselle C

Intel® Customer Support

Re:I218-LM - Delayed frames

Caguicla_Intel — Mon, 31 Aug 2020 06:45:08 GMT

Hello Armin,

Good day!

We hope you don't mind sharing the additional information below that would help us in investigating the issue.

1.) Have you tried using the latest driver of I218-LM from our Intel download site or from Congatec? If yes, can you share if they have the same behavior? If no, is there any reason why do you want to use in-tree driver?

2.) You mentioned that you use your own measurement software, can you please share if you have tried using different tools to further isolate the issue?

Looking forward to your reply.

We will follow up after 3 business days in case we don't hear from you.

Best regards,

Crisselle C

Intel® Customer Support

Re: Re:I218-LM - Delayed frames

ipe_age — Tue, 01 Sep 2020 08:04:50 GMT

Hi Crisselle,

1. We will try to install the newest driver on our system and check if the problem still persists.

2. The problem can also be seen in the kernel log messages we added. However, it only seems to happen in specific situations that include our software being executed. We think it might have something to do with the specific memory or IO access patterns involved. But this is hard to confirm as we do not understand the possible reasons why the data might not be processed. That's why we asked the question in the first place. We hope that someone can explain, what is actually happening from the point of view of the ethernet controller.

Thank you for your help,
Armin

Re:I218-LM - Delayed frames

Caguicla_Intel — Wed, 02 Sep 2020 02:43:29 GMT

Hello Armin,

Thank you for the reply.

We will patiently wait for your update once you've tried the latest driver. Once we heard from you, we will see for the results and check if this still needs to be escalated to our higher level for further investigation.

Awaiting to hear from you.

Should there be no reply, we will follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

Re: Re:I218-LM - Delayed frames

ipe_age — Thu, 03 Sep 2020 11:02:22 GMT

Hi Crisselle,

we now managed to install the newest e1000e driver (3.8.4) and repeat our tests. Unfortunately the problem still persists.

Thanks for your help,
Armin

Re:I218-LM - Delayed frames

Caguicla_Intel — Fri, 04 Sep 2020 03:24:03 GMT

Hello Armin,

Thank you for the update.

Please allow us to further check this with our engineers. We will provide an update as soon as possible but no later than 2-3 business days.

Hoping for your kind patience.

Best regards,

Crisselle C

Intel® Customer Support

Re:I218-LM - Delayed frames

Caguicla_Intel — Wed, 09 Sep 2020 02:53:22 GMT

Hello Armin,

Thank you for the patience on this matter.

Please be informed that your query will be best answered by our Embedded Connectivity Support team since your question is more on design in/controller on your embedded measurement-system. We will help you to move this post to the designated team.

Please feel free to contact us if you need assistance from Ethernet support team.

May you have a great day!

Best regards,

Crisselle C

Intel® Customer Support

Re: I218-LM - Delayed frames

CarlosAM_INTEL — Wed, 09 Sep 2020 18:12:33 GMT

Hello, @ipe_age:

Thank you for contacting Intel Embedded Community.

Based on your previous communications, we suggest verifying that your implementation has been validated and tested by the developer of the motherboard related to this situation.

Best regards,

@CarlosAM_INTEL.

Re: I218-LM - Delayed frames

ipe_age — Mon, 14 Sep 2020 15:00:43 GMT

Hi,

thanks for your answers.

Our main problem currently is that the effect is quite hard to reproduce. It seems to depend on circumstances not directly related to the Ethernet interface like CPU load or writing to external storage media. This makes it quite hard to pinpoint which part is actually responsible for the observed behavior. It might be caused by some other hardware blocking DMA access or even by our own software.

Currently our goal is to reproduce the effect more consistently and isolate the problem further. To do so, we try to better understand what might cause the DD flag to not be set or maybe the data to not be written.

Our primary question at this time is: Which hardware, firmware and/or software steps are exactly involved and required until the driver can observe the changed DD flag?

We hoped that someone with more knowledge about the Ethernet controller could give us some hints towards possible causes. We were posting here not to have our problem fixed, but to get further information, in which direction we might have to perform further tests.

Thank you for your help,
Armin

Re: I218-LM - Delayed frames

CarlosAM_INTEL — Mon, 14 Sep 2020 19:44:25 GMT

Hello, @ipe_age:

Thanks for your update.

Based on our last communication, could you please confirm that you have contacted the manufacturer of the affected design to verify our previous suggestion?

Best regards,

@CarlosAM_INTEL.