Why I have loss during capture data?

[Miladdona]

Hi everyone,
I'm working with Intel(R) Ethernet CNA x710 DA4 FH that has 4 ports and I want to capture high rate data (more than 3 Gb per second per port). First, I tested to capture 3.3 Gb rate data from one port. It was good without any loss. Fig.1 shows the CPU cores load. But when I want to capture data from two or more ports simultaneously, I have a lot of losses, also CPU load is just on one core. Figs 2-4 show CPU load for 2, 3, and 4 port data capturing respectively.
anyone can help me to distribute load on different cores? I think it can remove losses.
Or how can I capture data without loss?
I work by Windows server 2016 OS, WinPcap 4.1.3 to capture data, Visual studio 2013 and C program.
Fig.5 shows my Ethernet card feature.
Thank you in advance.

[Caguicla_Intel]

Hello Milad

I hope this message finds you well!

We just noticed that you are using NVM Version:"8.10" on your X710-DA2. Will you be able to try updating it to latest 8.15 and retest if it will still encounter the same issue? This is the time that we will be able to test the latest firmware since the X710-DA4 could not be updated.

Based on SSU with Part Number:"J11367-005"

NVM Version:"8.10"

Here is the download link for Non-Volatile Memory (NVM) Update Utility for Intel® Ethernet Network Adapter 700 Series Version: 8.15 (Latest)

https://downloadcenter.intel.com/download/24769/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Ethernet-Network-Adapter-700-Series

Awaiting to hear from you.

We will reach out after 3 business days in case we don't receive a reply.

Best regards,

Crisselle C

Intel® Customer Support

[Mike_Intel]

Hello Milad,

I hope that you are having a good day. We just want to check if you tried our last recommendation.

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

Best regards,

Michael L.

Intel® Customer Support

[Miladdona]

Hi,

I apologize for the delay.

I installed the new version of NVM (8.15) but I got the same results.

I mean I couldn't update it correctly. I got the results for DA2 as I installed NVM for DA4!

Could you please let me know if you could test DA4 (DA2) with your device and config.

Thank you in advance.

Best regards,

Miladdona

[Mike_Intel]

Hello Milad,

I am really sorry, can you further explain what happen during the update of DA2?

Are you having the same issue? Can you share a screenshot?

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

Best regards,

Michael L.

Intel® Customer Support

[Miladdona]

Hi,

I downloaded the latest version of NVM from the link that you sent me in the previous posts:

https://downloadcenter.intel.com/download/24769/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Ethernet-Network-Adapter-700-Series

I extracted it for windows and install but I am still having the same issue!

I sent you a screenshot. Please find attached!

Thank you in advance.

Best regards,

Miladdona

[Mike_Intel]

Hello Milad,

Thank you for the clarification. We need to check the issue and get back to you once we have an update or a recommendation.

Please give us 3 to 4 working days for us to further investigate the issue.

Thank you.

Best regards,

Michael L.

Intel® Customer Support

[Caguicla_Intel]

Hello Miladdona,

I hope you are doing good!

Please be informed that this request is currently escalated to our engineering level for further investigation. Rest assured that we will provide an update as soon as we heard from them but no later than 2-4 business days.

Hoping for your kind patience.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello Miladdona,

Thank you for the patience on this matter.

While this issue is still under investigation, our engineers noticed that both cards are installed in the system. With this, we hope you don't mind removing the first card (X710-DA4) and reinstall the drivers for the X710-DA2 (clean up as much as possible, fully uninstall) then also try SW 25.6 when reinstalling the card. 

Intel® Ethernet Adapter Complete Driver Pack

https://downloadcenter.intel.com/download/22283/Intel-Ethernet-Adapter-Complete-Driver-Pack

Please feel free to let us know of the results once the recommendations above are performed.

Looking forward to your reply.

Should there be no response from you, we'll make sure to reach out after 3 business days.

Best regards,

Crisselle C.

Intel Customer Support

[Caguicla_Intel]

Hello Miladdona,

Good day!

We'd like to check if you ate able to read our previous message? If yes, kindly advise if you are able to perform our recommendations and share some update on this one.

Looking forward to your reply,

We will follow up after 3 business days in case we don't receive a reply.

Best regards,

Crisselle C.

Intel Customer Support

[Caguicla_Intel]

Hello Miladdona,

How are you doing?

Please let us know if you are able to perform our suggestions and feel free to share the result with us after testing. 

Our Engineering team would also like to have the SEL (System Event Log) for this, they want to check if the card is installed in a PCI slot that is capable of X8. We hope you don't mind providing this for further investigation.

Hoping to hear from you soon.

We'll make sure to reach out after 3 business days in case we don't receive a reply.

Best regards,

Crisselle C.

Intel Customer Support

[Miladdona]

Hi,

Sorry for the delay!

I made some differences. I had Windows server 2016 on a HDD before. Now, I install a SSD and install Windows server 2016 on it.

I removed the first card (X710-DA4) and only install the X710-DA2. After downloading the SW 25.6, I installed it by the link you sent me:

https://downloadcenter.intel.com/download/22283/Intel-Ethernet-Adapter-Complete-Driver-Pack

I also updated the NVM! You can see it as an image in attach.

I got SSU and SEL and sent you. Please find attached.

Unfortunately I still have loss!

Thank you in advance.

Best regards,

Miladdona

[Caguicla_Intel]

Hello Miladdona,

No problem at all. Appreciate your effort in trying out the recommended steps and sharing the information requested.

Please allow us to continue investigating this with our engineers. We will get back to you as soon as we heard from them but no later than 2-4 business days.

Hoping for your kind patience.

Best regards,

Crisselle C.

Intel Customer Support

[Caguicla_Intel]

Hello Miladdona,

Apologies for the delay on this matter.

We'd like to inform you that we are still actively working on this issue with our higher level Engineers. Rest assured that we are doing our best to sort this out as soon as possible. We will get back to you for an update within 2-4 business days.

Thank you for your kind understanding!

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello Miladdona,

Thank you for the patience on this matter.

After further validation, our engineers did not find anything in the Window's Event Log related to the issue. Also, they are unable to investigate this issue using your custom C application, but if you can reproduce the issue using an industry standard tool, they can investigate it further.

Let us know if you have additional questions and clarifications.

Looking forward to your reply.

We'll make sure to reach out after 3 business days in case we don't receive a reply.

Best regards,

Crisselle C.

Intel Customer Support

[Miladdona]

Hi,

Thank you for your effort!

Do you know any industry standard tool?

My C application is only a way, please let me know if you know any good ways.

I only want to get from all ports a high-rate data (e.g. 9 Gbps).

Thank you in advance.

Best regards,

Miladdona

[Caguicla_Intel]

Hello Miladdona,

Appreciate your swift response.

Please allow me to check with our engineers if they can suggest any industry standard tool. Rest assured that we will get back to you within 1-3 business days.

Best regards,

Crisselle C.

Intel Customer Support

[Caguicla_Intel]

Hello Miladdona,

Good day!

Will you be able to capture the issue using Wireshark Trace and provide us the logs? (Wireshark trace with .cap file) Once we have the logs, we will share it with our engineers if they can further check and confirm the issue.

Looking forward to hear from you.

Should there be no response, I’ll make sure to reach out after 3 business days.

Best regards,

Crisselle C.

Intel Customer Support

[Miladdona]

Hi,

Thank you for your response.

I have checked the Wireshark before but it is not a good tool for high rate! It doesn't work well in this case.

I did my test with my C application with a Solarflare Ethernet without loss (high rate) but I have had loss by Wireshark.

Could you please tell me how to test your cards? I want to get high rate data continuously (for example for one hour).

Do you have any other tool?

Thank you in advance.

Best regards,

Miladdona

[Caguicla_Intel]

Hello Miladdona,

You're welcome.

Please allow us to forward your reply to our engineers. We will get back to you as soon as we heard from them but no later than 2-4 business days.

Best regards,

Crisselle C.

Intel Customer Support

[Caguicla_Intel]

Hello Miladdona,

Good day!

We'd like to inform you that we are still actively working on this issue with our higher level Engineers. Rest assured that we will keep you posted and get back to you within 2-4 business days.

Hoping for your kind patience.

Best regards,

Crisselle C.

Intel Customer Support

[Caguicla_Intel]

Hello Miladdona,

Thank you for the patience on this matter.

Kindly refer to below information for the update from our engineering team.

Here are some tuning notes that should be helpful in getting multi port performance to improve in window.

Please refer to the detailed documentation in the Ethernet User Guide (https://downloadcenter.intel.com/download/28410/Adapter-User-Guide-for-Intel-Ethernet-Adapters) in a section called “Optimizing Performance.”

Here are the major tunable parameters:

Adjust RSS settings (controlled through RSS configuration, see https://docs.microsoft.com/en-us/powershell/module/netadapter/set-netadapterrss?view=win10-ps)

For multiple adapter ports, pin queues to non-overlapping CPU cores. Avoid core 0 since it performs system tasks. Increase number of RSS queues if needed. (in this case, this may be difficult as you only have 8 cores available on the client processor)

Example:

Get-NetAdapter -InterfaceDescription *X710*

Set-NetAdapterRss -name <adapter name> -NumberOfReceiveQueues $numrss -BaseProcessorNumber $numoffset -BaseProcessorGroup $localnode

Adjust RSS numa profile to "Closest" to allow more efficient processing of queues on the local numa node.

Example:

Set-NetAdapterRss -name <adapter name> -Profile Closest

Running more application threads will allow more queues and CPU cores to be used.

Most relevant section from the user guide:

Visit page 40 "RSS Queues" of Ethernet User Guide 

Awaiting to hear from you.

Should there be no response, I’ll make sure to reach out after 3 business days.

Best regards,

Crisselle C.

Intel Customer Support