Why I have loss during capture data?

[Miladdona]

Hi everyone,
I'm working with Intel(R) Ethernet CNA x710 DA4 FH that has 4 ports and I want to capture high rate data (more than 3 Gb per second per port). First, I tested to capture 3.3 Gb rate data from one port. It was good without any loss. Fig.1 shows the CPU cores load. But when I want to capture data from two or more ports simultaneously, I have a lot of losses, also CPU load is just on one core. Figs 2-4 show CPU load for 2, 3, and 4 port data capturing respectively.
anyone can help me to distribute load on different cores? I think it can remove losses.
Or how can I capture data without loss?
I work by Windows server 2016 OS, WinPcap 4.1.3 to capture data, Visual studio 2013 and C program.
Fig.5 shows my Ethernet card feature.
Thank you in advance.

[Caguicla_Intel]

Hello Miladdona,

Thank you for posting in Intel Ethernet Communities. 

Please share the driver and firmware version used and confirm if these were downloaded from the link below.

Downloads for Intel® Ethernet Converged Network Adapter X710-DA4

https://downloadcenter.intel.com/product/83965/Intel-Ethernet-Converged-Network-Adapter-X710-DA4

For our record purposes, kindly provide the brand and model of your motherboard.

We hope to hear from you soon. 

Should there be no response from you, I’ll make sure to reach out after 3 business days. 

Best regards,

Crisselle Franz

Intel® Customer Support Technician

[Miladdona]

Thank you for answer.

My driver is: Intel® Network Adapter Driver for Windows Server 2016* (version 25.2 Latest)

and my firmware is: Non-Volatile Memory (NVM) Update Utility for Intel® Ethernet Adapters 700 Series—Windows* (version 6.80 Previously Released)

and downloaded from your link for windows server 2016:

https://downloadcenter.intel.com/product/83965/Intel-Ethernet-Converged-Network-Adapter-X710-DA4

My motherboard brand and model is: ASUS X299 SAJE

and my processor is: Intel(R) Core(TM) i7-9800X CPU @ 3.80GHz

Thank you in advance.

Best regards,

Miladdona

[Caguicla_Intel]

Hello Miladdona,

You're welcome, appreciate your swift response.

While we are checking your request, can you confirm if the Receive Side Scaling (RSS) is the feature that you are looking for when you say 'anyone can help me to distribute load on different cores?' We may also need to further check if RSS is indeed supported on Intel® Ethernet Converged Network Adapter X710-DA4 and Windows Server 2016 and give you an update. Please refer to the link below for more information regarding Receive Side Scaling on Intel® Network Adapters

https://www.intel.com/content/www/us/en/support/articles/000006703/network-and-i-o/ethernet-products.html 

You may also visit this link and scroll down to Receive Side Scaling and Receive Side Scaling queues for additional details.

Advanced Settings for Intel® Ethernet Adapters

https://www.intel.com/content/www/us/en/support/articles/000005593/network-and-i-o/ethernet-products.html

Your prompt reply is highly appreciated.

Should there be no response, I’ll make sure to reach out after 3 business days.

Best regards,

Crisselle Franz

Intel® Customer Support Technician

[Miladdona]

Thank you for answer.

You're right! I checked the RSS (it's enabled) and changed the RSS Base Processor Number for each connection (different processor number for each connection). Now, I can distribute load on different cores but still I have loss. Fig.1 show the load on my system.

I really don't know how can I fix the loss!

Thank you in advance for your time and consideration.

Best regards,

Miladdona

[Caguicla_Intel]

Hello Miladdona,

You're welcome, glad to be of help. 

Can you share the following information that would help us in further investigating the issue? 

1.) Is the Flow control configuration on the switch and network adapter set to "Rx & Tx Enabled"?

2.) System Support Utility log of your system. This will allow us to check your Adapter details and configuration. Kindly refer to the steps below.

a- Go to https://downloadcenter.intel.com/download/25293/Intel-System-Support-Utility-for-Windows-?product=91600

b- Open SSU.exe

c- Mark the box "Everything" and then click "Scan".

d- When finished scanning, click "Next".

e- Click on "Save" and attach the file to a post.

3.) Will you be able to provide logs from your Packet analyzer? 

4.) Have you tried using a different tool in analyzing the packets to check if they have the same results.

5.) What is the cable type and module used. Please be advised that we have a list of Compatible SFP+ Modules, SFP Modules, and Cables for Intel® Ethernet Server Adapter X710 Series. Can you confirm if the cable and module used are listed on the this link? 

https://www.intel.com/content/www/us/en/support/articles/000007045/network-and-i-o/ethernet-products.html

Awaiting to hear from you.

I’ll make sure to reach out after 3 business days in case we don't hear from you.

Best regards,

Crisselle Franz

Intel® Customer Support Technician

[Miladdona]

Hi,

In my system network adapter is set to "Rx & Tx Enabled" and sent you my system log by "SSU".
I use "cisco SFP-10G-SR" module,
and
"NEXANS ZC LSZH FIBER OPTICAL CAVLE MM 50/125UM OM3" cable.
My SFP modules are not in the list (because Intel® Ethernet Server Adapter X710 Series just support Intel brands of SFP) but when I capture data from one port I didn't have loss!

Thank you in advance.

Best regards,

Miladdona

[Caguicla_Intel]

Hello Miladdona,

Thank you for providing the requested information.

Before we further check, can you share if your already tried the newly release driver from SW 25.4 package? If not yet, will you be able to try it out and check if this will help in packet loss issue? Latest driver for X710 from this package is 1.13.104.0

Intel® Ethernet Adapter Complete Driver Pack

https://downloadcenter.intel.com/download/22283/Intel-Ethernet-Adapter-Complete-Driver-Pack

Hoping to hear from you. 

In case we don't receive a reply, we will follow up after 3 business days.

Best regards,

Crisselle Franz

Intel® Customer Support

[Miladdona]

Hi,

Thank you for answer.

Driver for X710 on my system was 1.12.177.0.

I downloaded Intel® Ethernet Adapter Complete Driver Pack from your link and installed it, now driver for X710 on my system is 1.13.104.0, but I still have loss during capture data.

What should I do?

Thank you in advance for your consideration.

Best regards,

Miladdona

[Caguicla_Intel]

Hello Miladdona,

Thank you for trying out the latest driver and we are sorry to heat that it didn't help in resolving the issue. 

Can you share more information about your testing setup? Is it a peer to peer connection? Have you tried installing the adapter on a different PCIe slot?

You may consider visiting the link below for some recommendations on Tuning Throughput Performance for Intel® Ethernet Adapters

https://www.intel.com/content/www/us/en/support/articles/000005811/network-and-i-o/ethernet-products.html

Looking forward to your reply. 

Should there be no response, I’ll make sure to reach out after 3 business days.

Best regards,

Crisselle Franz

Intel® Customer Support

[Miladdona]

Hi,

Yes, my connection is peer to peer and I have tested the adapter on different PCIe slot.
I set the below features:
Interrupt Moderation Rate = off
Jumbo Frames = 9014
Flow Control = Rx & Tx Enabled
Transmit Descriptors buffer = 4096
Receive Descriptors buffer = 4096

But I still have loss!!!

I really do not know what can I do!

Thank you in advance for your consideration.
Best regards,
Miladdona

[Caguicla_Intel]

Hello Miladdona,

Thank you for the effort in trying out the recommended steps.

Please allow us to further check this on our end. We will give you an update as soon as possible but no later than 3 business days.

Hoping for your kind patience.

Best regards,

Crisselle Franz

Intel® Customer Support

[Caguicla_Intel]

Hello Miladdona,

Good day!

Before we further check this with our engineers, we hope you don’t mind sharing the following that would help with the investigation. 

1.Please ensure that Flow Control option is also enabled on the switch port.

2. Can you confirm that there is no data loss only if 1 of the 4 ports is tested? Regardless which port is it as long as the data was captured from only 1 port? 

3.Have you tried using a different cable?

4.Have you tried connecting the Intel Ethernet adapter to another switch port or another switch.

5.Is it possible to update the NVM to its latest version to check if it will help with the issue? 

Refer to the link below on How to update the firmware of an Intel® Ethernet Converged Network Adapter X710 Series Card

https://www.intel.com/content/www/us/en/support/articles/000030492/network-and-i-o/ethernet-products.html

Note: See the readme file and release notes first for more information, such as limitations and prerequisites, UEFI requirements, and how to use a script.

6.Will you be able to use Intel® Ethernet SFP+ SR Optics and Intel® Ethernet SFP+ LR Optics as these are the only 10-Gbps optical modules supported? 

If the issue still persists after trying out the above steps, please provide another SSU logs. Additionally, if you can provide the test result or logs form your packet analyzer it would be highly appreciated. 

Looking forward to your reply.

Should there be no response, I’ll make sure to reach out after 3 business days. 

Best regards,

Crisselle C.

Intel® Customer Support 

[Miladdona]

Hi,

Thank you for your time.

I checked the Flow Control again and found it disabled! It was for Data Center Bridging (DCB). I disabled the DCB and set the Flow Control on "Rx & Tx Enabled" for all my ports.
There is no data loss when I test only 1 of the 4 ports (regardless which port is it).
I tested different cables.
I connected the Intel Ethernet adapter to different PCIe.
I downloaded the "NVMUpdatePackage_700_Series_v8_10"  from your link and got some issues that you can see in Fig.1.
I use "cisco SFP-10G-SR" module, and unfortunately I do not have access to Intel® Ethernet SFP+ SR Optics or Intel® Ethernet SFP+ LR Optics!

After this steps, I still have loss.
Because I need capture data with high rate, I can't use "Wireshark" (due to data loss) and for this reason I have developed a C program that use pcap. If you think it can help you I can share or send it to you?

Thank you for your consideration.
Best regards,
Miladdona

[Miladdona]

Hi,

In the previous post I forgot to send you SSU log.

I sent you new SSU log. Please find attached.

Best regards,

Miladdona

[AlfredoS_Intel]

Hi Lecrafter129,

Thank you for providing that information.

We will continue checking this and the information that provided is invaluable to us.

Please allow us some time to check on this. We hope for your understanding regarding this

We will get back to you no later than 3 business days from now.

Best Regards,

Alfred S

Intel® Customer Support 

[Caguicla_Intel]

Hello Miladdona,

Apologies for the delay.

Please be informed that we are still checking your request. Rest assured that we will provide an update as soon as possible but no later than 2-3 business days.

Hoping for you kind patience.  

Best regards,

Crisselle C.

Intel® Customer Support 

[Caguicla_Intel]

Hello Miladdona,

Thank you for the patience on this matter.

Can you confirm if this is this the only utility that you used in measuring the packet loss? Does it have the same result if you try a different utility? Will you be able to provide both logs for further validation? If in case that you are using only one utility (the C program you developed), please do share the logs with us. 

Looking forward to your reply. 

Should there be no response, I’ll make sure to reach out after 3 business days. 

Best regards,

Crisselle C.

Intel® Customer Support 

[Caguicla_Intel]

Hello Miladdona,

Good day!

We'd like to follow up the information requested for us to continue with the investigation. Let us know if you have additional questions and clarifications.

Awaiting to hear from you. 

We will follow up after 3 business days in case we don't receive a reply.

Best regards,

Crisselle C.

Intel® Customer Support 

[Miladdona]

Hi,

Thank you for your response.

I only use my C program to test losses. I sent you my C program. Please find it attached.

It is a multi-threaded program to capture data and write them to files. (Now, write data thread is inactive and I only test capturing data). you can set "DataPacketLen"  the length of your packet,  "LANCount"  the number of port that you want to get data simultaneously, and "LAN_Board" the LANs ID (in order of numbers) in "GlobalFunctions.h"  file.

In our packets we have a global counter called GPN (Global Packet Number) that we check this field for loss. It is usually "unsigned int" in our packets and we need to explicitly specify the start byte number for this field. You can set the "startByteInGPN" in "Communicator.h" file.

Please let me know if need any additional information.

Best regards,

Miladdona

[Caguicla_Intel]

Hello Miladdona,

Thank you for providing the requested information.  

Please allow us to forward the logs to our engineering team. We will get back to you as soon as we heard from them but no later than 2-3 business days.

Hoping for your patience.

Best regards,

Crisselle C.

Intel® Customer Support