Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4811 Discussions

Capture X722 dropped packet

jimmy_tusim
Beginner
3,066 Views

Hi,

     I am using X722 Ethernet Adapter on my server. System monitor reports that there are packets dropped on the network interface(reports around every 30 seconds), as I dig into the driver code (i40e), I find the error is rx_dropped. But my server still runs well. So I am wondering how to capture the dropped packets on the ethernet interface which cause rx_dropped to increase so that I can better find the possible bugs.  Thank you.

Best

0 Kudos
1 Solution
Mike_Intel
Moderator
2,937 Views

Hello jimmy_tusim,


Thank you for patiently waiting for our update.


Upon checking with our engineering team, there is no way to monitor ring usage while the system is running.

And also, upon looking at details that you provided, this is not a retail X722-DA2.

The list shows X722 for 10GBASE-T, the only X722 cards are SFP+ not Twisted pair.

And the EtrackID shown does not match up with any cards that we have.


The NVM shows 3.25, there should be a newer version, depends on if it's a LOM and who the OEM is.


So for us to know where to direct you for further support. Kindly provide the following details.


  1. What is the model of your board?
  2. I know I already asked this before and you said that the card is already installed in the server but having the markings of the card should tell us what is the manufacturer of the card.


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.



Best regards,

Michael L.

Intel Customer Support Technicians


View solution in original post

0 Kudos
11 Replies
Mike_Intel
Moderator
3,053 Views

Hello ,


Thank you for posting in Intel Ethernet Communities. 


For us to understand the issue better, I need to gather the following details:


  1. Are you using onboard network card or a PCIe card?
  2. What is the brand and model of your server?
  3. Please provide a screenshot of the pocket loss test.


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel Customer Support Technicians


0 Kudos
Mike_Intel
Moderator
3,029 Views

Hello jimmy_tusim,


I just want to make a follow up on the information that I requested below.


  1. Are you using onboard network card or a PCIe card?
  2. What is the brand and model of your server?
  3. Please provide a screenshot of the pocket loss test.


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel Customer Support Technicians


0 Kudos
jimmy_tusim
Beginner
3,025 Views

Hi Michael,  

   Thanks for your reply!

  •    I am using a PCIe card. X772. Here is some output after I run ethtool --driver LAN and lspc 

driver: i40e

version: 1.4.25-k
firmware-version: 3.25 0x800009d3 1.1638.0
expansion-rom-version:
bus-info: 0000:1a:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

And some hardware configuration.

 - "Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 09)"
                    Supports auto-negotiation:"Yes"
                    Advertised auto-negotiation:"Yes"
                    Auto-negotiation:"on"
                    Availability:"UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1"
                    Capabilities:"[40] Power Management version 3"
                    Capabilities:"[50] MSI: Enable- Count=1/1 Maskable+ 64bit+"
                    Capabilities:"[70] MSI-X: Enable+ Count=129 Masked-"
                    Capabilities:"[a0] Express Endpoint, MSI 00"
                    Capabilities:"[e0] Vital Product Data"
                    Capabilities:"[100] Advanced Error Reporting"
                    Capabilities:"[140] Device Serial Number 0c-31-53-ff-ff-ad-00-c4"
                    Capabilities:"[150] Alternative Routing-ID Interpretation (ARI)"
                    Capabilities:"[160] Single Root I/O Virtualization (SR-IOV)"
                    Capabilities:"[1a0] Transaction Processing Hints"
                    Capabilities:"[1b0] Access Control Services"
                    Caption:"Not Available"
                    Default IP Gateway:"Not Available"
                    DHCP Enabled:"Yes"
                    DHCP Lease Expires:"Not Available"
                    DHCP Lease Obtained:"Not Available"
                    DHCP Server:"Not Available"
                    Driver:"i40e"
                    Driver Path:"/lib/modules/4.19.57-041957-generic/kernel/drivers/net/ethernet/intel/i40e/i40e.ko"
                    Driver Provider:"Intel Corporation, <e1000-devel@lists.sourceforge.net>"
                    Driver Version:""
                    Duplex:"Full"
                    Firmware Version:"3.25 0x800009d3 1.1638.0"
                    Interface:"LAN"
                    I/O Ports:"Not Available"
                    IP Address:"192.168.100.3"
                    IP Subnet:"255.255.255.0"
                    MAC Address:"c4:00:ad:53:31:0d"
                    Location:"1a:00.1"
                    Manufacturer:"Intel Corporation"
                    Net Connection ID:"Ethernet"
                    Port:"Twisted Pair"
                    Power Management:"Supports Wake-on: g"
                    Power Management:"Wake-on: g"
                    Speed:"10000Mb/s"
                    Supported link modes:"1000baseT/Full, 10000baseT/Full, Supported pause frame use: Symmetric"
                    Advertised link modes:"1000baseT/Full, 10000baseT/Full, Advertised pause frame use: No"
                    Partner advertised link modes:"Not Available"
               - "Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 09)"
                    Supports auto-negotiation:"Yes"
                    Advertised auto-negotiation:"Yes"
                    Auto-negotiation:"on"
                    Availability:"UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1"
                    Capabilities:"[40] Power Management version 3"
                    Capabilities:"[50] MSI: Enable- Count=1/1 Maskable+ 64bit+"
                    Capabilities:"[70] MSI-X: Enable+ Count=129 Masked-"
                    Capabilities:"[a0] Express Endpoint, MSI 00"
                    Capabilities:"[e0] Vital Product Data"
                    Capabilities:"[100] Advanced Error Reporting"
                    Capabilities:"[140] Device Serial Number 0c-31-53-ff-ff-ad-00-c4"
                    Capabilities:"[150] Alternative Routing-ID Interpretation (ARI)"
                    Capabilities:"[160] Single Root I/O Virtualization (SR-IOV)"
                    Capabilities:"[1a0] Transaction Processing Hints"
                    Capabilities:"[1b0] Access Control Services"
                    Caption:"Not Available"
                    Default IP Gateway:"10.130.16.253"
                    DHCP Enabled:"Yes"
                    DHCP Lease Expires:"08:07:11"
                    DHCP Lease Obtained:"Aug 11 16:07"
                    DHCP Server:"10.130.16.253"
                    Driver:"i40e"
                    Driver Path:"/lib/modules/4.19.57-041957-generic/kernel/drivers/net/ethernet/intel/i40e/i40e.ko"
                    Driver Provider:"Intel Corporation, <e1000-devel@lists.sourceforge.net>"
                    Driver Version:""
                    Duplex:"Full"
                    Firmware Version:"3.25 0x800009d3 1.1638.0"
                    Interface:"WAN"
                    I/O Ports:"Not Available"
                    IP Address:"10.130.21.4"
                    IP Subnet:"255.255.248.0"
                    MAC Address:"c4:00:ad:53:31:0c"
                    Location:"1a:00.0"
                    Manufacturer:"Intel Corporation"
                    Net Connection ID:"Ethernet"
                    Port:"Twisted Pair"
                    Power Management:"Supports Wake-on: g"
                    Power Management:"Wake-on: g"
                    Speed:"1000Mb/s"If you need 
                    Supported link modes:"1000baseT/Full, 10000baseT/Full, Supported pause frame use: Symmetric"
                    Advertised link modes:"1000baseT/Full, 10000baseT/Full, Advertised pause frame use: No"
                    Partner advertised link modes:"Not Available"
  •         The server is customized server. And the cpu, RAM were not fully loaded (~70%) when drop happened.
  • the attach file is a simple log recording the udp socket buffer usage, some result given by 'ethtool -S LAN', and 'ethtool -c LAN'(we turn on the adaptive rx). and the number of intterupts each network queue generate every 0.5s. The log is quite self-explainable.  And the expected throughput is around 400Mb/s.

The drop happened on several servers, after we changed the ring buffer size from 512 to 2048, most of the server worked well but only one server still has packets dropped. These servers are of the same configuration. I am very confused. If you need more information, just let me know. Thank you!

Best,

Jimmy

 

0 Kudos
jimmy_tusim
Beginner
3,024 Views

The previous reply may not be much relevant to the post title, but I am working on the packet lossing problem on the ethernet interface. They all help.

0 Kudos
Mike_Intel
Moderator
3,017 Views

Hello jimmy_tusim,


Before we proceed, let me clarify the part when you said that the server is a customized server and the other components are not running on normal configuration. Are you designing a system?


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel Customer Support Technicians


0 Kudos
jimmy_tusim
Beginner
3,013 Views

Hi Michael,

     I am not designing the system, other components are running on normal configuration. What I mean is that when the drops happened, other components (cpu,RAM) are running well. Attached is the full hardware component list. Thank you.

Best,

jimmy

0 Kudos
Mike_Intel
Moderator
3,008 Views

Hello jimmy_tusim,


Thank you for the clarification. Let me gather more details about the network card for me to further assist you.


  1. You mentioned that the you are using a PCIe X722, please provide the PBA number of the card of a picture of the card focusing on the markings.
  2. Where did you download the driver that you used?
  3. You also mentioned that you have other server with similar configuration and only one is having pocket drops. Have you tried updating the firmware of the X722 card?
  4. Can you also Run the command ethtool –i


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel Customer Support Technicians


0 Kudos
jimmy_tusim
Beginner
3,004 Views

Hi Michael,

    1. The card is already installed on a remote server to which I can not  have access right now. I am afraid I can not provide such information.

     2. I am using the system default driver(i40e), here is the link to kernel                                                                            https://github.com/alibaba/cloud-kernel/releases?after=v4.19.70

    3. No, but all the servers are of the same configuration and running the same code.  Increasing the ring buffer from 512 to 2048 solved the dropping problem for them but not for this one. That's why I get confused. ( Increasing the buffer size to 4096, reduces the drop but also introduces unacceptable delay to the our code).

     4.  ethool -i LAN gives :

driver: i40e
version: 2.3.2-k
firmware-version: 3.25 0x800009d3 1.1638.0
expansion-rom-version:
bus-info: 0000:1a:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

        To sum up, what I want to know is: 1)  how to monitor the ring buffer usage when the system is running. As binary search for the optimal ring buffer size costs too much. 2) how to capture the dropped packets( like, directly dumping them to the disk?).   Thank you!

Best,

Jimmy

TuSimple, Inc.

0 Kudos
Mike_Intel
Moderator
2,961 Views

Hello jimmy_tusim,


Thank you for providing the information that I requested. Let me further check your 2 questions for us to provide an answer.

Please give us 2 to 3 working days to provide an update.


Best regards,

Michael L.

Intel Customer Support Technicians


0 Kudos
jimmy_tusim
Beginner
2,956 Views
0 Kudos
Mike_Intel
Moderator
2,938 Views

Hello jimmy_tusim,


Thank you for patiently waiting for our update.


Upon checking with our engineering team, there is no way to monitor ring usage while the system is running.

And also, upon looking at details that you provided, this is not a retail X722-DA2.

The list shows X722 for 10GBASE-T, the only X722 cards are SFP+ not Twisted pair.

And the EtrackID shown does not match up with any cards that we have.


The NVM shows 3.25, there should be a newer version, depends on if it's a LOM and who the OEM is.


So for us to know where to direct you for further support. Kindly provide the following details.


  1. What is the model of your board?
  2. I know I already asked this before and you said that the card is already installed in the server but having the markings of the card should tell us what is the manufacturer of the card.


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.



Best regards,

Michael L.

Intel Customer Support Technicians


0 Kudos
Reply