dropped multicast traffic after few days on IT210-AT

[KnowJoe]

Hello,

I have an issue with the Intel® Ethernet Controller I210-AT. After some amount of time it stops receiving multicast traffic. We observed this issue many times in different scenarios but most of the time it was correlated with quite heavy multicast load.

We have an application listening on two multicast groups, let's name them A and B. On A, the application receives many controlling packets meaning a large number of small data chunks. Based on A there are also many dynamically created groups, let’s say C. On C groups we receive the actual data (larger payload). After some amount of time it stops receiving traffic from A, which also stops creating C groups. Command netsh int ip show joins shows correct multicast groups with references. In wireshark we can’t see packets addressed to A being received by the operating system. We checked using a hardware sniffer that packets reach the Network Adapter. Using commands: Get-NetAdapterStatistics -IncludeHidden | Format-List  -Property *, Get-Counter -Counter '\Network Adapter(*)\*' we can deduce that load drops drastically from ~100000 packets/sec to ~100 packets/sec (packets from B multicast group). Everything remains in this state until reboot or running wireshark in promiscuous mode (we already checked that setting promiscuous mode on NIC manually also works).

In our testing environment we performed a lot of tests and here we have some conclusions:

With a load of ~1600 multicast packets received/sec, ~170000 bytes received/sec, we have 100% reproduction rate of this issue after 2 hours - 2 days (usually under 10 hours).

With the same load, we changed the Network Card to Broadcom NetXtreme Gigabit Ethernet (BCM5761) and the issue didn’t reproduce for over 2 weeks (after this time we had to reinstall the machine).

Our tests covered different Windows 10 versions, drivers including the newest one and two computer models based of different motherboards:

Windows 10: 1607, 1809

Drivers: e1r65x64: 12.15.184.0 (10/10/2017), 12.16.6.0, 12.16.7.0 (06/28/2023), e1r68x64: 12.18.11.1 (7/16/2020)

Motherboards: Supermicro X11SSL-F, X11SCM-F

We also tried with disabled power saving options, and disabled speed/duplex auto-negotiation.

SSU file added in attachment.

Do you have any idea what may be the root cause of this issue, or what else should we check?

[Yogaeasvaran]

Greeting John

Thank you for contacting Intel Customer Support.

We have review the SSU log and it seems there is no issue with the settings.

We recommend you update the firmware and Ethernet drivers.

If issue does not resolve, then we suspected it could be the hardware issue.

If upon updating the BIOS and drivers does not exhibit this issue, we'd suggest to double check with Supermicro support. 

There might be changes done on the functionality of the I210-AT Ethernet Controller with BIOS update version 2.2.

Regards,

Yogaeasvaran

Intel Customer Support

[KnowJoe]

Thanks for the answer Yogaeasvaran! What kind of firmware and drivers do you recommend? We tried a lot of different drivers without success. But what is worth mentioning - the problem didn't occure on Intel I219-LM.

Do you have any other ideas?

Regards!

[Yogaeasvaran]

Greeting John

Thank you for the feedback.

May I know that you install and update drivers from intel website or Supermicro website support?

If upon updating the BIOS and drivers does not exhibit this issue, we'd suggest to double check with Supermicro support. 

Regards,

Yogaeasvaran

Intel Customer Support

[Yogaeasvaran]

Greeting John

Regarding your case number 06003468, we are following up to find out if you were able to complete the actions, we previously recommended.

Please reply to this thread to confirm, so we can continue helping with a resolution. Looking forward to receiving your reply!

Regards,

Yogaeasvaran

Intel Customer Support

[KnowJoe]

Hi, 

It didn't help - the issue still occurs on the newest BIOS version and drivers. 

Regards,

J

[Yogaeasvaran]

Greeting John

Thank you for the feedback.

Since this driver's update does not fix this issue, we recommend you contact OEM/Supermicro support to isolate this issue.

Regards,

Yogaeasvaran

Intel Customer Support

[KnowJoe]

Do you have any other ideas what else can be checked? Or how to debug such problems?

[Yogaeasvaran]

Greeting John

Thank you for the feedback.

We have understood the issue but unfortunately, we don't have any other option to isolate this issue.

You may contact supermicro support as there would be specific setting or configuration need to be done.

Therefore, we request you to contact supermicro for this issue.

Regards,

Yogaeasvaran

Intel Customer Support

[Yogaeasvaran]

Greeting John

Thank you for the feedback.

Regarding your case number 06003468, we are following up to find out if you were able to complete the actions, we previously recommended.

Please reply to this thread to confirm, so we can continue helping with a resolution. Looking forward to receiving your reply!

Regards,

Yogaeasvaran

Intel Customer Support

[KnowJoe]

Hello Yogaeasvaran,

I still need a resolution, the issue is recurs all the time. It is Intel network card, so I believe you can provide any driver or dedicated firmware to address such problem.

Regards,

John

[Yogaeasvaran]

Greeting John

Thank you for the feedback.

Here is the link below to download the drivers from Intel website:

Intel® Ethernet Controller I210-AT

Although Intel Ethernet Controller I210-AT you have update the drivers and firmware didn't fix the issue, you may try OEM drivers but when the hardware is integrated with certain Motherboard would have specific drivers to merge with.

Therefore, we suggest you contact OEM/Supermicro support for the drivers and the configuration guide to isolate this issue.

Regards,

Yogaeasvaran

Intel Customer Support

[KnowJoe]

Hello,

Updating driver to different version was my first idea, and I did it several times - didn't help (used newest, oldest, OEM, etc.). Still do not know what is the root-cause of this issue. 

Regards,

J

[Yogaeasvaran]

Greeting John

Thank you for the feedback.

Here is the link below to download the drivers from Intel website:

Intel® Ethernet Controller I210-AT

You may try to do clean installation first for this process.

Regards,

Yogaeasvaran

Intel Customer Support

[KnowJoe]

Hello,

As I wrote earlier - we used many different drivers from Intel and other websites and it did not help.

Regards,

J

[Yogaeasvaran]

Greeting John

Thank you for the feedback.

As we mentioned on previous email, if our Intel drivers is not working then you might contact Supermicro support as the board and drivers have certain configurations need to set up.

This would need their drivers' files to install and check the correct version that need to install.

Please do contact to supermicro support to rectify the issue.

Regards,

Yogaeasvaran

Intel Customer Support

[Yogaeasvaran]

Greeting John

Thank you for the feedback.

We are following up to find out if you were able to complete the actions, we previously recommended.

Please reply to this thread to confirm, so we can continue helping with a resolution. Looking forward to receiving your reply!

Regards,

Yogaeasvaran

Intel Customer Support

[KnowJoe]

Hello,

I tried to use different drivers with no luck. The issue still occurs. 

Regards,

J

[Yogaeasvaran]

Greeting John

Thank you for the feedback.

Did you manage to contact Supermicro support for this issue?

Please do contact to supermicro support to rectify the issue.

Regards,

Yogaeasvaran

Intel Customer Support

[Yogaeasvaran]

Greeting John

We are following up to find out if you were able to complete the actions, we previously recommended.

We will be closing this ticket on our next follow up.

Regards,

Yogaeasvaran

Intel Customer Support

[Yogaeasvaran]

Greetings John,

Kindly note that as we have not received any response from our previous follow-ups, we will be closing this request.

If you have any more questions in the future, please don't hesitate to post a new question, as this thread will no longer be monitored.

Thank You & Regards

Yogaeasvaran

Intel® Customer Support