Intel Ethernet I219-V connection dropped

[BalazsPal]

Hi,

We have a software suite we developed in-house and are selling for 20 years now with all kinds of different network adapters (Intel series, other brands). This time, when we were testing it on a new system with an AsRock motherboard, we realized after starting the software, the local communication stops working on Intel I219-V. It does not matter what this PC is connected to (another I219 PC, another NIC brand, etc.), it simply stops working. Latest driver from Intel or from the manufacturer (AsRock) didn't help along with all chipset drivers and BIOS updates / settings we tried. Nothing helps recovering the freeze after it happens (disconnect/reconnect, disable/enable, reset commands), only restarting the PC. If we try to change settings for the adapter, all Windows settings windows freeze, and if we restart the PC, after 300 seconds of timeout, we get a blue screen (BSOD) with DRIVER_POWER_STATE_FAILURE pointing out the kernel was unable to change the power state of this network adapter (presumably putting it to sleep or shut down state), because 1 thread exclusively locks it and 3 threads are waiting for the device - a deadlock maybe. If the adapter froze and we immediately restarted the PC, we could at least avoid the BSOD.

Debugging the issue gave a lot of information: sending a lot of data along several TCP sockets being open somehow fill up the transmit buffer or overloads it, and then after the TCP window is full (based on Wireshark) it no longer sends any data, although receive still works. Could not reproduce with iperf3 and days of huge amounts of data on maximum speed though. We increased the Transmit Buffers to maximum (2048), this was the only setting affecting the issue. It made the problem way-way less frequent (from instant freeze to a few hours of operation), but didn't solve the issue for good. The version of the driver was 19.12.2.45 when installing from Intel (28.0 release), or from the manufacturer site (AsRock Intel driver, same version: 19.12.2.45).

We tried a lot of settings from energy saver options to all offloading settings changes, literally all possible options in the advanced driver settings. On Windows side from TCP settings (window size and scaling, congestion control algorithms, timestamping...) to different socket setups in our applications, nothing helped. We tried some older drivers, they also did not work. We updated the BIOS, changed and tested several settings there, to no avail.

The solution:

Later, we found network drivers (only) with a newer version on some website for Asus motherboards (not ours: AsRock), with a version 19.12.2.50 - This was also found on some HP or Fujitsu websites. The driver date is a few monts old: 3/11/2022. We installed this version, it was actually accepted by Windows, and the problem instantly went away, at least it's being tested for 2 days in a row without issues (and default Transmit Buffers of 512!!).

My question is: can someone (from Intel?) please share the improvements and fixes between version 19.12.2.45 and 19.12.2.50? I really need this information to evaluate the reliability of this hardware as these PCs should run 24/7 in larger numbers without any network issues. The release notes for the major Intel versions did not help and did not contain any information regarding this NIC or version, I've read them from the last 2 years. The only info I found were these:

• https://community.intel.com/t5/Ethernet-Products/RSYNC-network-breakdown-issues-with-Intel-Ethernet-Connection-14/td-p/1444040 (the driver upgrade helped a customer with HP PC tower hardware and HP mentions some changes in their release here: https://support.hp.com/my-en/drivers/selfservice/swdetails/hp-z2-tower-g4-workstation/20063240/swItemId/vc-303253-1 )
• https://community.intel.com/t5/Ethernet-Products/Intel-i225-Network-Adapter-Drops-Connection-When-NAS-Backs-Up-PC/td-p/640676 (here they have issue with Synology NAS backup with encryption on, where increasing buffers helped)

Also if I recall correctly, we experienced the same issue with another onboard NIC (Intel I226-V), although this needs further testing as this does not happen instantly. Is this potentially fixed issue between the versions 45 and 50 maybe also existing in the Intel I226-V driver (1.1.3.34)?

I will contact AsRock of course to provide a customized driver update, although the ones found for Asus or HP contained the same exact 4 files without any modifications.

Let me know if you need further information.

[B_Y]

Hi BalazsPal,

Thank you for posting in Intel Ethernet Communities.

We appreciate your extra effort to assist the Intel community, at this moment we still looking into the network issue involving the Intel Ethernet I225, I226 and I219 chips.

While OEM or vendor companies like Asus offer defined and tailored drivers in accordance with their motherboard specifications, Intel only offers generic drivers.

If you have any questions concerning the features that have been added between versions 12.19.2.45 and 12.19.2.50, you may refer to the release notes beginning with release notes 27.2:

https://www.intel.com/content/www/us/en/download/15084/intel-ethernet-adapter-complete-driver-pack.html

Hope this clarifies.

Best regards,

BY_Intel

Intel Customer Support

[BalazsPal]

Hi B_Y,

What are the network issues you are looking into?

The problem is, I read through all the ReleaseNotes pdf files from 27.0 all the way to 28.0 (latest), but I did not find any information regarding the 12.19.2.45 and 12.19.2.50 changes for Intel 200 Series. It always says "None for this release" - that is why I am asking about specific changes between these two driver versions - what got fixed?

Please provide some info with the fixes and the possible fixes for I216 as well (if any). Thank you very much, have a nice day!

Best regards,

Balazs

[B_Y]

Hi BalazsPal,

I appreciate your response.

We are investigating network problems related to coverage, speed limits, connection loss, and many other things. Although we don't currently have an ETA for the updated driver release, the Intel development team is actively trying to get it out as soon as possible.

For a summary of the release notes, please see:

https://www.intel.com/content/www/us/en/support/articles/000057110/ethernet-products.html

Would you mind letting me know if you need any additional help from our end or if this discussion may be closed at this time?

Best regards,

BY_Intel

Intel Customer Support

[BalazsPal]

Hi B_Y,

Yes, I would like to find the changes between the mentioned driver versions (12.19.2.45 -> 12.19.2.50). I know the first 12.19.2.50 drivers appeared in November, but reading through the Release Notes from the last 2 years (!) I could not find any information.

Can you please tell me where to find all the changes between 12.19.2.45 -> 12.19.2.50 (or just list it here)? I would help us a lot making decisions in the near future.

Thank you very much, have a nice day!

Best regards,

Balazs Pal

[B_Y]

Hi BalazsPal,

Thank you for your patience.

The 12.19.2.50 version of the e1d drivers (I219) have only been released to OEM's, which system manufactures will publish their own customized drive, such as the HP driver you have shared here.

The information you found from the HP release notes is the extent of the changes from 12.19.2.45 to 12.19.2.50.

• Fixes issue where there would be no or intermittent Network Connectivity
• Fixes issue where proxy offload wouldn't work in S3 config when connected to LP that supports 1G + EEE
• Fixes issue where system would sporadically hang with CATT error LED on during coldboot and warmboot cycling.

Meanwhile, Intel generic Ethernet driver starting from package version 27.2 - supporting driver version 12.19.2.45, described all the features. Products such as I225 and I226, there is an active investigation and you may follow this thread for any updates:

https://community.intel.com/t5/Ethernet-Products/Intel-Communication-Intel-Ethernet-Controller-I226-Series-Random/td-p/1453177

Hope this clarifies. Please let me know if there is anything else we can do to assist you.

Best regards,

BY_Intel

Intel Customer Support

[BalazsPal]

Hi B_Y,

Okay, so there is no more detailed information about the fixes or a more complete list. Then I guess have all the available information I was looking for. I also haven't seen the thread you shared. Also, for that Intel I226-V I've found a 1.1.4.38 driver version on an ASUS forum website, some issues might have been already resolved in that driver version (the helpful forum: https://rog-forum.asus.com/t5/intel-700-600-series/drivers-intel-ethernet-wifi-bluetooth/td-p/878985/page/52 ). Thank you very much for your help!

Best regards,

Balazs Pal

[B_Y]

Hi BalazsPal,

Thank you for your reply and appreciate your sharing on the Asus rog.

As mentioned in previous comment, this page (https://www.intel.com/content/www/us/en/support/articles/000057110/ethernet-products.html) provides the current release notes for Intel® Ethernet Products. The notes are categorized by release version, from newest to oldest. 

Click the version to expand it into a summary of new features and changes in that version since the last release. Access the download links for the detailed release notes.

If Intel® Ethernet Controller is embedded on Original Equipment Manufacturer (OEM) system, it's best to check with the manufacturer as the software drivers linked from Intel page are generic versions, and can be used for general purposes. To avoid any potential installation incompatibilities on OEM system, Intel recommends checking with the OEM and use the software provided via system manufacturer.

Rest assured that we are working thoroughly on the current issue and a new release of driver. Should you need any further information, please do not hesitate to replay this thread.

Best regards,

BY_Intel

Intel Customer Support

[B_Y]

Hello BalazsPal,

I hope this message finds you well. I am just sending a follow up if you have any questions from the previous post. If so, please let us know.

In case we do not hear from you, I will make a follow up on May 10th 2023 . Thank you.

Best regards,

BY_Intel

Intel Customer Support

[AdamRayburnVitol]

Hello to the OP and Intel:

Thank you for posting.   We have 120 units of HP Elite 800 SFF G9 with the i219-LM network adapter that we deployed starting in August 2022.   

The first issue we addressed was the network card losing connection intermittently, or completely until you restart the computer.   In January, we rolled the units back to 12.19.0.16.

About 40 of those units have had BSOD in the last 3 months.   IThe most common BSOD is DRIVER_POWER_STATE_FAILURE referencing HDAudbus.sys, but there are a total of 8 Different BSOD.

We are unable to reproduce the BSOD on demand, even with adjusting the network settings.  It can happen weekly, bi-weekly, or monthly - there is no pattern.  

DPC_WATCHDOG_VIOLATION

DRIVER_POWER_STATE_FAILURE

CLOCK_WATCHDOG_TIMEOUT

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED

PAGE_FAULT_IN_NONPAGED_AREA

PFN_LIST_CORRUPT

SYSTEM_SERVICE_EXCEPTION

MEMORY_MANAGEMENT

We just discovered that all of our units reverted to 12.19.2.45 when someone reported lost network this week.   It may have been due to installing Win10 21H2 since.

The problems you describe here regarding the BSOD and lost Network sound very similar to what we have seen.  Our business has a number of in-house apps also built over many years.   I have yet to put WireShark on it to compare.

We are considering rolling out the new driver version 12.19.2.50.  HP website says it is Rev.P released 12/20/22, but unpacking it shows 11/3/22.

Can the OP and Intel perhaps reply to this thread to provide any suggestions or updates? 

[BalazsPal]

Hi AdamRayburnVitol,

We discovered that increasing transmit buffers can reduce the amount of freezings, but it still happened on maximum value (2048). YOu might try decreasing it to a quite low value to maybe increase the chances to hang the adapter.

Examining with wireshark I saw one single "TCP window full" message before it hung, even with adaptive TCP windows (autotuninglevel etc.) turned on or off. Also only the send did not work, the adapter still received the packets, it was just unable to send anything.

The BSODs happen I think because the whole driver hangs and ultimately something else will be blocked by it (e.g. on shutdown the DRIVER_POWER_STATE_FAILURE kicks in, as it can not put the device to lower power levels I think).

And if you can reproduce, it will be much easier to test the .50 driver version - and my guess is that it'll fix your issue.

B_Y: if there is no release notes at all about these versions, just the HP one, yes, feel free to close it. If there is a possibility to get the detailed release notes, I'd still be happy to see it.

Best regards,

Balazs

[B_Y]

Hello @BalazsPal ,

Please be informed that we will now close this request since we haven't received any response from our previous follow ups. Just feel free to post a new question if you may have any other inquiry in the future as this thread will no longer be monitored.

Hello @AdamRayburnVitol ,

Thank you for your feedback. If the system's I219-V/NIC is as PCIE.

You can do clean driver installation here

https://www.intel.com/content/www/us/en/support/articles/000058304/ethernet-products.html

Install new generic driver from here: 

https://www.intel.com/content/www/us/en/download/15084/intel-ethernet-adapter-complete-driver-pack.html

We would like to share some tips on How to Manually Install Intel® Network Adapter Drivers in Windows:

https://www.intel.com/content/www/us/en/support/articles/000022174/ethernet-products.html

But if it is onboard NIC, the most reliable driver will be from manufacturer OEM because of its dependency on manufacturer motherboard while using Intel ethernet chipset.

In your scenario involving I219-LM, Windows Event log/ Viewer or Wireless network report would mentioned the changes of driver. Unfortunately, involuntary driver update may occurred due to GPO, manufacturer or windows - whichever has highest priority based on your company policy or can be disabled from going automatically.

We believe manufacturers are working closely on latest driver in regards disconnection issue.

May I suggest you to consider firstly checking this thread below if it helps in your situation, then contact HP to seek for correct drivers

https://community.intel.com/t5/forums/forumtopicpage/board-id/ethernet-products/message-id/32786#M32786

Thank you.

Best regards,

BY_Intel

Intel Customer Support