82574L NIC on DZ77GA-70K failing repeatedly

[idata]

New Windows 8 install on DZ77GA-70K board. About every 24-48 hours the 82574L integrated NIC fails hard--all communication through it stops and Windows will not shut down--the NIC seems to be hanging it. I see the same thing when looking at the NIC properties when it is hung and trying to apply any changes (e.g. IP address change)--the properties dialog hangs.

Have updated to latest drivers and latest BIOS. Problems continue. Am about to RMA the board.

[idata]

It turns out using this driver is not a long term solution. I had yet another lockup while doing a large file transfer. Probably the only reason it didn't lock up sooner is that I wasn't using that NIC for large file transfers.

I ended up buying another NIC and putting it in there and giving up on the built-in NIC. Intel is obviously never going to fix this. Just consider this motherboard to come with only one working NIC. You will save time and money by doing this.

[idata]

I called Intel Support concerning this issue because I am affected too. The issue has been escalated to the developers. I also created a support case with Microsoft concerning the fact that Windows 8 cannot gracefully handle the failure to the point that the entire networking subsystem is crippled and the system cannot shut down.

When I disabled the device in Device Manager and then re-enabled it, Windows 8 would tell me that no driver is installed; I have shown the screenshots to Microsoft. I found a way of recovering without restarting Windows 8. I provided these instructions on my Intel support ticket so the developers can fx the issue.

Steps to reinitialize without the Windows 8 driver initializer becoming corrupted:

1) Go to Device Manager

2) Expand Network adapters

3) Open the 82574L properties

4) Go to the Link Speed tab

5) Click on Diagnostics

6) Go to the Hardware tab

7) Click Run Test (NOTE: all tests will fail)

8) Click Close

9) Go to the Driver tab

10) Click Disable

11) Click Enable

Network connectivity will return. If you run the hardware diagnostics again, you will see that they all pass now. I believe Intel should make the hardware test initialization part of the driver initialization. Something it does during the tests cleans up whatever corruption is present.

Let me know whether this does or does not work for you.

[idata]

David,

Do you have any news ? I cannot let the adapter enabled because it is part of a LACP Team and the connection/disconnection make the Team instable.

Now I am encouraging all people around me to ask for a refund because of a hidden malfunctionning !

[idata]

I am still working with Intel.

The glitch that I am experiencing occurs 1-3 times per 2 weeks for me.

The symptoms of this glitch are:

• All network connectivity stops and the network adapter displays the message No Network Connectivity
• The Windows networking stack becomes locked, preventing me from switching to other network adapters
• Disabling the driver does not work, prevents the driver from being re-enabled, and corrupts the NT kernel to the point of not being able to shut down

The workaround described in my previous post works for me.

If anyone would like an update on my ticket with Intel regarding, please send me a private message.

[ZKaku]

Hello,

I have the same problem as described above on a brand new DZ77GA-70K mainboard, latest BIOS (060) and network drivers (17.4) from the support page.

My OS is Windows 7 Ultimate x64.

The ethernet connection randomly fails, and there is no way to recover it, unless you reset the board or restart windows 7 (which is not working every time).

Can you post a ticket about motherboard issues on Intel's site? If you can, where?

Regards,

Kakci.

[idata]

Hello Kakci,

I have the same problem as described above on a brand new DZ77GA-70K mainboard, latest BIOS (060) and network drivers (17.4) from the support page.

My OS is Windows 7 Ultimate x64.

The ethernet connection randomly fails, and there is no way to recover it, unless you reset the board or restart windows 7 (which is not working every time).

For my issue, I was able to recover by using the hardware diagnostics that are found in the device driver's properties. The full instructions are in post 24 (/message/173947# 173947 http://communities.intel.com/message/173947# 173947) above. Do you see a "No Network Connectivity" or "Network Cable Unplugged" error for the network adapter? How frequently does the error occur?

Can you post a ticket about motherboard issues on Intel's site? If you can, where?

I always call them, but you can go to http://www.intel.com/p/en_US/support/contactsupport Contact Support for the full list of communications methods.

Regards,

David

[ZKaku]

Ok, thank you for your reply. At the moment I am testing the 2nd NIC on the motherboard to see, if that's affected by the same problem or not. If not I'm gonna use that, but would be great if Intel would fix this.

[idata]

This is nothing compared to the black level issue with their graphics cards:

/thread/29420 http://communities.intel.com/thread/29420

[idata]

Hello I am still looking for some ideas. It seems like my problem differs from those described in this post. I am running the DZ77GA70K with windows 7 pro. I have found that rather than complete failure I am getting loss of internet connectivity during or after very large file downloads (e.g. windows 8 upgrade files, etc).

The local network is fine. The problem occurs with both NICs and the connection can be restored by disabling and re-enabling the adapter in device manager (though the systray icon keeps its little yellow warning exclamation triangle).

Based on my research I have tried without success the following:

re-install NIC drivers

re install all chip set drivers

tried the CT drivers

disable certain network services such as Bonjour (disastrous)

Reboot the router, cable modem, etc.

(Never had same problem multiple pC and mac on same network. )

Any thoughts among you enthusiasts? This is my first build and it works great otherwise.

DZ77GA-70K

i3770k

EVGA 550Ti

16GB Vengeance RAM

cherrystone 240GB SSD

Seagate 1TB HDD

W7 Pro

Thanks

[idata]

By way of respons to David van Hoose's worthless reply, I thought I would mention that I and others have at various points thought we fixed this problem by installing different drivers and changing settings. It is easy to be fooled because the problem appears directly related to how much network traffic the 82574L sees. If you don't push a lot of bits through it, it will seem like there is not a problem. Then when it hits the threshhold, the NIC fails and brings down Windows with it. This is independent of driver, driver version, network equipment connected to it, etc.

So if you do think you have it fixed, be sure to test by running as much data through it as possible. I'm not sure the exact threshold because I don't have all day to do Intel's QA for them (and they're not paying me). But I saw the lockup reasonably quickly (< 1 hour) when transferring system images to a freshly booted DZ77GA. I had to finish the transfer using an external NIC.

WHICH, TO REMIND DAVID VAN HOOSE, DID NOT FAIL, DESPITE USING THE SAME NETWORK CABLE, SWITCH, LAN AS THE 82574L.

[idata]

Just put this machine together two weeks ago and am experiencing the exact same issue that the rest of you are.

I've tried every driver and every fix imaginable and thus far the only fix I can see will be to buy another NIC

Win7 x64, i3770k, 32GB of RAM

[idata]

The link below may answer your questions regarding random failures of the Intel 82574L. Hint: It's not the software or the drivers.

https://isc.sans.edu/diary/Intel+Network+Card+82574L+Packet+of+Death/15109 ISC Diary | Intel Network Card (82574L) Packet of Death

It would be most helpful if someone from Intel would post a link to the fix download, because it is not readily available on the net.

[KKiel]

Hi Otto,

Thanks for posting this. As I say in my http://blog.krisk.org/2013/02/packets-of-death.html original blog post I'm actively looking for people who are able to reproduce this issue.

Gaymaster - Are you able to reliably trigger this issue following http://www.kriskinc.com/intel-pod these instructions?

[idata]

Hi Kristian,

Intel released driver version 18.0 last month but did not put it on the DZ77GA-70K driver page; you can find it by searching for the 82574. The new update fixes the NT kernel meltdown by having errors reported to the kernel. That allows the kernel to remain sane if you disable the device. It does not fix the failure, but is a step forward.

http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=21642&lang=eng&OSVersion=Windows%208%2C%2064-bit*&DownloadType=Drivers http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=21642&lang=eng&OSVersion=Windows%208%2C%2064-bit*&DownloadType=Drivers

Since I know that the failure still exists with the latest BIOS and driver, I would like to find a predictive test for this issue so I can report it on my Intel support ticket. Do you have any tests that work with Windows and its tools, or any .NET or Win32 source code that I can run?

[idata]

So this is going to be a bit long but since Kristian and his team has done a tremendous effort I want this to be as thorough as possible.

I have been suffering from the NIC going down since I got the motherboard in September, and up until this week I haven't been able to fix the NIC problem, neither BIOS or driver updates have helped. The NIC went down at random intervals even with very little traffic.

Currently my DZ77GA-70K is with bios 0061 and operating system is Windows 7 x64. Until a week ago, driver version for the 82574L NIC was 11.17.27.0 (2012-06-18). It wasn't the newest available, just something that was left in use after trying different driver versions, but it didn't matter because the newer ones didn't work either.

Then I found out that Intel had released version 18 of their Driver CD, I downloaded it from http://downloadmirror.intel.com/22283/eng/18_0%20_CD.zip http://downloadmirror.intel.com/22283/eng/18_0%20_CD.zip . This was about a week ago. I ran the autorun.exe and selected "Install Drivers and Software". After it was done, I noticed that it had updated the Intel PROSet Version to 18.0.1.0 when looking at the "Link Speed" tab in the NIC properties. But sadly, in Driver tab I found that the driver hadn't updated at all, it was still the one I had had before. Now for some reason, don't know why, I decided to try to manually update the driver with the "Have Disk"-method, and pointed Windows to \PRO1000\Winx64\NDIS62 folder of the driver package. To my pleasant surprise, the 82574L updated to version 12.4.38.0 (2012-11-29).

So for about a week, the 82574L NIC hasn't gone down after updating to the new version. Though the interface has been on a very light load cause it is connected to a secondary network.

To the testing method, Kristian seems to be a Linux Wizard and may expect a bit too much from the general audience, I don't think many of us have previous experience of tcpreplay. Since my Linux experience is limited and I don't have a linux box readily configured, I had to go through a bit of trial and error. Hopefully this helps others if you are willing to try to help Kristian out.

I downloaded and installed Oracle VirtualBox to my Win8x64 laptop. Then I downloaded "Linux Mint 14 "Nadia" - MATE (32-bit)" .iso image from http://www.linuxmint.com/edition.php?id=119 http://www.linuxmint.com/edition.php?id=119 . Apparently Mint is the new flavour-of-the-month distribution, and for some reason I couldn't see the "tcpreplay" in the Ubuntu software center when I tried to run 12.04 Live CD I had previously made. So I decided to make a proper install to a virtual machine, because previously I have had trouble installing software to Live CD instances anyway.

I made a new Virtual Machine, selected "Linux 2.6". I chose "Bridged Adapter" for the network adapter and attached it to the LAN port of my laptop. I enabled "Enable PAE/NX" cause it seemed to require it (had to boot Mint in safe mode from the .iso because the normal mode didn't seem to work). After the Mint Live CD had loaded on the Virtual machine, I installed the Mint on the virtual machine.

After the virtual machine had rebooted, I chose Menu -> Software Manager and searched for "tcpreplay". I found one hit, I double-clicked it and chose "install".

Then I started Firefox on the Virtual Machine and downloaded both the pcap files from Kristians' site. Started "Terminal" and typed "cd Downloads". Now I was ready to start things out.

So the laptop and my desktop are connected to same ethernet switch. First I had connection to the other NIC (82579V) on the motherboard. I started Wireshark and started capturing the traffic on the 82579V NIC, just to know that the packets are actually coming through. I ran the "sudo tcpreplay -v -i eth1 pod-icmp-ping.pcap" and "sudo tcpreplay -v -i eth1 pod-http-post.pcap" on the Mint virtual machine and observed the Wireshark capture. Both packets came through, so I knew my setup was actually working.

I stopped the capture on 82579V and started capturing on 82574L. I switched the cable from 82579V to 82574L and the packets started flowing on Wireshark. I ran the commands on the Mint Virtual Machine and both packets came through on Wireshark, and what do you know, the NIC didn't die, connection was still on. So perhaps the new driver had actually fixed the problem I thought.

But be thorough, I disconnected the cable again, went to Device Manager and Rolled Back the 11.17.27.0 driver that only about a week ago had given me trouble. Just to be on the safe side, I rebooted the computer, started to capture Wireshark, connected the cable to 82574L and ran the commands. Both packets arrived and NIC didn't die, connection was still on.

Then I started my Desktop with the before mentioned Ubuntu 12.04 Live CD. It was giving me some error when I tried to install Wireshark on it, so unfortunately I couldn't actually verify whether the packets arrived from the Mint or not, but the connection was still running.

Kristian, is it possible that the 12.4.38.0 driver actually also updated the EEPROM on the NIC and the problem is gone now? I don't know how to dissect the driver files to find out what they are actually doing, but perhaps you have means to check out the files from the driver package. Or then again, maybe these problems are only related - it might be some other weird Ethernet frame combination that makes the 82574L malfunction on DZ77GA-70K, not the same combination that is giving trouble with you hardware.

Let me know if there's anything else I could assist with.

[idata]

The SANS Internet Storm Center website https://isc.sans.edu/diary/Intel+Network+Card+82574L+Packet+of+Death/15109 ISC Diary | Intel Network Card (82574L) Packet of Death has an easy method to test for it under Linux:

"For example, the commend: "ping -p 32 -s 1110 x.x.x.x" can crash an affected card remotely."

I have not been able to replicate this on a SuperMicro X7SPA-HF that has this chip. However, Kristian says in his blog that the versions of the chip with a BMC as not susceptible. These will mostly be found on server boards.

[idata]

So forget what I said about NIC seeming to be working. After the last post, I updated the driver back to version 12.4.38.0 and decided to start really putting 82574L to a test. When I started the transfer, the NIC crashed literally under 10 seconds. What a bummer. Rebooted and fired up Wireshark on both machines and started moving some data from my LAN server to the desktop computer. I managed to crash the NIC in 3 separate occasions. First attempt took about 3GB, 2nd attempt about 700MB and 3rd attempt almost 7 GB.

So I now have the Wireshark captures saved and comparing the captures I can roughly see at which point the desktop stopped receiving data from the server. My plan was to take the last few packets sent from the server and then use tcpreplay to play back the packets. However, when I export the packets to a .pcap file and try send them with tcpreplay, it just complains " Warning in send_packets.c:send_packets() line 178: Unable to send packet: error with PF_PACKET send() [1]: Message too long (errno = 90) ".

It has something to do with the packets being too large on the server end of the capture, I mean way over the 1500 bytes.I don't have jumbo frames enabled. I googled a bit and I think it's caused by "Large Send offload" enabled in the server's NIC. If there is no other way I guess I have to disable it and try to make it crash again. But it'll have to wait until tomorrow. Too bad all the samples I have now are wasted if there isn't a way around the problem in tcpreplay.

[idata]

Omonnig, I also tried the ping method from Mint Virtual Machine but it didn't crash the NIC.

I can't see to be able to replicate the problem accurately. Today I took off the "Large Send offload" from the NIC settings on the server. I managed to capture the moments of 82574L going down. I searched from the server Wireshark capture the last packet that came through to the desktop, then exported couple of frames before it and to the end until TCP retransmissions started.

I then replayed the capture on Mint Virtual Machine, but it didn't get through to the desktop computer. I noticed that in your example capture files, you had rewritten the mac addresses, so I then used the tcprewrite to rewrite the mac addresses to same as in your example. I replayed the rewritten capture and it came through to desktop, but 82574L still continued to work.

So I don't know what to do anymore - I approximately know where it might go wrong, but when I replay the capture the NIC doesn't break. So at least on this motherboard it doesn't seem that a single packet brings in interface down, it is something more complicated and I have no idea how to proceed further.

If there's anything positive about this, at least I forced myself to learn something new about lots of different things relating to the matter.

[idata]

I just updated Bios to version 0064. Also downloaded and installed the 18.1 Ethernet drivers, however, the package doesn't contain new drivers for 82574L. I even checked the .inf file, but the latest driver is still 12.4.38.0 (2012-11-29) which I had running already.

Anyway, the Bios update didn't help, 82574L lost connection after a few GB of data transfer. 6 months and counting and still no solution for the problem in sight.

[RStep8]

Good day, i have the same problem with 82574L,loosing network connection , and I have a little descicion of this issue. I took a little script :

cmdow @ /HID

:ping

set i=0

:noping

If %i%==5 (

echo %date% - %time%:Reconect>>auto.log

c:\windows\System32\devcon.exe disable PCI\VEN_8086&DEV_10D3&SUBSYS_00008086&REV_00

timeout /t 5

c:\windows\System32\devcon.exe enable PCI\VEN_8086&DEV_10D3&SUBSYS_00008086&REV_00

timeout /t 90

goto :ping)

Ping -n 1 -w 1000 192.168.5.2|Find "TTL=">nul

If %ErrorLevel%==0 (

goto :ping

) Else (

echo %date% - %time%:%i% ERROR>>auto.log

Ping -n 1 -w 1000 192.168.5.3|Find "TTL=">nul

If %ErrorLevel%==0 (

echo %date% - %time%:Error Lost>>auto.log

goto :ping

) Else (

set /A i=i+1

goto :noping)

)

it is starting with cmdow.exe and pinging two devices in our "problem " network,and if two of them are unreachable , devcon.exe utility restarts the port of the network card by device ID in device manager (PCI\VEN_8086&DEV_10D3&SUBSYS_00008086&REV_00 in my case) devcon.exe and cmdow.exe,you can download from internet and also read about them any information . I have 16 servers with this problem =( and this script is helping me a little.