Monitoring tools on Windows for Intel i350 to find cause for event 27

[CVBln]

Dear Community,

I'm looking for help concerning tools which might be useful to track down an issue we've been having with multiple i350 T4 v2 network cards on a single mainboard.

The affected ports are connected to GigE cameras (utilizing about 40 % of the bandwidth) and run without a problem for a few hours or even days before we are seeing simultaneous disconnects on all ports with event ID 27 (link disconnected). I have read that it might be an issue with the cables used, but it seems to systematic that the problem is with the cables. We have four cards in total built in and the card, without any cameras connected to it, is not affected by these simultaneous disconnects.

Our system is based on Windows 7 (I know it's not support, but never fix a running system) and I would like to find the cause of the disconnects. We have tested this with the current and older drivers. We have disabled EEE and other power saving options in Windows without any effect. We have also tried disabling interrupt throttling to no avail.

There seems to be something 'overflowing' or 'peaking' at some point and I would want to monitor some of the lower level values/states. As far as I can judge the Window Performance Monitor partly gathers useful information, but not all affected (internal) values.

Can anybody recommend tools/utilities which might be useful to continuously monitor the ports/cards to see correlation between the values and the disconnects?

Thank you in advance!

[Caguicla_Intel]

Hello CVBln,

Thank you for posting in Intel Ethernet Communities. 

We understand that you are looking for monitoring tools for Intel Ethernet adapter and we will be glad checking this for you. Before we proceed with that,kindly provide the following information that would help us to further check your request.

1.) You mentioned that you have tested this with the current and older drivers, can you please share the driver version tested and where was the driver downloaded from? 

2.) Please share the PBA and serial number of the adapter. You may refer to the link below on where to find the PBA number. Providing photos of the adapter focusing on the markings (white sticker) found on the physical card will be highly appreciated for us to double check on it. The PBA consists 6-3 digit number located at the last part of the serial number.

https://www.intel.com/content/www/us/en/support/articles/000007022/network-and-i-o/ethernet-products.html

3.) Is there a specific application running when the error occurred and how often does the issue occurs?

4.) Can you share where was Event ID 27 seen? 

5.) You mentioned 'We have four cards in total built in and the card, without any cameras connected to it, is not affected by these simultaneous disconnects.

Can you share the model of the other cards? Have you tried connecting your GigE cameras to other cards and check if same issue persists? 

6.) Looking at the Supported Operating Systems for Retail Intel® Ethernet Adapters (link below), I can confirm that Windows 7 is supported on an Intel® Ethernet Server Adapter I350-T4V2

https://www.intel.com/content/www/us/en/support/articles/000025890/network-and-i-o/ethernet-products.html

Can you share more information about your statement 'Our system is based on Windows 7 (I know it's not support, but never fix a running system)?

Looking forward to your reply.

We will follow up after 3 business days in case we don't hear from you.

Best regards,

Crisselle C

Intel® Customer Support

[CVBln]

Hello,

thank you for your quick response!

1.) We downloaded and tried at least the versions 22.1 and 25.0 from the Intel download page. We also received and tested packages with two versions (22.2 and 25.0) from the motherboard manufacturer (SuperMicro). I do not have the exact versions, but we also tried versions between 22.1 and 25.0 without progress.

2.) Since the systems are off-site I'll get this information to you a.s.a.p.

3.) Our own software (which acquires images from the cameras, etc.) is running when this error occurs, but I have seen the resets occuring without our software being started. The software has previously been stable without problems with the i350 cards for multiple years.

4.) Yes, sorry that I was not explicit. We are seeing them in the Windows Event Viewer, which contains the following message (e.g.).

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="e1rexpress" /> 
<EventID Qualifiers="40964">27</EventID> 
<Level>3</Level> 
<Task>0</Task> 
<Keywords>0x80000000000000</Keywords> 
<TimeCreated SystemTime="2020-08-27T16:23:28.764684000Z" /> 
<EventRecordID>44216</EventRecordID> 
<Channel>System</Channel> 
<Computer>H-350</Computer> 
<Security /> 
</System>
<EventData>
<Data /> 
<Data>Intel(R) I350 Gigabit Network Connection #11</Data> 
<Binary>0000040002003000000000001B0004A00000000000000000000000000000000000000000000000001B0004A0</Binary> 
</EventData>
</Event>

5.) All four cards are the same with the i350 controller. It's the fourth card which is not connected to any cameras and as such does not have a high resource demand.

6.) Summarized: I opened an issue with Intel support first and I got a reply that due to Windows 7 running the legacy drivers and that the legacy driver will not be further updated, that they cannot help me. Furthermore, since supposedly Intel does not test the network cards/controllers with other manufacturers' mainboards, they cannot guarantee compatibility. That is why I am trying to track down the issue myself, because it's possible even if we upgrade to Windows 10, that we might face the same problem(s).

Thank you and kind regards!

[Mike_Intel]

Hello CVBln,

Thank you for the update and we will check these details.

And also, when can we expect the markings that requested for us to check the cards?

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days.

Thank you.

Best regards,

Michael L.

Intel Customer Support Technicians

[Caguicla_Intel]

Hello CVBln,

Good day!

We'd like to follow up the PBA and serial number of the adapter for us to continue checking on your request.

We also need some clarifications on item 3 and 6, we hope you don't mind giving us additional information.

Item 3

Our own software (which acquires images from the cameras, etc.) is running when this error occurs, but I have seen the resets occuring without our software being started. The software has previously been stable without problems with the i350 cards for multiple years.

Is there any software or hardware changes made prior to the issue?

Item 6

Summarized: I opened an issue with Intel support first and I got a reply that due to Windows 7 running the legacy drivers and that the legacy driver will not be further updated, that they cannot help me. Furthermore, since supposedly Intel does not test the network cards/controllers with other manufacturers' mainboards, they cannot guarantee compatibility. That is why I am trying to track down the issue myself, because it's possible even if we upgrade to Windows 10, that we might face the same problem(s).

Can you share the link or ticket number you had with us? This is for our reference only and we will also try to check on it. 

Awaiting to your reply

Should there be no reply, we will follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[CVBln]

Hello,

sorry, that it took so long, but I received the information just today. Some of the affected network cards:

Serialnr. 1:  H47822-001

Serialnr. 2:  H47822-001

Serialnr. 3: H47822-001

3.) We did indeed also have a software change, but have tested it with both the new and old software versions and both exhibit errors only on the new hardware. There was no other hardware change before these issues arose.

6.) The support request number is 04789822.

Can you recommend any software tools to track down the cause of the problems?

[Caguicla_Intel]

Hello CVBln,

No problem at all, we appreciate your effort in sending the requested information.

Just to clarify, 'We did indeed also have a software change, but have tested it with both the new and old software versions and both exhibit errors only on the new hardware. There was no other hardware change before these issues arose.' Does this mean that the issue only occurs to these 3 new hardware? 

Can you share how many adapters are working without any issues and provide the PBA as well? 

I'm afraid to say that I didn't find any Intel monitoring software tools for Intel Ethernet adapter. But don't worry as I will check this with our engineers to further investigate the issue and confirm with them if there is a tool they can recommend.

Looking forward to hear from you.

We will follow up after 3 business days in case we don't receive a reply. 

Best regards,

Crisselle C

Intel® Customer Support

[CVBln]

Hello Crisselle C,

thank you for your quick reply!

There are more affected adapters. We've swapped PCs and at least two other systems with the same hardware (i.e. new mainboard and i350 T4v2 cards) exhibited the same errors.

In contrast there are hundreds of the previous i350 T4 and also i350 T4v2 adapters in other systems, which run without a problem on the previous mainboard.

Since these systems are off-site and a technician would need to remove the PCs from enclosures and open the PCs, it is difficult to get the information on all adapters. Furthermore, active systems cannot be taken offline without scheduled maintenance.

Thank you for your effort!

Kind regards,

CVBln

[Caguicla_Intel]

Hello CVBln,

Thank you for the reply. 

We fully understand that the information requested is difficult to get. We highly appreciate your time and effort on this matter.

Please allow us to continue checking on your request. We will get back to you as soon as possible but no later than 2-3 business days.

Hoping for your kind patience.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello CVBln,

Good day!

While we are checking on this, will you be able to provide System Support Utility from your system? This will allow us to check your Adapter details and configuration. Kindly refer to the steps below.

a- Go to https://downloadcenter.intel.com/download/25293/Intel-System-Support-Utility-for-Windows-?product=91600

b- Open SSU.exe

c- Mark the box "Everything" and then click "Scan".

d- When finished scanning, click "Next".

e- Click on "Save" and attach the file to a post.

By the way, please be informed that we removed the Serial Numbers provided but left the PBA on this thread as this is considered confidential information. 

Looking forward to your reply.

Should there be no response, I’ll make sure to reach out after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[CVBln]

Hello Crisselle,

I will receive an affected system in the next few days and can provide the PBAs (for now I only have the MAC addresses, which are probably part of the confidential information).

When I have the system I'll run SSU and post the saved information.

Kind regards,

CVBln

[Caguicla_Intel]

Hello CVBln,

Appreciate your swift reply.

We will patiently wait for the SSU logs. Yes, MAC address is also considered confidential information. Rest assured that we will keep this thread open and once the requested information was received, this request will be discussed with our engineers. 

Should there be no response, I’ll make sure to follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[CVBln]

Hello Crisselle,

attached you'll find the SSU log of an affected system.

Kind regards,

CVBln

[Caguicla_Intel]

Hello CVBln,

Thank you for providing the information requested. 

Please allow us to continue checking on this matter. We will provide an update as soon as possible but no later than 2-3 business days.

Hoping for your kind patience.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello CVBln,

Apologies for the delay as we are still checking on this.

We hope you don't mind clarifying some thing below providing the additional information as this will help us in further investigating the issue.

1.How many embedded (LAN on motherboard) and adapters in total are affected? One embedded and 3 adapters? Are they all installed in a single system?

2.No issues encountered as long as these GigE cameras are disconnected is that correct?

3.Are these systems connected to UPS? Have you considered that it might be under powered? Have you tried connecting it directly from wall socket?

4.Did these cards disconnect randomly or at the same time in the system?

5.How many systems with similar embedded and adapters are affected?

6.Have you tried changing the switch ports for isolation?

Looking forward to hear from you.

Should there be no response, I’ll make sure to reach out after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello CVBln,

Good day!

We'd like to follow up the additional information we requested for us to continue investigating the issue.

Hoping to hear from you soon.

We'll follow up after 3 business days in case we don't receive a reply.

Best regards,

Crisselle C

Intel® Customer Support

[CVBln]

Hello Crisselle,

thank you for following up on this issue.

After our partner switched PCs and saw the same errors they were able to test a second complete system in parallel for a few days last week (due to the ongoing issues this was not straightforward to do again at first). With this system, with the same modified hardware and Intel driver configuration, they weren't seeing the same effects.

Due to this they looked at the peripherals connected to the PCs and particularly the cameras. The cameras and their connected network cables, were things that could eliminate early on from the list of possible causes. By accident they were able to observe the power supply chain of the cameras randomly, but consistently simultaneous, with the network disconnected, switch off. It turns out there was a cold soldering joint in the power supply, which randomly (after hours or even days), caused the cameras to lose power.

I apologize for the uproar of this issue, but I'm glad that this was tracked down and is not so severe as previously assumed.

Thank you again for your assistance!

Kind regards,

CVBln

[Caguicla_Intel]

Hello CVBln,

Thank you for the update.

We are glad to hear that your team was able to identify the component that's causing the issue. In case you don't have additional assistance needed, please be informed that we will now proceed closing this request. Just feel free to post a new question if you may have any other inquiry in the future as this thread will no longer being monitored.

Thank you for choosing Intel!

Best regards,

Crisselle C

Intel® Customer Support