Intel X540-T1 periodically lost link

[NizhegorodtsevVS]

Hello, I have a personal server on my home network. This is Hyper-V Server 2019, it has Intel X540-T1 installed which is connected to ASUS XG-C100C in my personal computer. The server runs pfSense as my main router (using SR-IOV) and a few other VMs. The problem is that communication is lost from time to time. This happens about 2-3 times a day, for 5-60 seconds. I was unable to identify the exact pattern of the timing of this failure. Before that, ESX-i was used instead of Hyper-V, and this problem also occurred there.

At first, I thought it was overheating. I added a fan to the radator of my network card, but that didn't help. I also launched a script in the system, logging the state of the network card, using - "Get-IntelNetAdapterStatus -Name" Intel (R) Ethernet Converged Network Adapter X540-T1 "-Status General" once a second. But when the link was lost, this command reported that the temperature was normal.

I checked the "Intel X540 (in the server) <--> Realtek GbE (instead of ASUS, in my PC)" bundle, the problem also occurs. I also have Intel I350-T4 in the server, there is no problem in the "Intel I350 (in the server) <--> Realtek GbE (in my PC)" bundle with the same cable. It seems to me no problem with the cable and my ASUS NIC.

The log on the server contains the following events that occur when a link is lost:

31.01.2021, 22:22:27, (Event ID - 22, Source - Microsoft-Windows-Hyper-V-VmSwitch): Media disconnected on NIC /DEVICE/{1494283E-45E7-4C0B-826B-39F63A058B31} (Friendly Name: Intel(R) Ethernet Converged Network Adapter X540-T1).
31.01.2021, 22:22:27, (Event ID - 24, Source - Microsoft-Windows-Hyper-V-VmSwitch): NIC /DEVICE/{1494283E-45E7-4C0B-826B-39F63A058B31} (Friendly Name: Intel(R) Ethernet Converged Network Adapter X540-T1) is no longer operational.
31.01.2021, 22:22:27, (Event ID - 27, Source - ixgbt): Intel(R) Ethernet Converged Network Adapter X540-T1 Network link is disconnected.

31.01.2021, 22:22:49, (Event ID - 21, Source - Microsoft-Windows-Hyper-V-VmSwitch): Media connected on NIC /DEVICE/{1494283E-45E7-4C0B-826B-39F63A058B31} (Friendly Name: Intel(R) Ethernet Converged Network Adapter X540-T1).
31.01.2021, 22:22:49, (Event ID - 23, Source - Microsoft-Windows-Hyper-V-VmSwitch): NIC /DEVICE/{1494283E-45E7-4C0B-826B-39F63A058B31} (Friendly Name: Intel(R) Ethernet Converged Network Adapter X540-T1) is now operational.
31.01.2021, 22:22:27, (Event ID - 32, Source - ixgbt): Intel(R) Ethernet Converged Network Adapter X540-T1 Network link has been established at 1Gbps full duplex.
And this is repeated several times in a row until the network is established at 10gb or 1gb

I am using Cat7 cable - 10 meters. The latest driver is installed on the server - 4.1.197.0

Any information will be useful, ready to provide any information that will help in solving the problem. I really hope for your help, because this has been going on for more than 3 months and I seem to be losing my mind

[Caguicla_Intel]

Hello NizhegorodtsevVS,

Thank you for posting in Intel Ethernet Communities. 

We've sent an email for you to attach the logs and PBA number of the adapter that we are requesting for. Please be reminded to reply on my email and do not post the logs requested on this thread to avoid exposing confidential information.

Kindly provide the information below that would help in checking your query.

1.) System Support Utility log of your system. This will allow us to check your Adapter details and configuration. This would also help us identify if you are using an OEM or retail version of Intel Ethernet Adapter. Kindly refer to the steps below.

a- Go to https://downloadcenter.intel.com/download/25293/Intel-System-Support-Utility-for-Windows-?product=91600

b- Open SSU.exe

c- Mark the box "Everything" and then click "Scan".

d- When finished scanning, click "Next".

e- Click on "Save" and attach the file to a post.

2.) PBA of the adapter. You may refer to the link below on where to find the PBA number. Providing photos of the adapter focusing on the markings (white sticker) found on the physical card will be highly appreciated for us to double check on it. The PBA consists 6-3 digit number located at the last part of the serial number.

https://www.intel.com/content/www/us/en/support/articles/000007022/network-and-i-o/ethernet-products.html

3.) Just to clarify, you mentioned 'I checked the "Intel X540 (in the server) <--> Realtek GbE (instead of ASUS, in my PC)" bundle, the problem also occurs. I also have Intel I350-T4 in the server, there is no problem in the "Intel I350 (in the server) <--> Realtek GbE (in my PC)" bundle with the same cable. It seems to me no problem with the cable and my ASUS NIC.'

Does this mean that you have the identical setup when Intel X540 (in the server) <--> Realtek GbE (instead of ASUS, in my PC) and Intel I350 (in the server) <--> Realtek GbE (in my PC) are tested and the only difference when testing is the Intel X540? Are you using the same server system when X540 and I350 were tested?

Could you please elaborate the Intel X540 (in the server) <--> Realtek GbE (instead of ASUS, in my PC) and Intel I350 (in the server) <--> Realtek GbE (in my PC)? Does your ASUS PC has an embedded Realtek LAN? 

4.) What is the exact model of your server? 

5.) Are X540 and I350 both installed on the server at the same time when carrying out the test?

We look forward to your reply.

Should there be no response from you, I’ll make sure to reach out after 3 business days. 

Best regards,

Crisselle C

Intel® Customer Support

[NizhegorodtsevVS]

Hello, Crisselle Franz, thanks for the quick response.

1)  I cannot start the SSU utility because I have a Hyper-V Server (not Windows Server) without a GUI and it cannot be started (I get the error: This product not supported on this operation system). Are there any other options for this utility or another way?

2) Sent by email

3) Yes, that's right, these are identical setup. Intel X540 and Intel I350 are installed in the same server at the same time. Yes Realtek LAN is an embedded LAN in my motherboard.

4) This is a home self-assembled server with the following components:
- Motherboard - ASRock X470D4U
- Processor - AMD Ryzen 5 2600X
- RAM - Crucial 16GB DDR4 ECC UDIMM x2

5) Yes, they were installed at the same time, but the error occurred earlier, when there was only Intel-X540.

[Caguicla_Intel]

Hello NizhegorodtsevVS,

You're welcome, appreciate your effort in sending the requested information.

Please give us time to check on your query. We will get back to you as soon as possible but no later than 2-3 business days.

Hoping for your kind patience.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello NizhegorodtsevVS,

Good day!

We hope you don't mind providing additional information below that would help in further checking the issue and for some clarifications.

1.) Can you provide a photo of the other side of the adapter? This is for us to check if there is a BradyID or Yottamark sticker on it? (Please do send the photos on our email thread)

2.) Can you clarify that the problem is only with Intel X540-T1 regardless of which controller it is connected to?

3.) Have you tried connecting the Intel I350 to ASUS XG-C100C on your personal computer?

4.) Where was the driver downloaded from and have you tried an older version of driver from Intel?

Looking forward to your reply.

We will follow up after 3 business days in case we don't receive a reply. 

Best regards,

Crisselle C

Intel® Customer Support

[NizhegorodtsevVS]

Hello, CrisselleF!

1) I sent a photo, from all sides, of all numbers by email. I have BradyID on my card
2) Yes, I think the problem is in the Intel x540, regardless of what it is connected to.
3) Yes, today I spent the whole evening at the computer in this configuration (Intel I350 to ASUS XG-C100C) and did not get lost connections.
4) I downloaded the driver from your official site (here: Download Intel® Network Adapter Driver for Windows Server 2019*). I have not tried older drivers, which version should I try?

[Caguicla_Intel]

Hello NizhegorodtsevVS,

Thank you for providing the requested information.

Please allow me to further check this with our engineers. We will get back to you as soon as heard from them but no later than 2-3 business days.

Hoping for your kind patience.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello NizhegorodtsevVS,

Thank you for the patience on this matter.

We hope you don't mind providing additional information below as requested by our higher level engineer.

1.) Did you encounter the issue on first integration? If not, any software and hardware changes before the connection issue encountered?

2.) Can you try using the X540 adapter on other PCIe slot of the server board?

3.) Driver version 26 is already available. Will you be able to try this first and check if there will be any improvement?

https://downloadcenter.intel.com/download/22283/Intel-Ethernet-Adapter-Complete-Driver-Pack

Looking forward to hear from you.

Should there be no reply, we will follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[NizhegorodtsevVS]

Hello, sorry for the long answer.

1) Yes, this is the original configuration and there were no changes to it, and the problem persists throughout the entire period.
2) Ok, I'll check it in another PCI slot
3) Installed, I will watch.

I also bought a better quality 15m, cat 7 cable.

Since I do not know how to quickly check if the error still occurs, I may need additional time, I will let you know about the results within 3 days, thanks.

[Caguicla_Intel]

Hello NizhegorodtsevVS,

No problem at all, appreciate your effort in sending us a reply.

We will patiently wait for your update on this thread. If in case we don't hear from you, we will follow up after 3 business days. 

May you have a good day!

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello NizhegorodtsevVS,

Good day!

This is just a follow up to check the current status of your system. We hope that you are not encountering any issues with it. If in case you need more time or if you have additional questions and assistance needed, please let us know and we will be glad to be of help.

Awaiting to hear from you,

Should there be no response from you, I’ll make sure to reach out after 3 business days. 

Best regards,

Crisselle C

Intel® Customer Support

[NizhegorodtsevVS]

Hello! I have good news. Over the past few days, no link loss. I think a shorter and better cable solved the problem. Thank you for your help in resolving the issue, the topic can be closed.

[Caguicla_Intel]

Hello NizhegorodtsevVS,

Thank you for the update.

We are glad to hear that the issue has been resolved. We will now proceed with closure as per your advise. Thank you for your time and cooperation throughout the process. Just feel free to post a new question if you may have any other inquiry in the future as this thread will no longer be monitored.

May you have an amazing day!

Best regards,

Crisselle C

Intel® Customer Support