82599E/X520 Link status misdetection ESX 6.7

[MDKnk]

Hello,

our X520 Adapters sometimes do not detect the link down status when we do a switch reboot on ESX 6.7 which leads to loosing conectivity when data is transmitted over the down link of a redundant uplink in ESX 6.7.

It can be reproduced every 5-10 switch reboots. We involved server support, VMware support and switch support.

We tried driver 1.7.10, 1.7.20 and 1.8.7. With 1.8.7 we were not yet able to reproduce the issue - but according to release notes there was nothing fixed in that area. How can we find out if there were some internal changes which were not mentioned in the release notes?

Edit: We also tried different switches, notwork cards, switch firmwares and SFPs. It ONLY happens with these X520 network cards.

[AlfredoS_Intel]

Hi Mdknk,

Thank you for posting in our Intel® Ethernet Communities Page.

May we ask where you downloaded the driver?

May we know the brand and model of your system? Is the X520 built-in to the system board?

We look forward to hearing from you. If we do not get your reply, we will follow up after 3 business days.

Best Regards,

Alfred S

Intel® Customer Support

[MDKnk]

Hello Alfred,

we got the driver from here:

https://my.vmware.com/en/web/vmware/downloads/details?downloadGroup=DT-ESXI-67-INTEL-IXGBEN-187&productId=742

The card is a X520, installed (not on board but plugged in PCI) in Fujitsu M2 server systems (several different models from this series are affected).

[AlfredoS_Intel]

Hi Mdknk,

Thank you for your response.

As much as we would like to answer your question regarding what changes are present on the newest driver version, for VMWare, they have their own drivers since they have a completely different system. It would be best to contact them to find out what changes and improvements have they included on the driver releases.

We look forward to your reply. Should we not get your reply, we will follow up after three business days.

Best Regards,

Alfred S

Intel® Customer Support

[MDKnk]

Hello,

they told me the driver is not from VMware but the hardware vendor - which is Intel!!

Markus

[AlfredoS_Intel]

Hi Mdknk,

Thank you providing those information.

Please allow us some time to check on this. 

We will get back to you no later than 3 business days from now.

Best Regards,

Alfred S

Intel® Customer Support 

[AlfredoS_Intel]

Hi Mdknk,

Thank you for waiting for our update.

As much as we would like to assist you, our team confirms that all VMware driver inquiries should still be handled by VMware support because they are ones who validate and maintain hardware compatibility guides for various Intel components.

We went ahead and looked further on the link that you have provided where you downloaded the 1.8.7 driver. The release notes can actually be found on the same link that you have provided. It is in Release_Note_ixgben_1.8.7. You just need to create an account to download it. For your convenience, we went ahead and attached the file to this post.

For any other future concerns regarding VMWare drivers, please follow the instructions on this page, https://www.intel.com/content/www/us/en/support/articles/000005693/network-and-i-o/ethernet-products.html

We hope for your understanding regarding this. We will keep the thread open for 3 business days, in case you have further questions or feedback.

Best Regards,

Alfred S

Intel® Customer Support

[MDKnk]

Hello,

as we have said, VMware always tells me to talk to the vendor which is Intel.

And the release notes are my main problem: The issue we faced is not in there!

Markus

[AlfredoS_Intel]

Hi Mdknk,

Thank you for your feedback regarding the answer of our team.

We will try to check this one out and see if we can really provide any information or if we have any information regarding your inquiry. If the recommendation is still the same VMWare really has all the information, and you may need to insist. You can inform them that we suggested for you to do it.

Please allow us some time to check on this. We hope for your understanding regarding this

We will get back to you no later than 3 business days from now.

Best Regards,

Alfred S

Intel® Customer Support

[AlfredoS_Intel]

Hi Mdknk,

Thank you for your patience in waiting for an update.

Our engineers shared their initial findings regarding the VMWare driver. They said that the 1.8.7 driver is not a production driver out-of-tree driver and are surprised why you are the one contacting us. It should be VMWare contacting us on your behalf.

With that said, since you raised the issue with us, and your request is very important to us, our engineers will further check from engineering and developmental level, to provide you the answer that you require.

We will try to get back to you with an answer no later than 3 business days from now; although, we will reach out to you in case we need more information or if we already have developments regarding your concern.

Best Regards,

Alfred S

Intel® Customer Support

[MDKnk]

Any news?

[AlfredoS_Intel]

Hi Mdknk,

Thank you for your patience in waiting for an update.

Our engineers are currently working with the Intel ESX Development team to try to reproduce the issue that you are experiencing.

We will try to get back to you with an answer no later than 5 business days from now; although, we will reach out to you in case we need more information or if we already have developments regarding your concern.

Best Regards,

Alfred S

Intel® Customer Support

[MDKnk]

Do you want to have our VMware ticket number? We uploaded logs to VMware...

Markus

[AlfredoS_Intel]

Hi Mdknk,

We appreciate your offer to provide the VMWare ticket number.

It would really help us if you provide the ticket number. We would also deeply appreciate if you will provide us the logs that you shared with VMWare, as we have no have access to the logs.

Currently, our Intel ESXi development team are still debugging this issue

We look forward to your reply. Should we not get your reply, we will follow up after three business days.

Best Regards,

Alfred S

Intel® Customer Support

[MDKnk]

Hello,

is there a way to do the whole thing from here on non-publicly?

Mdknkde

[AlfredoS_Intel]

Hi Mdknk,

Thank you for your quick response and cooperation on this.

Yes there is a way for you to send things privately. We will send you an email shortly. Please reply with the information that we have asked.

We look forward to your reply. Should we not get your reply, we will follow up after three business days.

Best Regards,

Alfred S

Intel® Customer Support

[AlfredoS_Intel]

Hi Mdknk,

Thank you for providing this information.

We will continue checking this and the information that provided is invaluable to us.

Please allow us some time to check on this. We hope for your understanding regarding this

We will get back to you no later than 3 business days from now.

Best Regards,

Alfred S

Intel® Customer Support

[AlfredoS_Intel]

Hi Mdknk,

Thank you for your patience in waiting for an update.

Our Intel ESX development team were not able to reproduce the issue at this time, so they are having a meeting to determine next steps.

Please allow us more time to check on your concern.

We will try to get back to you with an answer no later than 3 business days from now; although, we will reach out to you in case we need more information or if we already have developments regarding your concern.

Best Regards,

Alfred S

Intel® Customer Support

[AlfredoS_Intel]

Hi Mdknk,

Thank you for your patience in waiting for an update.

Our engineers have a debug driver that they would like to ask your cooperation to test. The debug driver has small change that development believes may fix the issue.

The debug driver is highly confidential and because of this we need to know if you have an Non-Disclosure Agreement (NDA) form with us.

We have sent you an email as there will be a need to gather private details.

We look forward to your reply on the email.

Best Regards,

Alfred S

Intel® Customer Support

[MDKnk]

The thing is that the issue seems to be solved in the last version indeed (which is strange because nothing was in the release notes)  - i just want to know if that is realistic our if we did not do enough testing (unlikely)...

[AlfredoS_Intel]

Hi Mdknk,

Thank you for your reply.

We will check again with our engineers if we will pursue the debug driver option or to provide more insights on the current release since the issue was never reproduced on the new version.

Please allow us some time to check on this. We hope for your understanding regarding this.

In case they really want to provide debug driver we will use the details on the email for approval.

We will get back to you no later than 3 business days from now.

Best Regards,

Alfred S

Intel® Customer Support