topic Re: Intel X710 woes in Ethernet Products

Intel X710 woes

DHend8 — Mon, 23 Apr 2018 22:13:38 GMT

We have 8 brand new HPE DL380 Gen10 servers. Each of these servers has two HPE Ethernet 10Gb 2-port 562SFP+ Adapters. One is embedded and one on a PCI card. This card is based on the Intel X710 controller

https://www.hpe.com/us/en/product-catalog/servers/server-adapters/pip.hpe-ethernet-10gb-2-port-562sfp-adapter.8245220.html HPE Ethernet 10Gb 2-port 562SFP+ Adapter OID8245220 | HPE™

We hooked two DAC cables, one from each card to our Juniper switch. The two ports on the Juniper switch are standard access ports. After installed the HPE customized version of ESXi 6.5 U1 we went into the console and added the two nics. We put in an IP address, gateway, mask, and DNS servers and then rebooted the host. During reboot we had a continuous ping going to the IP address of the ESXi management interface. The ping returned part way through the boot process and after fully booted the ping stopped. If I remove one of the 10Gb nics from the management network, the ping returns. If I use two of the 1Gb interfaces on this server which are based on a Broadcom chipset, the management interface works fine. The server has had all of its firmware upgraded. We are running driver version 1.5.8 and firmware version 10.2.5. The firmware came from here, I think it corresponds to VMWare's version of firmware 6.00

https://support.hpe.com/hpsc/swd/public/detail?sp4ts.oid=1008830010&swItemId=MTX_87c83853cb5a4bc5949e9b0dd5&swEnvOid=4184# tab1 Drivers & Software - HPE Support Center.

I have a ticket opened with HPE but wanted to find out if Intel might have a solution for this

Re: Intel X710 woes

idata — Tue, 24 Apr 2018 01:18:30 GMT

Hi HendersonD,

Thank you for posting in Wired Communities. I am sorry to hear what happened and would like to clarify if this issue only occurred on Juniper switch. Have you tried checking with Juniper switch support?

I understand you already contacted HP* support, as this is an OEM NIC, we don't have control over the customization and modification of firmware and driver or other manufacturing changes done by the OEM NIC vendor, the OEM driver is more suitable for the OEM NIC. The driver and firmware version are different from the version for Intel X710 NIC.

As mentioned if you removed the 10 Gb NIC from the system, the ping will continue to work, did HP confirm the network card is the compatible model for this server in customized ESXi console?

I will try to check if there is any information, please continue to work with HP for further investigation.

Regards,

Sharon T

Re: Intel X710 woes

DHend8 — Thu, 26 Apr 2018 09:26:46 GMT

This appears to be a vlan tagging issue. External Switch Tagging should work but does not, Virtual Switch tagging does work. All the details are shown below. I am thinking this has to be the nic in these servers which is based on the Intel X710. Two reasons:

If I use two of the 1Gb ports on this server for the management network using External Switch Tagging, it works just fine. The 1Gb ports are Broadcom nics
I have a 5 year old IBM server that has two 10Gb nics plugged into the exact same switch. It is setup with External Switch Tagging on the management network and it works fine. The 10Gb nics in this server are Emulex

Other people in this forum have reported vlan tagging issues with the X710 under various flavors of Linux. Could it be the driver for this card under ESXi is the issue?

Re: Intel X710 woes

idata — Fri, 27 Apr 2018 02:05:48 GMT

Hi HendersonD,

Thank you for the additional information. There is an issue of using ESX i40en driver version 1.5.8.which other user post regarding Malicious driver detection using Intel X710 series. I am not sure if you encountered that error in the log, if this is relates to malicious driver detection, we are still waiting for the next driver release. With the additional information provided, we will do some checking. Thank you.

Regards,

Sharon T

Re: Intel X710 woes

DHend8 — Fri, 27 Apr 2018 08:47:57 GMT

Which log would show the malicious driver detection error? vmkernel log or some other log?

Re: Intel X710 woes

DHend8 — Fri, 27 Apr 2018 11:21:25 GMT

Sharon,

I am convinced that the driver for this nic card is the root cause of the issues we are seeing. Why do I say that?

I have another server plugged into the same switch using External Switch Tagging and an active/active setup for the ESXi management network and it is working fine. This server uses 10Gb Emulex nics
Even with the HPE servers using the Intel X710 nic, if I use two of the 1Gb ports on this server with External Switch Tagging and active/active for the ESXi management network, it works fine. The 1Gb nics on these HPE servers use a Broadcom nic
I had a Juniper engineer look at our switch setup which is very simple and he said everything is configured correctly
I had two different VMware engineers look at our setup and said it is not a problem with ESXi. Both of them said they have seen multiple problems with X710 nics
I have spent nearly 20 hours of my time trying to troubleshoot this issue including opening tickets with Juniper, HPE, and VMware
There are numerous reports of problems with the Intel X710 on this forum, VMware's forums, and several other sites. Several of these postings talk about vlan tagging issues

I am hoping that the new i40en driver fixes this issue. If you could please pass on this information (in particular the two diagrams in this thread that show External Switch Tagging not working and Virtual Switch Tagging working) I would appreciate it. It seems I cannot get anyone yet to accept ownership of this issue. Intel makes the controller that goes in these cards, VMware makes the driver the card uses, and HPE and Dell put this card in their server products. Surely everyone can get together to solve problems with this nic that stretch back two years.

Any ETA on when the newest i40en driver will be released?

Re: Intel X710 woes

TShil — Fri, 27 Apr 2018 14:50:24 GMT

HendersonD,

Thank you for your post. We were experiencing the same issues with the x710 and have spend weeks looking for a solution. We have working servers in the cluster but our new server with the x710 has management network failure after reboot and only multiple restarts on the management network would keep it up. We have confirmed with Juniper that the switch configuration looks good and VMware sees nothing of note. We are currently working with Dell when we came across your post.

We have tried different firmware/driver versions as well as switch configurations and nothing has fixed the issue. When we test by installing i350 copper cards connected to the same switch everything works fine.

So far your solution is working for us as we wait for a permanent. Thanks again.

Re: Intel X710 woes

DHend8 — Fri, 27 Apr 2018 15:01:37 GMT

Are you seeing the same behavior I am

Two 1Gb connections (not the X710 card, other nics) in active/active (under portgroup and vswitch) with External Switch Tagging and management network is fine
One X710 card active for the management network with External Switch Tagging and management network is fine
Two X710 cards in active/active (under portgroup and vswitch) with External Switch Tagging and management network passes no traffic
Two X710 cards in active/active (under portgroup and vswitch) with Virtual Switch Tagging and management network is fine

Re: Intel X710 woes

TShil — Fri, 27 Apr 2018 15:49:15 GMT

Yes, we see the same.

After a 4 hour support call with Juniper JTAC we confirmed its an arp issue with the x710. The x710 arp entry is in the juniper arp table while the ICMP pings fail. If we clear the x710 arp entry in juniper the ICMP starts to reply for about 20 pings then stops again. Then we placed a layer 3 interface on the access switch that feeds the x710 uplink so we can see a trace from both switches (access & core). Doing a trace on access & core while ICMP is running both switches see the packets bi-directionaly. This proves that the packets are getting to the x710 but the x710 stops replying. We believe the issue is related directly to the x710 and an arp issue when the port modes are in access. Next we followed your solution, changed the interfaces to truck, tagged vswitch0 with the vlan # and everything works fine.

We agree that your findings are correct when the x710s are teamed and have the same question to intel, what logs should show Malicious Driver Detection.

Re: Intel X710 woes

DHend8 — Fri, 27 Apr 2018 16:12:13 GMT

So are you thinking like I am that this is all related to the i40en driver? We are using the latest, 1.5.8 and having this issue

My hope is a new i40en driver will be released next week that fixes this. It sounds like a new driver release is imminent

Re: Intel X710 woes

TShil — Fri, 27 Apr 2018 16:47:27 GMT

We are seeing a mixed result. We are running other servers in the same environment with x710s running Firmware Version 18.0.17 & Driver Version 1.3.1 with no issues. The new server came with the same Firm/Driver combo and we were seeing issues out of the box. We updated/downgraded firmware and drivers during our troubleshooting with no success.

Our theory is that there is a corrupt version of the firmware during the manufacturing process or out on vendor sites. Once applied, even when you re-flash the firmware it leaves something behind in the x710 that presents the issue.

Re: Intel X710 woes

DHend8 — Fri, 27 Apr 2018 17:18:10 GMT

I am not sure how much each vendor, Dell and HP, are involved in the firmware and drivers for these cards but there is some level of involvement. The firmware on our X710 cards is version 10.2.5 which comes from HP's site

https://support.hpe.com/hpsc/swd/public/detail?sp4ts.oid=1008830010&swItemId=MTX_87c83853cb5a4bc5949e9b0dd5&swEnvOid=4184# tab3 Drivers & Software - HPE Support Center.

The driver is version 1.5.6 out of the box which also comes from HP's site

https://support.hpe.com/hpsc/swd/public/detail?sp4ts.oid=1008830010&swItemId=MTX_03c21f88fa3447e78d78770331&swEnvOid=4184# tab4 Drivers & Software - HPE Support Center.

Of course when I go to VMware's site I see the newer 1.5.8 driver which we have tried along with firmware version 6.01

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=37994 VMware Compatibility Guide - I/O Device Search

Not sure if HP's firmware version 10.2.5 is the same as VMware's 6.01. There is also the i40e driver version 2.0.7 on the VMware page which goes along with firmware 5.05

The version of the Dell firmware and driver are completely different

Just waiting for the right combination of firmware and drivers that makes all of this work

Re: Intel X710 woes

TShil — Fri, 27 Apr 2018 23:29:00 GMT

We tried every combo of Dell firmware and driver on the HCL with no success. If we didn't stumble across your post we would be still be guessing. There is no harm for us to trunk the ports and use Virtual Switch Tagging so for now we will go with your solution.

Thanks again.

Re: Intel X710 woes

DHend8 — Sat, 28 Apr 2018 02:01:32 GMT

In our case these are brand new servers that we just took delivery a few weeks back. These are replacing all of the 5 year old servers in our data center and DR site. Each server has two of the HPE nics based on the Intel X710 chipset with each nic having two 10Gb SFP+ ports. In our data center we are using the 1Gb broadcom nics for the management network. In our DR site though we are using two of 10Gb nics for the management network. Normally the management network is a low bandwidth application but we use Veeam for backup and replication. During replication our Veeam proxy server reads data directly from our SAN but writes data through vCenter. This means the target replication traffic goes through the management network, hence the need for 10Gb nics.

We will be using two of the 10Gb nics to carry iSCSI, VM, and vMotion traffic. If we do use trunk ports for the management network and Virtual Switch Tagging, my worry is what is going to happen on the other 10Gb ports. In other words, if I put these servers into production the way they are, will I see problems with storage, VM, or vMotion traffic? I am waiting to hear when the new i40en driver will be released and hopefully it will fix this problem. If you Google "Intel X710 ESXi" or "Intel X710 Linux" it is easy to find many cases where this nic has had issues stretching back nearly two years.

As soon as the new i40en driver is released and we have tested it, I will report back

Re: Intel X710 woes

DHend8 — Sun, 29 Apr 2018 17:40:35 GMT

Maverick85,

Any chance you could pass along your JTAC case number? I would like to give the Juniper engineer I am working with as well as the 3rd level support engineer at HPE any information that might help them solve this issue

Re: Intel X710 woes

nbala3 — Mon, 30 Apr 2018 11:33:52 GMT

Hi Everyone,

I am also facing similar issue. I am working on a HP DL560 Gen 10 server with HPE Ethernet 10Gb 2-port 562SFP+ Adapter.

[root@:~] ethtool -i vmnic6

driver: i40e

version: 2.0.7

firmware-version: 6.00 0x8000366c 1.1825.0

bus-info: 0000:4e:00.0

Above is the details of the current driver and firmware version I used. We are having intermittent performance issue while accessing remote desktops of VMs,

virtual machine hangs and slowness in application performance.

As recommended by HPE, I have updated the driver to ESXi 6.0 i40en 1.5.8.

Also ran the commands to disable the legacy driver and enable the native driver.

After rebooting the server, I have moved few machines to the server and all the VMs lost network connectivity.

It looks the adapters are not able to pass any traffic through it.

How can I fix the issue now, I have reverted back to the driver: i40e driver that shows performance problems.

Please someone let me know a way out!!

Re: Intel X710 woes

TShil — Mon, 30 Apr 2018 17:05:23 GMT

In our setup we are using two teamed 10gb nics for each: management/veeam | vSan (all flash) | vMotion | Virtual Machine Guests (Tagged)

If you are going to be using the two 10Gb nics to carry iSCSI, VM, and vMotion traffic i would assume you will need to trunk and tag if they will be on separate subnets. If they are all on the same subnet your will need to trunk and tag to keep the x710 working properly.

We believe that it should not matter but are confirming with VMware that there is no issue to tag/trunk our vSan & vMotion even thou they are dedicated.

These are vSAN articles but they mention tagging on a shared network.

"VLAN ID – If you are using VLANs to separate vSAN traffic, enter the relevant VLAN ID."

https://kb.vmware.com/s/article/2058368

Look at section "Allocating Bandwidth for vSAN by Using Network I/O Control"

https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.virtualsan.doc/GUID-031F9637-EE29-4684-8644-7A93B9FD8D7B.html

Re: Intel X710 woes

TShil — Mon, 30 Apr 2018 17:12:56 GMT

nidhinmds,

Our issue is that we lose all network connectivity not intermittent performance.

Is your 10gb network teamed and tagged?

What switches are you using?

Re: Intel X710 woes

DHend8 — Mon, 30 Apr 2018 22:04:40 GMT

Mavericks85

HPE support gave me this command to run

esxcli system module parameters set -m i40en -p LLDP=0,0

With two access ports, two nics devoted to the management network in active/active I now have connectivity. It appears to be an issue with lldp. Since we are a Juniper show we use lldp (standard) and not CDP (Cisco proprietary). I believe the command above disables lldp which is an issue. They are still working on the root cause of the issue with LLDP in our setup. I am still thinking a driver update will be needed to fix this but we are getting closer to a root cause

Re: Intel X710 woes

TShil — Mon, 30 Apr 2018 22:12:56 GMT

HendersonD

I will give that a try tomorrow and share my results.

Thanks