Intel x540-T2 adapter + ESXi 5.5 = Poor Transmit performance

[nfarm]

I found an issue and I've narrowed it down to the Intel driver.

Just to show the work I've put into this:

- I used iperf to test the bandwidth on all server types (Physical Linux install and inside a VM)

- I have an existing VMware support ticket and we've worked it down to the driver.

- I tested the issue with three different x540-T2 adapters and a x520-DA2 adapter.

Problem: When a server with the intel x540-T2 adapter is connected at 10Gbps and tries to transmit to another server that is connected at 1Gbps, All flows out of the 10Gbps adapter drops to a max of 1Gbps until the 1Gbps flows stop.

Let me clarify a bit:

Server A has an Intel x540-T2 adapter and is connected at 10Gbps.

Server B has an Intel x540-T2 adapter and is connected at 10Gbps.

Server C has a HP Server adapter and is connected at 1Gbps.

I start a Transmit flow from Server A to Server B and I see the iperf session reach full line rate (10Gbps)

I start a concurrent iperf flow From server A to Server C.

My expectation is that Server A will continue to transmit at 10Gbps (total), Server B will drop down 1Gbps to (8-9Gbps) to allow bandwidth for the 1Gbps flow to Server C.

BUT, this does not happen.

Server A will flop down to 1.5Gbps transmit and Server B and Server C will now receive 500-700Mbps until the flow to Server C stops then the flow from Server A to Server B goes back to 10Gbps.

Driver data:

If I use driver version 3.7.13.7 this problem does not exist! Everything works as expected and Server A continues to transmit at 10Gbps while Server B drops to 8Gbps+ and Server C receives at a full 1Gbps.

If I Update to any version after that, it has the transmit flop.

Driver Version 3.7.13.7 is the default driver used in the ESXi 5.5 build 1331820 install ISO.

Newer version tested: (these are listed as compatible with ESXi 5.5 on the VMware compatibility guide)

3.15.1.8

3.18.7

3.19.1

Some extra info from ESXi cli:

# This is the working Driver:#

~ # vmkchdev -l | grep vmnic

0000:01:00.0 8086:1528 8086:001a vmkernel vmnic00000:01:00.1 8086:1528 8086:001a vmkernel vmnic1

~ # ethtool -i vmnic0

driver: ixgbeversion: 3.7.13.7.14iov-NAPIfirmware-version: 0x80000389bus-info: 0000:01:00.0

~ # esxcfg-nics -l

Name PCI Driver Link Speed Duplex MAC Address MTU Descriptionvmnic0 0000:01:00.00 ixgbe Up 10000Mbps Full a0:36:9f:01:b6:28 9000 Intel Corporation Ethernet Controller X540-AT2

# This is one of the broken Driver versions:#

~ # vmkchdev -l | grep vmnic

0000:01:00.0 8086:1528 8086:0001 vmkernel vmnic00000:01:00.1 8086:1528 8086:0001 vmkernel vmnic1

~ # ethtool -i vmnic0

driver: ixgbeversion: 3.15.1.8iovfirmware-version: 0x80000389bus-info: 0000:01:00.0

~ # esxcfg-nics -l

Name PCI Driver Link Speed Duplex MAC Address MTU Descriptionvmnic0 0000:01:00.00 ixgbe Up 10000Mbps Full a0:36:9f:01:b6:28 9000 Intel Corporation Ethernet Controller X540-AT2

Things to note here are the VID and DID are still being detected correctly as 8086 and 1528.

Any thoughts or questions?

I have a ton of screen shots if needed to prove the case.

[sleo1]

Thanks. This is very useful to know as I have ESXi 5.5 installed, and was considering the purchase of a x540-T2 adapter.. I'm surprised no one from Intel has replied.

[nfarm]

Any attempt to reach Intel support results in, "Vmware is out of our scope of support."

I will continue to work on the issue.

Until then I would avoid any new purchase of this adapter if Intel refuses to support it.

Nick

[Patrick_K_Intel1]

Sorry for the tardy reply, been away for a while.

Unfortunately the answer is the same, though I can share some background as to why we do not provide support for VMware's driver. If it were truly Intel's driver we would have it available for download from our site.

However it is not an Intel driver. VMware takes our open source driver and modifies it specifically for their Operating System. It is VMware's driver, for their OS and available from their site.

I am aware that this is likely frustrating for you and others since it is Intel's hardware and a large percentage of the driver comes from our Open Source and while I empathize - it is truly a VMware piece of software that VMware is supposed to support.

If VMware needs assistance they will contact Intel and we will work directly with them.

I realize this does not help you directly or reduce your frustrations, however perhaps now you and others understand why Intel does not directly support the Network Drivers owned by VMware.

- Patrick

[KSMIT9]

the same thing happens on Linux on drivers 3.11.33-k and 3.18.7. I hope Intel fixes this soon.

[nfarm]

@ Patrick_Kutch

Im going on month three with VMware atm for this Driver update.

As theotherkevinsmith post, The driver issue is greater than just "Vmware.

VMware released a new 5.5 build and I'm working to test if the issue is fixed in this release. I will update this forum if it is resolved.

Nick

[ACova1]

Hi all,

Does anybody have any updates about this NIC issue?

I hope I'm wrong guys, but I think there is a network switch issue or 1GB NIC issue on server C. Nicholas said he had issues with the server C with 1GB NIC, but he does not have issues with server B.

[bmoff]

I have what may be related problems. How do you find driver version 3.7.13.7? Intel doesn't seem to list the numbers.

Thanks

[TTeig]

Any news on drivers for this NIC with VMware 5.5?