X540-T2 (ixgbe) poor performance over NFS or CIFS (1~2 Gbps)

[JNore]

Hi,

Can't reach more than 140 MB/s over NFS or CIFS.

I've tried everything: recompiled the latest driver from Intel (ixgbe-3.23.2.1), tweaked "/etc/sysctl.conf", set a bigger tmpfs on a ram drive. But nothing seems to work.

My NFS server is on Debian wheezy 3.2.65-1+deb7u2 x86_64, with 2 X540-T2, Intel(R) Xeon(R) CPU E5-2603 v3 Hexacore, SSD HDD 240 GB (O.S.) and Logic MegaRAID SAS-3 3108 with 4 disk on RAID10.

My 2 NFS clients are Centos 6.6 and Centos 7.1 with X540-T2.

What surprises me the most is that I have another scenario with the same version of Debian that works like 400MB/s with less powerful hardware: Server NFS Intel Corei3 and Client NFS Centos 6.6, both with Intel X520 (ixgb driver) interface.

Please helpppp!!!!

Iperf and dd tests:

iperf

------------------------------------------------------------

Client connecting to 10.199.50.1, TCP port 5001

TCP window size: 9.54 MByte (default)

------------------------------------------------------------

[ 3] local 10.199.50.3 port 27732 connected with 10.199.50.1 port 5001

[ ID] Interval Transfer Bandwidth

[ 3] 0.0-20.0 sec 22.8 GBytes 9.80 Gbits/sec

[ 3] 0.0-20.0 sec 22.7 GBytes 9.74 Gbits/sec

[ 3] 0.0-20.0 sec 23.0 GBytes 9.89 Gbits/sec

dd test:

time dd if=/dev/zero of=/media/ISOs/testfile bs=16k count=16384 && rm -f /media/ISOs/testfile

268435456 bytes (268 MB) copied, 2,0134 s, 133 MB/s

268435456 bytes (268 MB) copied, 2,3138 s, 116 MB/s

[SYeo3]

Hi Ingenieria-INS,

Thank you for contacting Intel.

I'll help you further check on this and will update you on our findings.

Sincerely,

Sandy

[JNore]

Really tanks Sandy_Intel.

Meanwhile this issue is solved, I will implement this strategy:

http://louwrentius.com/achieving-450-mbs-network-file-transfers-using-linux-bonding.html Achieving 450 MB/s network file transfers using Linux Bonding

That is supposed to double the speed at least ~300MB/s.

My final expectation is that it reaches ~1000MB/s at least.

I'm waiting, anxiously, to see the answer.

Regards.

[SYeo3]

Hi Ingenieria-INS,

Thanks for sharing your setup here.

The network performance depends on the whole system configuration - hardware, software and settings. With this, we would like to refer you to our website for some suggestions on network adapter performance tuning. You may also check out the Linux documentations under "Related Topics".

http://www.intel.com/support/network/sb/cs-025829.htm Network Connectivity — Tuning Intel® Ethernet Adapter Throughput Performance

Sincerely,

Sandy