10 Gigabit X540-AT2 NIC to 10 Gigabit X540-AT2 NIC has higher latency than I350 Gigabit NIC to 10 Gigabit X540-AT2 NIC

[zchen111]

I have a server s1 on machine m1 with I350 Gigabit NIC now, and it connect a other server s2 on machine m2 with 10 Gigabit X540-AT2 NIC. latency of s1 to s2 became higher when I move the s1 to machine m3 with 10 Gigabit X540-AT2 NIC, especially the p99 latency became 100ms from 50ms before. all my

os infomation is:

Linux rank10.adsys.zzzc.qihoo.net 2.6.32-220.7.1.el6.2.0.0.9.x86_64 #1 SMP Fri Jul 24 17:07:23 CST 2015 x86_64 x86_64 x86_64 GNU/Linux

I350 Gigabit NIC:

     UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

     RX packets:358096105778 errors:0 dropped:0 overruns:0 frame:0

     TX packets:609683976894 errors:0 dropped:0 overruns:0 carrier:0

     collisions:0 txqueuelen:1000 

     RX bytes:298975617063597 (271.9 TiB) TX bytes:604636509565365 (549.9 TiB)

     Memory:df120000-df140000 

10 Gigabit X540-AT2 NIC:

     UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

     RX packets:654625111 errors:0 dropped:0 overruns:0 frame:0

     TX packets:987400355 errors:0 dropped:0 overruns:0 carrier:0

     collisions:0 txqueuelen:10000 

     RX bytes:659065648587 (613.8 GiB) TX bytes:932960315226 (868.8 GiB)

ixgbe dirver:

driver: ixgbe

version: 3.4.8-k

firmware-version: 2.4-2

bus-info: 0000:07:00.0

And I use netperf to test the lantency from m1 to m2, m3 to m2, details are:

m1 to m2:

Minimum Latency Microseconds,Maximum Latency Microseconds,99th Percentile Latency Microseconds,Mean Latency Microseconds,Stddev Latency Microseconds

3310,6525,3874,3449.92,118.93

m3 to m2:

Minimum Latency Microseconds,Maximum Latency Microseconds,99th Percentile Latency Microseconds,Mean Latency Microseconds,Stddev Latency Microseconds

645,41216,981,807.27,367.51

the max latency is higher from m3 to m2。

[Crisselle_Intel]

Hello zchen111,

Thank you for posting in Intel Ethernet Communities. 

Please provide the following information for us to better understand your environment.

1.) I have a server s1 on machine m1 with I350 Gigabit NIC now, and it connect a other server s2 on machine m2 with 10 Gigabit X540-AT2 NIC. 

Does this mean that you have 2 servers with NIC I350 and X540-AT2 on each installed? Please clarify that it connects to other server. Is it connected on each other? Please share more details on this configuration.

Is it Server 1 having 1350 installed and server 2 with X540-AT2?

2.) Are both adapters/NIC are PCIe cards? Or is it embedded on the mother board?

3.) Please share the exact models of your servers.

4.) Please share the drivers loaded on each server/NIC and the exact operating system used.

5.) What do you mean by m3? Does this mean that you have 3 systems?

Looking forward to your response.

In case we don't hear from you, we will follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[zchen111]

hi CrisselleC_Intel

thank you for your response。

1）there are three different machine, configure are:

s1:

Linux version 2.6.32-220.7.1.el6.2.0.0.9.x86_64 (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Fri Jul 24 17:07:23 CST 2015

07:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

07:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

nic driver:

driver: igb

version: 3.0.6-k

firmware-version: 1.6-1

bus-info: 0000:07:00.0

s2:

Linux version 2.6.32-220.7.1.el6.2.0.0.9.x86_64 (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Fri Jul 24 17:07:23 CST 2015

01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

07:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 (rev 01)

07:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 (rev 01)

nic driver:

driver: ixgbe

version: 3.4.8-k

firmware-version: 2.4-2

bus-info: 0000:07:00.0​

s3:

Linux version 2.6.32-220.7.1.el6.2.0.0.9.x86_64 (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Fri Jul 24 17:07:23 CST 2015

01:00.0 Ethernet controller: Intel(R) I350 Gigabit Network Connection (rev 01)

01:00.1 Ethernet controller: Intel(R) I350 Gigabit Network Connection (rev 01)

07:00.0 Ethernet controller: Intel(R) Ethernet Controller X540-AT2 (rev 01)

07:00.1 Ethernet controller: Intel(R) Ethernet Controller X540-AT2 (rev 01)

nic driver:

​driver: ixgbe

version: 5.6.5

firmware-version: 0x80000625

bus-info: 0000:07:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

2) they all are PCIe cards

3) server are rpc with socket based on epoll

4) information are in point one

5) we have three machine, for same client and server, s1(client)->s2(server) are ok，s3(client)->s2(server) the latency are extremly higher

And we upgrade the driver on s3 to 5.6.5， the proglem can't solve yet。

[zchen111]

thank for your response again.

we upgrade our OS of machine s3 to new version, the problem is solved.

new version is:

Linux version 4.18.0-147.el8.x86_64 (gcc version 8.3.1 20190507 (Red Hat 8.3.1-4) (GCC)) #1 SMP Wed Dec 4 21:51:45 UTC 2019

[Crisselle_Intel]

Hello zchen111,

You are welcome.

We are glad to hear that the problem was now resolved. Just feel free to post a new question if you may have any other inquiry in the future. We will now proceed in closing this request.

May you have an amazing day!

Best regards,

Crisselle C

Intel® Customer Support