Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Novice
1,605 Views

Windows 7 performance issue using Adaptive Load Balancing

Adaptive Load Balancing, Windows 7/x64

2 NICs/team (X520-T2 and X540-T2), 4 nodes with one team each.

2 xs708ev2 switches, one connection per team per switch.

One NIC is always inactive.

Running current releases of Intel's Fortran and MPI on both CentOS7 and Windows 7 on the same nodes. Using one or the other system, not intermixing.

On same gear, CentOS 7 makes use of both NICs.

Same benchmark on Windows 7 (Class D FT, NPB) walltime=400s. On CentOS 7, walltime = 308s.

Running the same code on a single node using 16 cores to avoid network traffic (all messaging uses SHM):

Windows 7 walltime = 370s.

CentOS 7 walltime = 370s.

Casual observation of the switches: Windows 7 doesn't make use of the second switch but CentOS 7 does.

Here's my team_config entry for CentOS 7:

TEAM_CONFIG='{"runner": {"fast_rate": true, "name": "loadbalance","tx_hash": ["ipv4"],"tx-balancer": {"name": "basic"}}, "ports": {"enp2s0f0": {}, "enp2s0f1": {}}}'

Would like to make use of both NICs in each team under Windows. Oh, if both of the team's NICs are connected to the same switch, then both NICs will be actively used.

Message was edited by: Art Lazanoff each switch is configured with VLAN and LAG disabled. When both connections in a ALB are connected the same switch, both NICs are active, Windows task manager shows a 20gbps data rate, and clock time for the test program improves by about 40s. The ALB's "test switch" diagnostic reports no errors. Individually both switches perform the same way. When a team is connected using NIC-1 to switch 1 and NIC-2 to switch 2, The ALB "test switch" diagnostic reports "The switch is configured with aggregated ports, but the team type does not support port aggregation. The switch is configured with VLAN tagging, but the team has no VLANs. Reconfigure the switch for link aggregation, or disable VLAN tagging on the switch. "

0 Kudos
16 Replies
Highlighted
Community Manager
102 Views

Hi Art_l,

Thank you for posting in Wired Communities. Let me further check and can you clarify below information mentioned by you:

1) Running current releases of Intel's Fortran and MPI on both CentOS7 and Windows 7 on the same nodes-are you referring there are two OS (Windows 7 and CentOS7) installed on the same node?

2) 2 xs708ev2 switches, one connection per team per switch. - You mentioned you have 4 nodes (4 PC?) and each PC has one ALB team setup but one connection per team per switch? how do you connect the node to the switch? Two nodes to switch A and two nodes to switch B?

3) Running the same code on a single node using 16 cores to avoid network traffic (all messaging uses SHM):-what is the code you refer here? Are you using any software to run the traffic?

4) When a team is connected using NIC-1 to switch 1 and NIC-2 to switch 2, The ALB "test switch" diagnostic reports "The switch is configured with aggregated ports, but the team type does not support port aggregation-

ALB should connect to the same switch, if you are connecting NIC 1 in the same team to switch 1 then NIC 2 in the same team to switch 2, this kind of setup is call SFT. You may refer to this website for reference about teaming mode:https://www.intel.com/content/www/us/en/support/articles/000005667/network-and-i-o/ethernet-products... Teaming with Intel® Advanced Network Services

Please feel free to update me. Thank you.

Regards,

 

Sharon T
0 Kudos
Highlighted
Novice
102 Views

1) Each node has both Windows 7 and CentOS 7 installed. For any test, I boot the same operating on all of the nodes. No intermixing of the two operating systems for these tests.

2) All 4 nodes have to be able to communicate with each other. Each node is connected to two switches ... 4 nodes means 1 connection/node from NIC 1 to switch 1 and 1 connection from NIC 2 to switch 2. Everything runs as expected when all 4 nodes and 8 connections are made to a single switch.

3) The example code is compiled and linked to the MPI library. The MPI library itself makes use of SHM between processes running on the same node. The MPI package includes the mpiexec command that is used to place and pin processes (code and data placed on the same core) on one or more nodes, typically one process per core. It will run one copy of the executable for each of the requested MPI proceses. For example, when running on one node only using 16 cores, there will be 16 copies of the test program running on this system. When running on 4 systems, I'll be running 4 copies of the program on each node. The test program is pre-configured to run on a choice of a power of 2 number of cores. I have another test that runs 32 MPI processes on 32 cores: 16 on node 1, 8 on node 2, and 4 each on nodes 3 and 4. In this test case the program's arrays are a constant size, and each copy has size/16 or size/32 amount of the problem to work on. The total program size for the Class D problem is ~116GB.

example command lines execution (can be run from a Command Prompt):

1) 16 MPI processes on a single node

mpiexec -n 16 ft.D.16

2) 16 MPI processes on 4 nodes, 4 processes/node, ft.D.16 is the D sized FT problem compiled for 16 MPI processes.

mpiexec -host a1.local -n 4 ft.D.16 : -host a2.local -n 4 ft.D.16 : -host a3.local -n 4 ft.D.16 : -host a4.local -n 4 ft.D.16

3) 32 MPI processes on 4 nodes, mixed number of processes/node, ft.D.32 is the D sized FT problem compiled for 32 processes.

mpiexec -host a1.local -n 16 ft.D.32 : -host a2.local -n 8 ft.D.32 : -host a3.local -n 4 ft.D.32 : -host a4.local -n 4 ft.D32

This example is one of the NASA parallel benchmarks, and I'm sure several other Intel groups already have this code. See the FT description here:

https://www.nas.nasa.gov/publications/npb.html NAS Parallel Benchmarks

I can add that I disable Window's swap file, and I disable memory pre-fetching. Linux reports 0 paging while running these tests.

4) my interpretation of Intel's ALB capability is that multiple switches are supported and that this variant does not require any specific capabilities in the switch other than the connections themselves. There are times when I use two more systems for file servers that have a single connection to the first switch. Both Windows and LInux ALB can successfully connect/transfer data to these nodes (Windows file sharing/Samba).

I will take your suggestion and look at SFT, too.

thanks,

Art

0 Kudos
Highlighted
Community Manager
102 Views

Hi Art Lazanoff,

Thank you for the detail information. ALB connects to single switch and it distribute the load transmissions based on destination address and can be used with any switch. Let me further check regarding your setup.

Regards,

Sharon T

0 Kudos
Highlighted
Novice
102 Views

Thanks, Sharon.

I configured SFT on my 4 nodes yesterday and found no effective difference from ALB. The second NIC remains inactive for the duration of a test run. MPI benefits from higher bandwidth because for sending, the data is already in the core's memory and for receiving, the core will receive the data into its own memory for use in subsequent calculations.

While I can get good MPI performance configuring each of the first NICs in one subnet and the second NICs into another subnet, this approach won't help large file transfers. These days NVME SSDs are operating on PCI-e 4x connections. I'm using one on the workstation I'm writing this on: it can read at 1225MB/s and it can write at 1747MB/s (measured by a user program). Dual 10gig-e connections are important for large file transfers.

I believe the Linux teaming drivers are POSIX compliant and might even have been written by Intel. I was hoping that the Windows version of ALB would perform comparably to the Linux version. Maybe I haven't discovered yet whether there's a better choice or a better set of configuration options.

Art

0 Kudos
Highlighted
Community Manager
102 Views

Hi Art_l,

 

 

Thank you for the update about the test done. Previously you mentioned by putting the 2 network adapters connected to the same switch, ALB status = active and the wait time improved by 40s. Can you confirm if this actually address your concern ?

 

 

Thanks,

 

Sharon T
0 Kudos
Highlighted
Novice
102 Views

Putting all the NICS from 4 teams onto one 8 port switch works but means all 8 ports are occupied. I need to connect more devices to the switch including a file server. With the Linux setup (one NIC/team connected to a switch), the switch has 4 free ports that I can use for other equipment.

0 Kudos
Highlighted
Community Manager
102 Views

Hi Art_l,

 

 

Thank you for the reply. ALB works and distributed the load on the same switch. Are you saying that if you used Linux OS, you can setup the NIC in team and connects to two switches (e.g. NIC 1 connects to switch 1, NIC2 connects to switch 2) and gets the both link active?

 

 

Thanks,

 

Shaoon T
0 Kudos
Highlighted
Novice
102 Views

yes, a Linux ALB team does work with 2 switches. Much improved clock time for the FT case along with observing the switches' LEDs confirms this for me. My tests include running both operating systems on the same desktops/NICs and switches with the cabling in place. I can readily send you my CentOS team configuration files (less than 20 lines of text). This teaming package is distributed with RedHat's and CentOS' distribution.

I got the idea that Intel's ANS also supports multiple ALB switches from this Intel ANS link:

https://www.intel.com/content/www/us/en/support/articles/000005667/network-and-i-o/ethernet-products...

A quote from this page:

"Features available using Intel ANS include:

 

•Link Aggregation

Combines multiple adapters into a single channel to provide greater bandwidth. Bandwidth increase is only available when

connecting to multiple destination addresses. ALB mode provides aggregation for transmission only while RLB, SLA, and IEEE 802.

3ad dynamic link aggregation modes provide aggregation in both directions. Link aggregation modes require switch support, while

ALB and RLB modes can be used with any switch."

There is the additional plus: non-teamed connections like my file server work with both the Linux and the Intel teams. I connect the non-teamed NICs to the same switch as the first NICs of each team.

According to Intel's current MPI product, it's possible to intermix Linux and Windows versions of the apps: say, have two Linux MPI ranks and two Windows MPI ranks running on 4 platforms. I believe the original MPI design included support of non-heterogenous environments. I haven't been trying to do this (yet). Organizations with larger numbers of nodes and a variety of apps might legitimately want to do this kind of computing.

Art

0 Kudos
Highlighted
Novice
102 Views

The Linux gnome performance monitor reports separately both inbound and outbound communications data rates. Performance in either direction using ALB on 2 NICs, each NIC connected to a different switch, reaches ~2.3GB/s, about 85% of the peak rate of what a dual 10gig-e connection might deliver.

0 Kudos
Highlighted
Community Manager
102 Views

Hi Art Lanzanoff,

 

 

With regards to the inactive state in the team, please disable the probe option in the advance property of the Team, this would enable all adapters in the team to be in active state. Intel ANS uses different method such as link status and port traffic to detect link partners. For more detail, you may refer to this link at https://www.intel.com/content/www/us/en/support/articles/000007536/network-and-i-o/ethernet-products...

 

 

Hope this information help.

 

 

Regards,

 

Sharon T
0 Kudos
Highlighted
Novice
102 Views

yes, disabling probes enables both NICs to be active, but it breaks the network. I booted each of the systems and they couldn't find each other. This morning with probes disabled I experimented by connecting all 4 teams' NICs to a single switch. In this setting everything works and maybe runs a bit faster, too. After confirming everything works OK on a single switch, I repeated the effort to connect the second NIC member of each team to the second switch. Rebooted all the systems. Again nothing works: none of the systems can ping or traceroute the other systems. Before each boot, I flushed the routes so the boot process would refresh the routes.

Thanks again,

Art

0 Kudos
Highlighted
Community Manager
102 Views

Hi Art_l,

 

 

Thank you for the update. Let me further check.

 

 

Regards,

 

Sharon T
0 Kudos
Highlighted
Novice
102 Views

Hi Sharon,

I can report that I got this ALB configuration to work this weekend. The missing piece: addtion of a SPF+ cable to connect the 2 Netgear switches in a "up-link" setup. Use of this cable enables both switches to send/receive data and no Windows 7 configuration changes are needed. Also, the team's network switch tests report good switch operation for all 4 teams. The Windows teams started to work after I plugged in this cable. No boots were necessary. Use of a RJ45 cable is not sufficient for this uplink solution to work. On one platform, the diagnostic reported switch configuration problem, and I "fixed" this by swapping its NICs's switch connections.

So, 2 NICs/team, each of the first NICs are connected to one switch and the second NICs are connected to the second switch. The first switch has 2 additonal connections for non-teamed 10gig-e desktops.

Use of the SPF+ cable with these Netgear switches disables one of the switches RJ45 ports ... the switch supports a max of 8 connections.

My LInux platforms don't appear to "notice" the SPF+ cable, and they continue to work as before.

My guess is that the SPF+ cable provided some means for ALB to "discover" the teams' connections.

Art

0 Kudos
Highlighted
Community Manager
102 Views

HI Art_Lazanoff,

 

 

Thank you for the update and sharing your resolution. Indeed good to know the setup works for you. The information would be helpful for other in case they will have the same setup.

 

 

Regards,

 

 

Sharon T

 

0 Kudos
Highlighted
Novice
102 Views

HI Sharon,

I hope you'll file a bug report for this issue. My effort is small scale compared to most organizations. Using ALB the way I've configured my teams/switches appears to be consistent with the purpose of ALB: redundant network components where failover can reliably occur combined with the ability to make use of higher bandwidth when all the components are working OK. At the least some note in the user guide would save other folks a lot of time and effort. Also, SPF+ cables are not cheap. I can make use of a short, less expensive cable, and most stores won't stock this kind of cable.

Sincerely,

art

0 Kudos
Highlighted
Community Manager
102 Views

Hi Art_l,

 

 

Thank you for bringing this matter to our attention. We will notify our engineering team. Thank you.

 

 

Regards,

 

Sharon T
0 Kudos