LACP Setup with Intel Multi-Vendor Teaming: Problem?

[bl2]

Hey all,

I've been acquiring some parts over the past 6 months to a year and have always wanted to setup link aggregation so just started playing with it the other night. My components are as follows:

http://www.thecus.com/product.php?PROD_ID=42 Thecus N4200Eco NAS (4 x 3 TB WD Reds in RAID 10)

http://countries.netgear.com/Products/Switches/SmartSwitches/GS724T.aspx Netgear Prosafe GS724Tv3 switch

PC with an http://www.asus.com/Motherboards/Intel_Socket_1155/P8Z68_DELUXE/ Asus P8Z68 Deluxe Motherboard (1 x Intel 82579 GbE, 1 x Realtek 8111E GbE), SSD primary drive (C:), 7200 RPM WD Black secondary (D:)

Now in doing a bunch of reading, I know what the limitations of 802.3ad are (or what I think they should be). You can't get a 2 Gb pipe for a single stream but 2 x 1 Gb for 2 or more streams should be doable. One thing I can't confirm is whether or not these streams have to be opposite to each other or if they can be in the same direction, and this is pretty much my main reason for this post.

Here are a couple of photos of my NAS/switch config pages:

https://www.dropbox.com/s/ea83r48qoi6bb1v/LAG%20config.jpg https://www.dropbox.com/s/ea83r48qoi6bb1v/LAG%20config.jpg

https://www.dropbox.com/s/2vrv7vwekfnpeoy/NAS%20config.jpg https://www.dropbox.com/s/2vrv7vwekfnpeoy/NAS%20config.jpg

My desktop was not on when I took that screenshot, hence why it says link down under LAG state. I installed version 17.4 of the Intel Network Adapter Driver for Windows 7 and configured the two NIC's as a team. They're both showing as Active in the settings, with a 2 Gb link. Originally, I had the GS724T configured with the LAG types as Static instead of LACP, and one of the adapters would show as Standby, but putting them to LACP (as you're supposed to) fixed that and they both became Active. Don't have any pictures right now but I can post some later if someone wants.

So I did some file transfer tests, nothing overly scientific but I think it demonstrated my point. The N4200Eco is a pretty powerful little machine with those drives in RAID 10 but I doubt it could actually service both ports at their full capacity. I think I was careful not to bottleneck any hardware in the tests though. My laptop also has GbE so that helped:

Test 1:

Large file copy from laptop->desktop(D:), desktop(C:)->NAS

laptop->desktop: ~70 MB/s

desktop->NAS: ~85 MB/s

Test 2:

Large file copy from NAS->laptop, NAS->desktop

NAS->laptop: ~60 MB/s

NAS->desktop: ~75 MB/s

Test 3:

Large file copy from laptop->desktop(C:), NAS->desktop(D:)

laptop->desktop: ~60 MB/s

NAS->desktop: ~60 MB/s

In Test 1, Task Manager was showing about 75% network utilization. The MB/s values I have above may have been average values, but it was definitely using both gigabit links (one in, one out) for these transfers.

In Test 2, because I was using diverse computers, it was hard to check network utilization through Task Manager. But I checked the packet numbers of the switch and could see packets coming out on both links, so it looks like the NAS was sending out on both GbE's at a decent rate to the two PC's. I discovered just now that the NAS has a nice page on the web interface showing real time traffic over each of the LAN ports, so I'll have to check that out next time.

In Test 3, Task Manager on my desktop showed the network utilization not going above 50%, highest I think it went was about 47%. The switch's web interface showed me that packets were only coming through on one of the desktop's links.

Based on the above short tests, I think my NAS is working well, with both links transmitting in either direction at any time. I think my PC is only allowing one link to send and one to receive, which I don't think should be right. There definitely shouldn't have been any hardware bottlenecks in Test 3. Is this a limitation of 802.3ad, or Intel ANS, or Multi-Vendor Teaming, or do my settings need to be tweaked? I know there's a lot of config yet that can be done with the NIC's on the PC.

If anyone has any ideas let me know, but if I find anything else out I'll report back here.

Cheers,

Blake

[Mark_H_Intel]

Hi Blake,

When you have a file transfer between point a and point b, the traffic is typically only sent on one of the ports in the LAG. Check out the presentation at http://www.ieee802.org/3/hssg/public/apr07/frazier_01_0407.pdf http://www.ieee802.org/3/hssg/public/apr07/frazier_01_0407.pdf.

Some quotes from the presentaiton that are applicable to what is happening in the tests described:

• All packets associated with a given "conversation" are transmitted on the same link to prevent mis-ordering (slide 3)
• [LAG] Does not increase the bandwidth for a single conversation (slide 7)
• [LAG] Achieves high utilization only when carrying multiple simultaneous conversations (slide 7)
• LAG is good, but it's not as good as a fatter pipe

The file transfer only creates a single TCP conversation, so you will not get a high utilization of your network bandwidth. At least you have fault tolerance. If either port in the team went down, the file transfer can still complete using the other link.

Mark H

[bl2]

Hey Mark,

Thank you for the quick reply to my question, much appreciated. I haven't had the chance to do any more testing but may do some tonight.

I did check out that presentation; came across it while doing research for my original post. Everything you're saying makes sense, but what defines a conversation? If I transfer one file from my NAS to my desktop using SAMBA, then separately start transferring another file (NAS -> desktop), is that still the same "conversation"? I was able to prove that I can at least transfer and receive the full 2 Gbps (1 Gbps down, 1 Gbps up) concurrently, but is there any way to get 2 conversations in one direction?

In my Test 3 above, the files were coming from 2 separate devices to my desktop. Shouldn't that be two conversations? They're from separate devices after all...

I'll do more testing when I can and update this thread accordingly. Thanks again for the info!

Cheers,

Blake

[Mark_H_Intel]

On the team's outgoing or transmit traffic, a hash table is used to assign flows destined for a particular end client to one of the ports in the team. The hash algorithm uses a Layer 3 hash index derived from the last octet of the end client's IP address. The transmit traffic from the server is then transmitted through the member in the team corresponding to that index in the hash table.

New data flows from the server are assigned to the least loaded member in the team, and this member is placed in the table corresponding to the client hash index.

Because the hash index is created based on IP addresses, in this case all transfers to the same IP address are one conversation. So a separate TCP conversation (different port) won't transmit out a different physical port. If your desktop was transferring files to and from two hosts with different IP addresses, then the transmitted packets to host B, would be going out a different physical port than the packets being transmitted to host A. (Assuming that the last octet of the IP address on host A and host B are different.)

For the receive side, the switch controls the traffic returning to the ANS intermediate driver used for teaming, and I don't know how how the switch decides on the port to use.

The behavior of the ANS driver used for teaming is described in the paper at http://www.intel.com/network/connectivity/resources/doc_library/white_papers/254031.pdf http://www.intel.com/network/connectivity/resources/doc_library/white_papers/254031.pdf. The paper is old, so the OS information and some of the tables are outdated. However, the basic operation is still the same as described in the paper.

Mark H