Intel Network Latency

[mariov652]

Hello,

This is my first post to the group.

I am not really a software developer, more a learning Network Architect recently working in a HPC-type field. I have become very interested in this field recently and hope to give back to the community as I learn more myself.

I hope someone will be able to provide more info or link to some sites regarding my questions. If you don't have any answers, maybe some suggestions on how I could find the answer out myself...


We have found the Intel network adaptors to be one of the best with regards to low latency / high throughput. We are looking to optimize settings to provide consistantly best response times / lowest network card latency and have run some tests with two systems connected via a cross-over cable between them.


One question which we don't have an answer for yet is:
We notice a difference (allbeit small) in response times between pings when we send many pings at once as opposed to sending, say one ping every one or two seconds. For example, for 64-byte packets the response time differences between pinging every second ranges from ~63us to ~76us; however when we send pings every 100us, the response time differences between pings ranges from ~61s to ~65us.

Is it possible to minimise this 'jitter' or difference between response times between each ping? In other words is it possible to consistantly achieve a 'lowest' response time?


Any guidance is appreciated.


Thanks,
Mario

[Andres_M_Intel4]

Mario,

There are several things you can do to improve the Ethernet latency, for instance the customization of driver options and the removal of general purpose services which usually add overhead to packet processing without providing benefits on an HPC environment.

I have relevant experience on Linux-based OSes, but before sending specific details on this it would be useful to know first your actual environment.

Thanks!

-- Andres

[mariov652]

Quoting - Andres More (Intel)

Mario,

There are several things you can do to improve the Ethernet latency, for instance the customization of driver options and the removal of general purpose services which usually add overhead to packet processing without providing benefits on an HPC environment.

I have relevant experience on Linux-based OSes, but before sending specific details on this it would be useful to know first your actual environment.

Thanks!

-- Andres

Hi Andres,

First off, thanks for your reply.


For the moment, we are running a Windows environment. We are trying to minimize the time spent on the various network interfaces.

network-wise, the systems receive multicast data streams, and intermittently send TCP packets in the other direction (with the usual TCP SYN, SYN ACK following).

Customizations done so far are to minimize all unneeded services on the systems, connectivity upgraded to 1Gbps from 100Mbps, driver settings: Interrupt Moderation disabled, Adaptive inter-frame space disabled, flow control disabled. Also the nagle algorithm has been disabled to avoid queuing/buffering of data.

Using internal measurements, we've measured latency on the cards to be around ~30us one-way for directed TCP traffic (similar results for UDP). This seems to be in-line with other results I've found.

To possibly improve It seems we need to start looking at the TCP/IP parameters (similar to disabling nagle algorithm) to optimize the connections. Connectivity is all LAN based, so no WAN connections.

We have noticed that sending TCP packets every 500us or 1second produces greater latency results (and increased jitter) than when sending single TCP packets every 50us or 100us. I think this may be due to TCP/IP slow-start mechanism. Any ideas on how to improve this?


Thanks,
Mario

[Andres_M_Intel4]

Mario,

Nice work, it seems that you already covered everything I can think off right now.
You shouldn't be that far from the best configuration.

Perhapsyou may find useful tocontact the Linux e1000/e1000e/igb drivers development team at http://e1000.sf.net, they might point you to other sources of information.

-- Andres

PS, just in case you miss it 'Interrupt Moderation Using Intel GbE Controllers' is a great how-to on interrupt moderation customization. (download.intel.com/design/network/applnots/ap450.pdf)

[mariov652]

Quoting - Andres More (Intel)

Mario,

Nice work, it seems that you already covered everything I can think off right now.
You shouldn't be that far from the best configuration.

Perhapsyou may find useful tocontact the Linux e1000/e1000e/igb drivers development team at http://e1000.sf.net, they might point you to other sources of information.

-- Andres

PS, just in case you miss it 'Interrupt Moderation Using Intel GbE Controllers' is a great how-to on interrupt moderation customization. (download.intel.com/design/network/applnots/ap450.pdf)

Thanks Andres,

In fact, I had already found that document previously, thanks for the link though.

You're right, a lot of the advice I get is to test in Linux. I do hack with Linux too, but so far I haven't been able to get the same timescales as with Windows for the nic latency tests - I'll try the sourceforge community though.


Cheers,
Mario