inter-core communication

[forums_mp]

I'm close to procuring a machine with Dual 2.26GHz six-core intel Xeon L5640 'Westmere' Processors. Could someone point me in the right direction on where I could obtain source code (preferred)and/or documentation on inter-core communication?

I suspect each core communicates across a shared memory bus and ideally I'd like to get performance numbers on communication between the cores.

An aside: So given 2 processors with 6 cores/processor and 48 Gigs of memory.If memory serves the spec sheet claims 6 channels with 8 Gigs per channel. It's unclear to me how the processors interact with the channels and I'll assume the both processors can access all 48 Gigs. Long story short, if anyone's aware of documentation that'll show a block diagram of the processor module I'd appreciate it

Thanks in advance.

[Ilnar]

as far as i know, in case NUMA is enabled, each processor has it's own 6 channels and local memory banks

[TimP]

I'll paraphrase, as I don't think it's possible to make a quick guide here which could save you hours of reading the docs but help you find all the details. Sites such as anandtech tend to give you an accurate and more accessible presentation than you could find elsewhere.
Each 6-core CPU owns 3 memory channels to which it has a direct QPI connection. A CPU can access the memory channels of the other CPU indirectly, via the QPI connection between CPUs, giving transparent access to all of the RAM on the motherboard.
8 GB/channel presumably refers to 2 RAM slots with 4GB DDR3.
Most platforms are set up with BIOS NUMA option disabled by default, which means that cache lines are interleaved, alternating between local and remote channels. There are significant performance implications. AMD Opteron introduced a NUMA scheme several years before it was introduced in Nehalem.
So far, the Westmere platforms don't differ from the predecessor Nehalem.
You seem to be hoping to over-simplify the organization, and I'm not competent nor is there space here to describe details fully, so you may not be interested in the following:
On the 6 core CPUs, cores 0 and 1, and 2 and 3, share paths to L1 cache, so there are the same number of paths (4) as the precessor Nehalem. The 6 core CPUs introduced a ring organization for communicating L1 cache among the cores. The inclusivity/exclusivity of the 3 cache levels remains similar to Nehalem.

[forums_mp]

|| The 6 core CPUs introduced a ring organization for communicating L1 cache among the cores.

Apologies in advance if my questions appears trivial. You clearly seem to be on top of this stuff.

So if I opt to communicate with core 3 from core 0, given the 'ring organization', I'll communcation through
core 1 and core 2 prior to reaching core 3?

Do you know if intel has any software - I suspect they should if they've done benchmarks - online that demonstrates communication between cores? I suspect communication is predicated upon 'shared memory' - if you will. ie. each processor has a specific area within memory with which to read from/write to?

[TimP]

Again, I'll refer elsewhere for explanation of ring cache .

[forums_mp]

Got it. Last (I believe) question(s). I think the answer is yes, nontheless, I have two RJ45 (ethernet) connections. Can any 'core' access the connections? IOW can I read from/write to the ethernet ports from any core.

Trying to determine a way to multicast ethernet data to 6 cores, which ideally could be resolved by having a decidated core that'll receive the data and write the data to offsets in memory for the 6 cores. Similarily this dedicated core will read data from offsets and write data x6 out.

If my questions appear silly, my apologies....just trying to determine how best to use/exploit the resources/architecture.

Thanks in advance.

[TimP]

You may find the irqbalance docs interesting, as this is the usual way in current linux of optimizing distribution of processing of interrupts (e.g. from a NIC). An interesting side effect of the presence of irqbalance is the possibility of using 'irqbalance -debug' to display the relationship between BIOS core numbering and cache association.