some question about QPI

Yinchao_Z_ — Thu, 31 Mar 2016 02:06:37 GMT

hi,

I use pcm tool to monitor qpi performance, and now i am confused with some question.

1. with qpi frequency set as 9.6GT/s ,what's the max bandwidth it can be ? I mean from point to point like from socket1 to socket0?

2. I see the source code to find out how the pcm calculate the qpi Utilization , it shows that : max_bytes = (double)(double(max_speed) * double(getInvariantTSC(before, after) / double(m->getNumCores())) / double(m->getNominalFrequency()));

can you tell me what is the factor meaning ?

3. In my test result , in 4cpu mechine , the outgoing QPI traffic(in a second ) can up to 23GB with 94% utilization, but in another mechine with 16socket sometimes it can be 24G with 77% utilization ,the denominator seems change all the time ?

In my view ,the max_bytes in a second should be the max_speed 19.2 GB/s ,but in my test it was not.

really confused. help !

thank you !

@John McCalpin

Yinchao_Z_ — Tue, 05 Apr 2016 01:25:14 GMT

@John McCalpin

can you help me ?

thank you very much !

QPI links have an "effective

McCalpinJohn — Tue, 05 Apr 2016 19:16:17 GMT

QPI links have an "effective width" for data of 16 bits in each direction, so the peak data bandwidth of a link running at 9.6 GT/s is 19.2 GB/s per direction.

Socket-to-socket connections in 4 socket-systems are a single link wide, while many/most 2-socket systems use two links to connect the two sockets.

This peak bandwidth number is not attainable for a large number of reasons. Part of the bandwidth is used for requests (loads or snoops) and part is used for responses without data (mostly snoop responses). Traffic going in both directions will see contention between the requests and data responses.

In many systems it is not possible to get close to the maximum bandwidth for some transaction types, suggesting that there are inadequate buffers for some transaction types. I have only tested this extensively on 2-socket systems -- the results may be completely different on larger systems, since the allocation of buffers is likely to be different. These limitations vary by processor generation and (more recently) by the "snoop mode" of the system.

Xeon E5 (v1, Sandy Bridge) was only able to drive ~66% of peak QPI bandwidth for reads, and much less than that for combinations of reads with either writebacks or streaming stores.
Xeon E5 v2 (Ivy Bridge) introduced the "Home Agent" snoop mode, which allows much better performance -- up to 80% of peak for reads, and improvements of up to 100% for reads+writebacks and improvements ~70% for reads+streaming_stores.
Xeon E5 v3 (Haswell, using "Home Agent" snoop mode) retains these improvements and adds up to 25% improvement in workloads that are limited by read+streaming_store performance.

I have tested performance of 4-socket systems, but only with Xeon E5 v1 (Sandy Bridge) processors, and the performance was very low -- about 1/2 of the percentage of peak QPI bandwidth that I saw on the 2s systems (and the peak is only 1/2 because only one link is used between each pair of chips). I don't know if this can be attributed to coherence traffic or if there is another cause. I have not tested any 4s systems newer than Sandy Bridge (Xeon E5-4650), so I don't know how the performance has changed.....

topic some question about QPI in Software Tuning, Performance Optimization & Platform Monitoring

some question about QPI

@John McCalpin

QPI links have an "effective