resource stalls on Sandy Bridge

[Yunqi_Z_]

Hi all,

I'm trying to measure the resource stalls on a SandyBridge machine. As the SDM introduces, I can break the stalls down to different causes as shown below.

RESOURCE_STALLS:ANY - 0x5301a2
RESOURCE_STALLS:LB - 0x5302a2
RESOURCE_STALLS:RS - 0x5304a2
RESOURCE_STALLS:SB - 0x5308a2
RESOURCE_STALLS:ROB - 0x5310a2
RESOURCE_STALLS:FCSW- 0x5320a2
RESOURCE_STALLS:MXCSR - 0x5340a2

To my understanding, the number of ANY is supposed to be equal to the sum of the other 6 numbers. But it seems to be wrong based on my experiments.

RESOURCE_STALLS:ANY: 397349463242
RESOURCE_STALLS:LB: 5379423
RESOURCE_STALLS:RS: 304513727418
RESOURCE_STALLS:SB: 2248871709
RESOURCE_STALLS:ROB: 18753462189
RESOURCE_STALLS:FCSW: 0
RESOURCE_STALLS:MXCSR: 0

Is that right, or there are more other causes of resource stalls that could not be counted? What are those causes?

Thank you very much!

[Patrick_F_Intel1]

Hello Yunqi,

From http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/amplifierxe/win/ug_docs/reference/pmn/events/resource_stalls.html, it doesn't look like the .ANY event counts the other events.

Pat

[Yunqi_Z_]

Hi Patrick,

I assume so, but the numbers are telling something different from that assumption. :-(

Do you know anyone or anyhow I could verify this with Intel?

Thank you very much!

[Patrick_F_Intel1]

When you say "the numbers are telling something different from that assumption"... can you be more explicit? What is the assumption you are making and how are the numbers proving/disproving the assumption?

Pat

[Yunqi_Z_]

Hi Patrick,

Sorry for the confusion. I have the results of one experiment in the first post, where you could do the calculation and find that the .ANY is not equal to the sum of the rest stalls.

.ANY = 397349463242

REST = 5379423(.LB) + 304513727418(.RS) + 2248871709(.SB) + 18753462189(.ROB) + 0(.FCSW) + 0(.MXCSR) = 325521440739

Thank you very much for your help!

[Patrick_F_Intel1]

Right.

I would not expect the .ANY to be equal the rest.

What makes you think that .ANY would be equal to the rest?

Pat

[Yunqi_Z_]

Hmm, I thought .ANY was the sum of the resource stalls no matter what the cause is. Problem solve. Thank you so much! :-)

[Patrick_F_Intel1]

The name is not the most clear but they already used .OTHER for another sub-event.

Usually, when an event includes all the sub-events then the umask is the bitwise OR of the sub-events. In this case if you wanted to make a .ALL event, the umask would be 0xff.

Pat

[Bernard]

Hi Yunqi,

jumping late to interesting discussion:) Below is detailed explanation of the event RESOURCE_STALLS.ANY meaning.

The number of instructions in the pipeline waiting for execution or retirement reached the limit the processor can handle.
The number of load or store instructions in the pipeline waiting for retirement reached the limit the processor can handle.
There is an instruction in the pipe that can be executed only when all previous stores complete and their data is committed in the caches or memory. Fore example, SFENCE and MFENCE instructions require this behavior.
The pipeline recovers from a mispredicted branched that was executed.
The floating-point unit ( FPU ) control word is written.

As you can this event is not a sum of the events measured by you in your post.

[Yunqi_Z_]

Cool! Thank you very much iliyapolak! :-)

[Bernard]

You are always welcome :)

[Brian_V_]

Is there any way to convert a RESOURCE_STALL.SB count back into how much performance impact I am getting?  I am doing a large FFT operation with FFTW and MKL and I have well vectorized floating-point, but lots of poorly vectorized load/store operations since complex numbers are interleaved.  I suspect that is exhausting the store buffer, but how do I quantify this effect?

Brian

[McCalpinJohn]

The cost in cycles can be as large as the count from the RESOURCE_STALLS.SB event, or as small as zero, depending on the extent to which the store buffer full stalls overlap with other stalls and/or other work.  

If this is not bad enough, it is even more confusing in parallel workloads because of the additional 0%-100% range of possible overlap of stalls across threads.   So in the parallel case, one increment of the RESOURCE_STALLS.SB event can have a net execution time cost of anywhere between zero cycles and 1 cycle times the number of cores used in the job.

[Travis_D_]

Yunqi Z. wrote:

Hi Patrick,

Sorry for the confusion. I have the results of one experiment in the first post, where you could do the calculation and find that the .ANY is not equal to the sum of the rest stalls.

.ANY = 397349463242

REST = 5379423(.LB) + 304513727418(.RS) + 2248871709(.SB) + 18753462189(.ROB) + 0(.FCSW) + 0(.MXCSR) = 325521440739

Thank you very much for your help!

In this case, the umask does not operate the same as other events: each umask bit does not represent independent, mutually exclusive events. All the bits except 0x01 (the .ANY event) operate like that, but the ANY bit includes those events plus some other stall causes which are not covered by the remaining bits. So I believe that umask=0x01 will always be greater than or equal to umask=0xFE.

The likely cause is that the full list of stall events are split across at least two performance counter events: RESOURCE_STALLS and RESOURCE_STALLS2. If a stall cause falls into the STALLS2 categories, it won't show up in any of the specific stall causes for RESOURCE_STALLS, but it will show up in the ANY bit, so you can could all stalls regardless of cause with only event (rather than trying to add STALLS and STALLS2.