Software Tuning, Performance Optimization & Platform Monitoring
Discussion regarding monitoring and software tuning methodologies, Performance Monitoring Unit (PMU) of Intel microprocessors, and platform updating.

resource stalls on Sandy Bridge

Yunqi_Z_
Beginner
1,256 Views

Hi all,

I'm trying to measure the resource stalls on a SandyBridge machine. As the SDM introduces, I can break the stalls down to different causes as shown below.

RESOURCE_STALLS:ANY - 0x5301a2
RESOURCE_STALLS:LB - 0x5302a2
RESOURCE_STALLS:RS - 0x5304a2
RESOURCE_STALLS:SB - 0x5308a2
RESOURCE_STALLS:ROB - 0x5310a2
RESOURCE_STALLS:FCSW- 0x5320a2
RESOURCE_STALLS:MXCSR - 0x5340a2

To my understanding, the number of ANY is supposed to be equal to the sum of the other 6 numbers. But it seems to be wrong based on my experiments.

RESOURCE_STALLS:ANY: 397349463242
RESOURCE_STALLS:LB: 5379423
RESOURCE_STALLS:RS: 304513727418
RESOURCE_STALLS:SB: 2248871709
RESOURCE_STALLS:ROB: 18753462189
RESOURCE_STALLS:FCSW: 0
RESOURCE_STALLS:MXCSR: 0

Is that right, or there are more other causes of resource stalls that could not be counted? What are those causes?

Thank you very much!

0 Kudos
13 Replies
Patrick_F_Intel1
Employee
1,256 Views
0 Kudos
Yunqi_Z_
Beginner
1,256 Views

Hi Patrick,

I assume so, but the numbers are telling something different from that assumption. :-(

Do you know anyone or anyhow I could verify this with Intel?

Thank you very much!

0 Kudos
Patrick_F_Intel1
Employee
1,256 Views

When you say "the numbers are telling something different from that assumption"... can you be more explicit? What is the assumption you are making and how are the numbers proving/disproving the assumption?

Pat

0 Kudos
Yunqi_Z_
Beginner
1,256 Views

Hi Patrick,

Sorry for the confusion. I have the results of one experiment in the first post, where you could do the calculation and find that the .ANY is not equal to the sum of the rest stalls.

.ANY = 397349463242

REST = 5379423(.LB) + 304513727418(.RS) + 2248871709(.SB) + 18753462189(.ROB) + 0(.FCSW) + 0(.MXCSR) = 325521440739

Thank you very much for your help!

0 Kudos
Patrick_F_Intel1
Employee
1,256 Views

Right.

I would not expect the .ANY to be equal the rest.

What makes you think that .ANY would be equal to the rest?

Pat

0 Kudos
Yunqi_Z_
Beginner
1,256 Views

Hmm, I thought .ANY was the sum of the resource stalls no matter what the cause is. Problem solve. Thank you so much! :-)

0 Kudos
Patrick_F_Intel1
Employee
1,256 Views

The name is not the most clear but they already used .OTHER for another sub-event.

Usually, when an event includes all the sub-events then the umask is the bitwise OR of the sub-events. In this case if you wanted to make a .ALL event, the umask would be 0xff.

Pat

0 Kudos
Bernard
Valued Contributor I
1,256 Views

Hi Yunqi,

jumping late to interesting discussion:) Below is detailed explanation of the event RESOURCE_STALLS.ANY meaning.

The number of instructions in the pipeline waiting for execution or retirement reached the limit the processor can handle.
The number of load or store instructions in the pipeline waiting for retirement reached the limit the processor can handle.
There is an instruction in the pipe that can be executed only when all previous stores complete and their data is committed in the caches or memory. Fore example, SFENCE and MFENCE instructions require this behavior.
The pipeline recovers from a mispredicted branched that was executed.
The floating-point unit ( FPU ) control word is written.

As you can this event is not a sum of the events measured by you in your post.

0 Kudos
Yunqi_Z_
Beginner
1,256 Views

Cool! Thank you very much iliyapolak! :-)

0 Kudos
Bernard
Valued Contributor I
1,256 Views

You are always welcome :)

0 Kudos
Brian_V_
Beginner
1,256 Views

Is there any way to convert a RESOURCE_STALL.SB count back into how much performance impact I am getting?  I am doing a large FFT operation with FFTW and MKL and I have well vectorized floating-point, but lots of poorly vectorized load/store operations since complex numbers are interleaved.  I suspect that is exhausting the store buffer, but how do I quantify this effect?

Brian

 

0 Kudos
McCalpinJohn
Honored Contributor III
1,256 Views

The cost in cycles can be as large as the count from the RESOURCE_STALLS.SB event, or as small as zero, depending on the extent to which the store buffer full stalls overlap with other stalls and/or other work.  

If this is not bad enough, it is even more confusing in parallel workloads because of the additional 0%-100% range of possible overlap of stalls across threads.   So in the parallel case, one increment of the RESOURCE_STALLS.SB event can have a net execution time cost of anywhere between zero cycles and 1 cycle times the number of cores used in the job.

0 Kudos
Travis_D_
New Contributor II
1,256 Views

Yunqi Z. wrote:

Hi Patrick,

Sorry for the confusion. I have the results of one experiment in the first post, where you could do the calculation and find that the .ANY is not equal to the sum of the rest stalls.

.ANY = 397349463242

REST = 5379423(.LB) + 304513727418(.RS) + 2248871709(.SB) + 18753462189(.ROB) + 0(.FCSW) + 0(.MXCSR) = 325521440739

Thank you very much for your help!

In this case, the umask does not operate the same as other events: each umask bit does not represent independent, mutually exclusive events. All the bits except 0x01 (the .ANY event) operate like that, but the ANY bit includes those events plus some other stall causes which are not covered by the remaining bits. So I believe that umask=0x01 will always be greater than or equal to umask=0xFE.

The likely cause is that the full list of stall events are split across at least two performance counter events: RESOURCE_STALLS and RESOURCE_STALLS2. If a stall cause falls into the STALLS2 categories, it won't show up in any of the specific stall causes for RESOURCE_STALLS, but it will show up in the ANY bit, so you can could all stalls regardless of cause with only event (rather than trying to add STALLS and STALLS2.

0 Kudos
Reply