https://community.intel.com/t5/Analyzers/Suggestions-Needed-for-Finding-a-Locking-Waiting-Problem-using/m-p/762987#M3 I have an application running on a Dual Xeon i9 with a total of 8 2-way HT cores for 16 concurrent threads of execution.<BR /><BR />My application is very complex, with on the order of 100 various types of concurrent threads being scheduled by the Linux scheduler.<BR /><BR />When run at full speed, it runs well for 20-30 minutes and then starts to degrade. It look as if there is a lock or a series of queued events which need to be cleared and are causing the machine to be mostly quiescent for 1-2 seconds. This happens periodically every 4-6 seconds.<BR /><BR />I tried a minimal locks-and-waits sampling, but even with the sampling paused, there was enough of a slowdown in the throughput of the system that it never got into the stall for 1-2 seconds state.<BR /><BR />When I run HW-based sampling, the stall behavior shows up. But, I do not know how to find out what is stalling from the HW-based sampling, since when it is stalled, there are no hardware events to sample. I could make some guesses based on what is not running (showing up in samples) during the stalls, but it is essentially everything, so this is problematic.<BR /><BR />Do you have any suggestions?<BR /><BR />One possibility is to use spin locks, which would show up as execution<BR />at the PC of the spin code, but this is likely to distort the behavior<BR />similar to what the locks-and-waits sampling did.<BR /><BR />Thank you for any suggestions,<BR /><BR />Todd<BR /><BR /><BR /><BR />Todd Bezenek<BR /> Computer Architect / Performance Analyst<BR /> bezenek@gmail.com Thu, 14 Apr 2011 22:20:01 GMT todd-bezenek 2011-04-14T22:20:01Z https://community.intel.com/t5/Analyzers/Suggestions-Needed-for-Finding-a-Locking-Waiting-Problem-using/m-p/762987#M3 I have an application running on a Dual Xeon i9 with a total of 8 2-way HT cores for 16 concurrent threads of execution.<BR /><BR />My application is very complex, with on the order of 100 various types of concurrent threads being scheduled by the Linux scheduler.<BR /><BR />When run at full speed, it runs well for 20-30 minutes and then starts to degrade. It look as if there is a lock or a series of queued events which need to be cleared and are causing the machine to be mostly quiescent for 1-2 seconds. This happens periodically every 4-6 seconds.<BR /><BR />I tried a minimal locks-and-waits sampling, but even with the sampling paused, there was enough of a slowdown in the throughput of the system that it never got into the stall for 1-2 seconds state.<BR /><BR />When I run HW-based sampling, the stall behavior shows up. But, I do not know how to find out what is stalling from the HW-based sampling, since when it is stalled, there are no hardware events to sample. I could make some guesses based on what is not running (showing up in samples) during the stalls, but it is essentially everything, so this is problematic.<BR /><BR />Do you have any suggestions?<BR /><BR />One possibility is to use spin locks, which would show up as execution<BR />at the PC of the spin code, but this is likely to distort the behavior<BR />similar to what the locks-and-waits sampling did.<BR /><BR />Thank you for any suggestions,<BR /><BR />Todd<BR /><BR /><BR /><BR />Todd Bezenek<BR /> Computer Architect / Performance Analyst<BR /> bezenek@gmail.com Thu, 14 Apr 2011 22:20:01 GMT https://community.intel.com/t5/Analyzers/Suggestions-Needed-for-Finding-a-Locking-Waiting-Problem-using/m-p/762987#M3 todd-bezenek 2011-04-14T22:20:01Z https://community.intel.com/t5/Analyzers/Suggestions-Needed-for-Finding-a-Locking-Waiting-Problem-using/m-p/762988#M4 Hi Todd,<BR /><BR />Assume that you are working on latest Update 2. <BR /><BR />Since you complicated application consumes system resource high, I suggest to use command line to profile. I want to know if the stall was caused by the tool, or application-self. So you can launch application manually and use command line to profile whole system (e.g. amplxe-cl -collect lightweight-hotspots -analyze-system -duration xxx. Thus, all applications in system will be profiled - not only for your app only)<BR /><BR />For LocksandWaits analysis, please try below: (avoid big overhead when monitoring in spin-locks)<BR />amplxe-cl --collect locksandwaits -knob collect-spin-data=false-knob collect-signals=true -follow-child -- your_app your_args<BR /><BR />Hope it helps.<BR /><BR />Regards, Peter<BR /> Fri, 15 Apr 2011 01:32:49 GMT https://community.intel.com/t5/Analyzers/Suggestions-Needed-for-Finding-a-Locking-Waiting-Problem-using/m-p/762988#M4 Peter_W_Intel 2011-04-15T01:32:49Z