"nonexistent" reset events sync objects

[apour]

We are unable to utilize 12-core Xeon machine over ~65% with our .NET app. I'm trying to use Amplifier XE to find where the problem is, but I'm quite confused by what the analysis show.

The analysis summary (hotspots by concurrency or lock & waits) states that signification amount of CPU time is spent in synchronization or threading overhead. Among top waiting objects are Auto/Manual Reset Events. When I analyze waiting threads I often see "Wait / Start: ../ Duration:.. / Synch Object: Auto Reset Event / Source: " - but in the source code (C#) there are no reset events at all. I also quickly checked the generated bytecode. And there are also lots of Transitions around these nonexistent reset events.

How should I read such analysis results? Thank you a lot in advance.

[Peter_W_Intel]

What was analysis type you used? Hotspots? If so,you cannot select view for "Locks & Waits".

If youused "Locks & Waits" -
1. As default, summary report has CPU time spent in Sync Objects,which could be in your code (C#?)orlibararies (MS libraries? linked with your code). Probably some eventsdoesn't exist in your code, but used in libraries.

2. If you don't care of such event (they spentlittle CPU time), you can change view to Hotspots or Concurrency that the result will focus on hot functions.

Regards, Peter

[apour]

Thank you for your attention.

I tried both Concurrency and Lock & Waits analysis, so I could select view for "Locks & Waits".

Wait time is significant, I think:

---
Elapsed Time: 395.235s
Total Thread Count: 293
Wait Time: 10061.902s
Spin Time: 60.740s
Wait Count: 297,744
CPU Time: 635.523s
Paused Time: 324.350s
---


I tried to track down 2 of these "nonexistent" reset events.


1) In our app code, we have

Array.Copy(path, pathInt, path.Length);

This line is reported in the analysis as one place where thread transitions and waitings occur with sync object "Auto Reset Event".
This is a call to

[ReliabilityContract(Consistency.MayCorruptInstance, Cer.MayFail), SecuritySafeCritical]
public static void Copy(Array sourceArray, Array destinationArray, int length)
{
    if (sourceArray == null)
    {
        throw new ArgumentNullException("sourceArray");
    }
    if (destinationArray == null)
    {
        throw new ArgumentNullException("destinationArray");
    }
    Copy(sourceArray, sourceArray.GetLowerBound(0), destinationArray, destinationArray.GetLowerBound(0), length, false);
}

which continues to

[MethodImpl(MethodImplOptions.InternalCall), SecurityCritical, ReliabilityContract(Consistency.MayCorruptInstance, Cer.MayFail)]
internal static extern void Copy(Array sourceArray, int sourceIndex, Array destinationArray, int destinationIndex, int length, bool reliable); 

OK - I did not track this into CLR so maybe there is something inside CLR which I cannot see. I can try to avoid this call (eg. copy arrys in loop), but

2) another auto reset event that contributes to waits and wait time is supposed to be in our class ServerImpl.cs line 347:

public void ProcessInputMessage() {
DateTime start = DateTime.UtcNow;
// ensure single view on once loaded entities
try {
ProcessInputMessageInternal(transportMessage); // THIS is line 347
} catch (Exception e) {
...

But ProcessInputMessageInternal is just another method in the same class! I'm quite lost.


Thank you,
Ales

[Peter_W_Intel]

Some of "invisible" sync objects could be in wait function like as SendMessage, WaitForSingleObject, EnterCriticalSection, etc. And File I/O operation includes sync-obj, also socket I/O has wait objects.

I believe that ProcessInputMessageInternal has wait-time.

Regards, Peter