Headless GPU Server Hangs, How to Debug?

[Bob_Kirnum]

Running a new test scenario on our GPU system.  It is rather simple, just make a few calls in and start a few transcoding sessions.  Everything starts fine (CPU and GPU load is under 50%) as before, however, if I leave the sessions up for some period of time (a few minutes) the system hangs.  The server is located in a rack in a lab.  I connect to it via. SSH and all terminals hang.  There is no obvious core dump.  The terminal in the lab connected via. the KVM switch shows nothing.  In fact, even during boot up there is little displayed.  Any suggestions for how to debug this issue?

Thanks - Bob

lspci -nn -s 00:02.0
00:02.0 Display controller [0380]: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller [8086:041a] (rev 06)

processor       : 1

vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel(R) Xeon(R) CPU E3-1285 v3 @ 3.60GHz

Running SDK 2016

[Surbhi_M_Intel]

Hi Bob - You can run the tracer logs along with your experiments to capture what's happening. Media SDK tracer is available at the installed directory, check the readme file available at same location to configure tracer and run along with your tests. 

[Bob_Kirnum]

Is this apitrace?  If so, I am having a hard time getting this to work.  Is there a usage example for the CLI invocation?

[Surbhi_M_Intel]

Hi Bob - There is a readme in the file, which has the process to configure the tool. Check this threads for reference to get the tool configured - https://software.intel.com/en-us/forums/intel-media-sdk/topic/596058 
Once the tool is configured, it should automatically save the file in the location(set while installing) . Can you explain the issue you are seeing in installing and using the tool?

-Surbhi

[Bob_Kirnum]

Ah, ok.  I thought that tracer was Windows only.  I'll follow the instructions and give it a try and report back.

Thanks - Bob