Pinplay's Pinpoints fails on some SPEC2006

[Biswas__Ananda]

Hi,
I'm using Pinplay tool v3.11-97998 pinpoints script to determine representative regions for Spec 2006 benchmarks for the reference input sets. 21 SPEC06 benchmarks ran fine on pinpoints.py, but the other 8 (povray, calculix, hmmer, libquantum, h264ref, tonto, wrf, sphinx3) failed. I cannot determine the reason for their failure. I tested running h264ref  (i) with reference input set, without pinpoints and it did not throw any error (ii) with the test input on pinpoints, and that too completed without errors. 
Any suggestions/tips will be very helpful. Thank you!
Below are some more details:

1.  Command for h264ref:
   • ./<binary_name> -d foreman_ref_encoder_main.cfg > foreman_ref_main_encodelog.out 2>> foreman_ref_main_encodelog.err
2. Std out from running the script:

   • ***  Generating whole program pinballs [log_whole]  ***    May 22, 2020 09:35:38
     logger.py --log_file whole_program.ref/h264ref.ref  --global_file global.dat.18410 --cfg config.cfg  "/home/user/specbenchtest/benchspec/CPU2006/464.h264ref/run/run_base_ref_gcc43-64bit.0000/h264ref -d foreman_ref_encoder_main.cfg > foreman_ref_main_encodelog.out 2>> foreman_ref_main_encodelog.err"

     log.py --compressed=bzip2  --log_file whole_program.ref/h264ref.ref --log_options "-log:syminfo -log:pid "  --global_file global.dat.31270 --cfg config.cfg  -- "/home/user/specbenchtest/benchspec/CPU2006/464.h264ref/run/run_base_ref_gcc43-64bit.0000/h264ref -d foreman_ref_encoder_main.cfg > foreman_ref_main_encodelog.out 2>> foreman_ref_main_encodelog.err"

     /home/user/pin/pin -t /home/user/pin/extras/pinplay/bin/intel64/pinplay-driver.so -log -xyzzy  -log:basename whole_program.ref/h264ref.ref  -log:mt 0 -log:compressed bzip2 -log:syminfo -log:pid  -- /home/user/specbenchtest/benchspec/CPU2006/464.h264ref/run/run_base_ref_gcc43-64bit.0000/h264ref -d foreman_ref_encoder_main.cfg > foreman_ref_main_encodelog.out 2>> foreman_ref_main_encodelog.err

     ***  Finished generating whole program pinballs [log_whole]  ***    May 22, 2020 16:05:15

     ***********************************************************************************************
     pinpoints.py ERROR: A problem occurred in phase - Whole program pinball generation [log_whole]

     ***  TRACING: END  ***    May 22, 2020 16:05:15

3. h264ref.ref.info.status

   • Phase: Deleting tracing instance files                                            : Passed
     Phase: Whole program pinball generation [log_whole]                               : Failed 6:29:37.311588 (23377.311588)

4. foreman_ref_main_encodelog.err

   • 26974.67user 2.78system 7:29:42elapsed 99%CPU (0avgtext+0avgdata 171420maxresident)k
     0inputs+9896outputs (0major+1426501minor)pagefaults 0swaps
     23369.58user 2.06system 6:29:37elapsed 99%CPU (0avgtext+0avgdata 171012maxresident)k
     9096inputs+9888outputs (39major+1426876minor)pagefaults 0swaps

5. whole_program.ref/h264ref.ref_14118.log.txt

   • Enabled messages [UnixVersion()] 44  info: Informational messages (enabled by default)
     Logger finished sanity check
     Logger first configuration phase completed
     Region selector activated
     [0] Pinball basename: /home/user/specbenchtest/benchspec/CPU2006/464.h264ref/run/run_base_ref_gcc43-64bit.0000/whole_program.ref/h264ref.ref_14118
     Found VDSO area from Pin AUXV at 0x7ffd519e6000:0x7ffd519e8000
     Loading /home/user/specbenchtest/benchspec/CPU2006/464.h264ref/run/run_base_ref_gcc43-64bit.0000/h264ref(0x000400000:0x000481f63)
     Loading /lib64/ld-linux-x86-64.so.2(0x7f44d2418000:0x7f44d243d38f)
     [0] Started logging thread OSPid: 14118 OSTid: 14118
     [0] region-start
     [0] + RECEIVED AND PROCESSED START event: ip:0x7f44d2418c30 time:Fri May 22 09:35:38 2020
     Replacing AT_SYSINFO_EHDR
              from AT_SYSINFO_EHDR, 0x7ffd519e6000
              to AT_IGNORE,0
     [0] Stopping all threads - processing controller event
     [0] Stopping all threads succeeded
     [0] + STARTING NEW REGION at: ip:0x7f44d2418c33 time:Fri May 22 09:35:38 2020
     [0]  Region# : 0
     [0] Region has 1 existing threads(s)
     [0] Resuming all threads - complied processing controller event
     [0] InitRegion : 0
     Loading /lib/x86_64-linux-gnu/libm.so.6(0x7f44b98b2000:0x7f44b9bba0f7)
     Loading /lib/x86_64-linux-gnu/libc.so.6(0x7f44b94b4000:0x7f44b987d99f)
     [0] ThreadFini: inside region. thread at 0x7f44b9580746 @icount 598416603803
     [0] EndRegion called for pp_id: 0
     [0] End of Region at icount: 598416603803
     [0] ThreadFini: logger thread at 0x7f44b9580746
     [0] Closing global resources.
     [0] RemoveVsyscallFromRanges: removed pages: 0
     [0] Wrote address ranges: pagecount: 5766. Logged text pages count:5766 Logged data pages count:0 (total logged pages:5766)
     RemoveTextFileDuplications for /home/user/specbenchtest/benchspec/CPU2006/464.h264ref/run/run_base_ref_gcc43-64bit.0000/whole_program.ref/h264ref.ref_14118

6. I checked dmesg and did not find anything suspicious / seg faults / out of memory etc.

[Harish_P_Intel]

Hi Ananda:

Thanks for including all the details/files; that is really helpful.

log.txt file shows a normal end to the whole-program logging process. Did you run out of disk space by any chance?

Could you please check if the whole-program pinball is correctly generated: see if the *.address file is non-empty.

Try replaying the whole-program pinball if it seems complete.

It is possible that the whole-program generation worked but returned a non-zero status which caused the pinpoints.py script to report an error.

-Harish