SDE - Pinball replay not reproducible with PIN_SafeCopy

Research_CSL · ‎02-06-2023

We're running into an issue where we see weird behaviour with the reproducability of a recorded pinball on 32-bit executables.

Occasionally (~1% of the runs with statically linked executables, ~85% of the runs with dynamically linked executables) we see a difference in output generated by a pintool using PIN_SafeCopy. This behaviour doesn't seem to happen on 64-bit executables (we tested this by running the same Pin tool on the same input program, but compiled using 64-bit architecture).

We brought down the Pin tool and the input program to a bare minimum (see bottom of this post), but we still get this inconsistent behaviour.

Could this be possibly related to a warning we get when compiling the Pin tool ("/usr/bin/ld: /sde-external-9.7.0-2022-05-09-lin/pinkit/sde-example/lib/ia32/libsde.a(sde-init.o): warning: relocation in read-only section `.text.startup'
/usr/bin/ld: warning: creating DT_TEXTREL in a shared object")?

We reproduce this as follows:

- Compile the program (optionally with -static)

- Create a Pinball with

sde -log:mt -log -log:basename repro/pinball -- repro.exe

- Replay twice with

sde -t32 pintool.so -t64 pintool.so -output repro.1.log -replay -replay:basename repro/pinball -replay:addr_trans -- nullapp
sde -t32 pintool.so -t64 pintool.so -output repro.2.log -replay -replay:basename repro/pinball -replay:addr_trans -- nullapp

- Diff and observe the different values every now and then. We notice that when a difference occurs, one of the replay output (the wrong one?) is always returning zeroes for Pin_SafeCopy.

SDE version: 9.7.0-2022-05-09-lin

#include <iostream>
#include <thread>

int main(int argc, char** argv)
{
    return 0;
}

#include <fstream>
#include <iostream>
#include <set>
#include <string>

#include "pin.H"
#include "sde-init.H"

static std::ofstream csv_memory_instructions;

static PIN_MUTEX csv_memory_instructions_lock;

// Option (-o) to set the output filename.
KNOB<std::string> KnobOutputFile(KNOB_MODE_WRITEONCE, "pintool", "output", "",
                                 "Specify the filename of the output file");

// Run before every memory access, i.e. both read and write.
VOID MemoryAccessBefore(THREADID threadID, ADDRINT instruction_address,
                        UINT32 mem_op, ADDRINT memory_address, ADDRINT size) {
  PIN_LockClient();
  RTN routine = RTN_FindByAddress(instruction_address);
  SEC section = RTN_Sec(routine);
  IMG image = SEC_Img(section);
  std::string image_name = IMG_Name(image);
  PIN_UnlockClient();

  // Copy the read/written value.

  unsigned char value_buf[size];

  PIN_LockClient();
  auto ret = PIN_SafeCopy(value_buf, reinterpret_cast<void *>(memory_address), size);
  PIN_UnlockClient();
  assert(ret == size);

  PIN_MutexLock(&csv_memory_instructions_lock);
  csv_memory_instructions << std::hex << image_name << "::" << RTN_Name(routine)
                          << "::" << instruction_address - IMG_LoadOffset(image)
                          << " tid: " << threadID << ": ";
  for (unsigned int i = 0; i < size; ++i) {
    csv_memory_instructions << (unsigned int)value_buf[i] << ' ';
  }
  csv_memory_instructions << '\n';
  csv_memory_instructions.flush();
  PIN_MutexUnlock(&csv_memory_instructions_lock);
}

// Instrumentation routine run for every instruction.
VOID OnInstruction(INS instruction, VOID *v) {
  UINT32 num_mem_operands = INS_MemoryOperandCount(instruction);

  for (UINT32 mem_op = 0; mem_op < num_mem_operands; ++mem_op) {
    ADDRINT size = INS_MemoryOperandSize(instruction, mem_op);

    if (INS_MemoryOperandIsRead(instruction, mem_op)) {
      INS_InsertPredicatedCall(instruction, IPOINT_BEFORE,
                               reinterpret_cast<AFUNPTR>(MemoryAccessBefore),
                               IARG_THREAD_ID, IARG_INST_PTR, IARG_UINT32,
                               mem_op, IARG_MEMORYOP_EA, mem_op, IARG_ADDRINT,
                               size, IARG_END);
    }
  }
}

int main(int argc, char *argv[]) {
  // Initialise the PIN symbol manager.
  PIN_InitSymbols();

  // Initialise Pin and SDE.
  sde_pin_init(argc, argv);
  sde_init();

  // Open the CSV files.
  csv_memory_instructions.open(KnobOutputFile.Value().c_str());

  // Register instruction callback.
  INS_AddInstrumentFunction(OnInstruction, nullptr);

  // Prevent the application from blocking SIGTERM.
  PIN_UnblockSignal(15, true);

  // Initialise mutexes.
  PIN_MutexInit(&csv_memory_instructions_lock);

  // Start the program (never returns).
  PIN_StartProgram();

  return EXIT_SUCCESS;
}

AdyT_Intel · ‎02-07-2023

I checked the code that you posted and I think that you might have an issue in line 29.

You are allocating a buffer (on the stack) with dynamic value.
I think that this is not well supported by the pinCRT (C runtime and C++ compiler runtime libraries).

Therefore, I would try it with new and delete or using a constant big enough for your use.
Something like 1024 will fit most cases.

The SafeCopy function will return 0 when the value_buf is illegal pointer.

Research_CSL · ‎02-07-2023

Hi,

The original (much longer) Pin tool used new/delete, but we've retried with the next (slightly adapted to the original posted above to use new/delete), and we're still running into the same issue.

Fixing the buffer size to a constant value rather (1024) than sizing it just appropriately didn't help out either unfortunately.

#include <fstream>
#include <iostream>
#include <set>
#include <string>

#include "pin.H"
#include "sde-init.H"

static std::ofstream csv_memory_instructions;

static PIN_MUTEX csv_memory_instructions_lock;

// Option (-o) to set the output filename.
KNOB<std::string> KnobOutputFile(KNOB_MODE_WRITEONCE, "pintool", "output", "",
                                 "Specify the filename of the output file");

// Run before every memory access, i.e. both read and write.
VOID MemoryAccessBefore(THREADID threadID, ADDRINT instruction_address,
                        UINT32 mem_op, ADDRINT memory_address, ADDRINT size) {
  PIN_LockClient();
  RTN routine = RTN_FindByAddress(instruction_address);
  SEC section = RTN_Sec(routine);
  IMG image = SEC_Img(section);
  std::string image_name = IMG_Name(image);
  PIN_UnlockClient();

  // Copy the read/written value.

  unsigned char *value_buf = new unsigned char[size];

  PIN_LockClient();
  auto ret = PIN_SafeCopy(value_buf, reinterpret_cast<void *>(memory_address), size);
  PIN_UnlockClient();
  assert(ret == size);

  PIN_MutexLock(&csv_memory_instructions_lock);
  csv_memory_instructions << std::hex << image_name << "::" << RTN_Name(routine)
                          << "::" << instruction_address - IMG_LoadOffset(image)
                          << " tid: " << threadID << ": ";
  for (unsigned int i = 0; i < size; ++i) {
    csv_memory_instructions << (unsigned int)value_buf[i] << ' ';
  }
  csv_memory_instructions << '\n';
  csv_memory_instructions.flush();
  PIN_MutexUnlock(&csv_memory_instructions_lock);

  delete[] value_buf;
}

// Instrumentation routine run for every instruction.
VOID OnInstruction(INS instruction, VOID *v) {
  UINT32 num_mem_operands = INS_MemoryOperandCount(instruction);

  for (UINT32 mem_op = 0; mem_op < num_mem_operands; ++mem_op) {
    ADDRINT size = INS_MemoryOperandSize(instruction, mem_op);

    if (INS_MemoryOperandIsRead(instruction, mem_op)) {
      INS_InsertPredicatedCall(instruction, IPOINT_BEFORE,
                               reinterpret_cast<AFUNPTR>(MemoryAccessBefore),
                               IARG_THREAD_ID, IARG_INST_PTR, IARG_UINT32,
                               mem_op, IARG_MEMORYOP_EA, mem_op, IARG_ADDRINT,
                               size, IARG_END);
    }
  }
}

int main(int argc, char *argv[]) {
  // Initialise the PIN symbol manager.
  PIN_InitSymbols();

  // Initialise Pin and SDE.
  sde_pin_init(argc, argv);
  sde_init();

  // Open the CSV files.
  csv_memory_instructions.open(KnobOutputFile.Value().c_str());

  // Register instruction callback.
  INS_AddInstrumentFunction(OnInstruction, nullptr);

  // Prevent the application from blocking SIGTERM.
  PIN_UnblockSignal(15, true);

  // Initialise mutexes.
  PIN_MutexInit(&csv_memory_instructions_lock);

  // Start the program (never returns).
  PIN_StartProgram();

  return EXIT_SUCCESS;
}