Compiling SGX Applications With Coverage

[BenLD]

Hi,

When trying to compile my SGX application using GCC with the --coverage flag in order to use gcov, I get the following compilation errors:

```

/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `gcov_write_block':
(.text+0x23): undefined reference to `fwrite'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `gcov_read_words':
(.text+0xed): undefined reference to `fread'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `gcov_error':
(.text+0x1d4): undefined reference to `__vfprintf_chk'
/usr/local/bin/ld: (.text+0x204): undefined reference to `fopen'
/usr/local/bin/ld: (.text+0x216): undefined reference to `stderr'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `__gcov_open':
(.text+0x39a): undefined reference to `open'
/usr/local/bin/ld: (.text+0x3d2): undefined reference to `fdopen'
/usr/local/bin/ld: (.text+0x3eb): undefined reference to `setbuf'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `__gcov_close':
(.text+0x438): undefined reference to `fclose'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `__gcov_read_summary':
(.text+0x79a): undefined reference to `__popcountdi2'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `__gcov_seek':
(.text+0x8f9): undefined reference to `fseek'
/usr/local/bin/ld: (.text+0x902): undefined reference to `ftell'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `gcov_do_dump':
(.text+0xf2f): undefined reference to `strcpy'
/usr/local/bin/ld: (.text+0x10cc): undefined reference to `stderr'
/usr/local/bin/ld: (.text+0x10db): undefined reference to `__fprintf_chk'
/usr/local/bin/ld: (.text+0x1107): undefined reference to `fseek'
/usr/local/bin/ld: (.text+0x1bf3): undefined reference to `fseek'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `__gcov_exit':
(.text+0x1e87): undefined reference to `stderr'
/usr/local/bin/ld: (.text+0x1e91): undefined reference to `fclose'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `__gcov_rewrite':
(.text+0x31e): undefined reference to `fseek'
collect2: error: ld returned 1 exit status

I believe that's because of the modified libc used when compiling SGX applications.

Is it possible to diagnose coverage in SGX applications (using gcov or some other method)?

[Junli_S_Intel]

Yes, we could get the code coverage data for SGX application.  For non-enclave(non-trusted) part, just like normal application. For enclave(trusted) part, you need do some work-around to get the coverage data.

[BenLD]

Thanks, but since the vast majority of my code is in the trusted part, I won't get much benefit from instrumenting the non-trusted part. What's the work-around you're referring to? I couldn't find any.

[JesusG_Intel]

Hello Ben,

You can't use gcov to get coverage for enclave code but you can use the Intel VTune Profiler to find hotspots, or areas of high CPU utilization, in your enclave code.

Use the Intel® VTune™ Amplifier Application 2016 Update 2 and higher to measure the performance of Intel® Software Guard Extensions (Intel® SGX) applications, including the enclave.

Follow these steps:

1. Download and install Intel® VTune™ Profiler
2. Use sgx-hotspots in the VTune command line:

vtune -collect sgx-hotspots -- /home/test/myApplication

You can read more about it here: Intel SGX Developer Reference for Linux, section "Performance Measurement using Intel® VTune(TM) Amplifier"

Is this similar to what you are looking for?

[BenLD]

Thank you. Unfortunately, VTune won't work for me since I'm not looking to profile my project for performance issues, but rather use gcov (https://gcc.gnu.org/onlinedocs/gcc/Gcov-Intro.html#Gcov-Intro) or something similar to analyze tests coverage, e.g. which lines of code in my project is executed as result of the tests. Usually I just compile the code using GCC with the --coverage flag and execute the tests, but as stated in the original post, it doesn't work.

[JesusG_Intel]

Hi Ben,

Good news. The open source DCAP Quote Verification Library uses Bullseye (bullseye.com) for code coverage testing: https://github.com/intel/SGXDataCenterAttestationPrimitives/tree/master/QuoteVerification/QVL.

They don't explicitly explain how to do it but you can explore the project files to see how they use Bullseye for coverage testing.

[BenLD]

Thank you, but looking at their CMakeLists.txt it seems like they compile with --coverage, same as I do, but I get errors such as: undefined reference to `fclose`, possibly because of the way sgx modify the interaction with the filesystem as in here?

https://github.com/intel/linux-sgx/blob/master/common/inc/sgx_tprotected_fs.h

I wonder how their code even compiles.

[JesusG_Intel]

Ben,

You are trying to call the untrusted gcov library from within your enclave and that is why you are getting all those errors. In CMakeLists.txt, https://github.com/intel/SGXDataCenterAttestationPrimitives/blob/master/QuoteVerification/QVL/Src/CMakeLists.txt, they use the --coverage flag under If (NOT BUILD_ENCLAVE).

Refer to the Trusted Libraries section of the Intel SGX Developer Reference for Linux for more information on library calls you can make directly from within enclaves. Any other calls you need to make to functions outside of the enclave have to be wrapped in an OCALL.

[BenLD]

Thank you. So, basically, it's either I'm re-writing gcov or there's no way to generate coverage reports for the enclave code? In other words, there's currently no existing solution for estimating tests' code coverage of sgx applications?

[JesusG_Intel]

Hello Ben,

In one of my previous responses I mentioned that the engineering team uses Bullseye (bullseye.com) for measuring code coverage within enclave code.

You can see how they use this tool in the Quote Verification Library, https://github.com/intel/SGXDataCenterAttestationPrimitives/blob/ab8d31d72f842adb4b8a49eb3639f2e9a789d13b/QuoteVerification/QVL/README.md. Search for "Bullseye" and "coverage" in the Github repository.

I am not familiar with Bullseye so, unfortunately, I cannot provide more details than this.

[BenLD]

Thank you. I think bullseye is not the issue - they seem to compile with "--coverage" and then use bullseye, but I can't even get the code compiling with "--coverage" in the first place.

[JesusG_Intel]

Hello Ben,

I'm sorry for causing confusion. It is not possible to gather coverage data from within an enclave. The workaround the engineering team uses is to use one ocall to transfer the coverage data out of the SGX enclave. I don't know if this is enough for you to figure out the solution but I am trying to get the next level of detail.

[BenLD]

Thank you. Any news on how they do it?

[JesusG_Intel]

Hello BenLD,

I have not heard back from engineering. It may be a while until I can get a response but I will respond here as soon as I have more guidance.