topic Compiling SGX Applications With Coverage in Intel® Software Guard Extensions (Intel® SGX)

Compiling SGX Applications With Coverage

BenLD — Sun, 11 Oct 2020 15:10:28 GMT

Hi,

When trying to compile my SGX application using GCC with the --coverage flag in order to use gcov, I get the following compilation errors:

```

/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `gcov_write_block':
(.text+0x23): undefined reference to `fwrite'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `gcov_read_words':
(.text+0xed): undefined reference to `fread'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `gcov_error':
(.text+0x1d4): undefined reference to `__vfprintf_chk'
/usr/local/bin/ld: (.text+0x204): undefined reference to `fopen'
/usr/local/bin/ld: (.text+0x216): undefined reference to `stderr'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `__gcov_open':
(.text+0x39a): undefined reference to `open'
/usr/local/bin/ld: (.text+0x3d2): undefined reference to `fdopen'
/usr/local/bin/ld: (.text+0x3eb): undefined reference to `setbuf'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `__gcov_close':
(.text+0x438): undefined reference to `fclose'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `__gcov_read_summary':
(.text+0x79a): undefined reference to `__popcountdi2'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `__gcov_seek':
(.text+0x8f9): undefined reference to `fseek'
/usr/local/bin/ld: (.text+0x902): undefined reference to `ftell'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `gcov_do_dump':
(.text+0xf2f): undefined reference to `strcpy'
/usr/local/bin/ld: (.text+0x10cc): undefined reference to `stderr'
/usr/local/bin/ld: (.text+0x10db): undefined reference to `__fprintf_chk'
/usr/local/bin/ld: (.text+0x1107): undefined reference to `fseek'
/usr/local/bin/ld: (.text+0x1bf3): undefined reference to `fseek'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `__gcov_exit':
(.text+0x1e87): undefined reference to `stderr'
/usr/local/bin/ld: (.text+0x1e91): undefined reference to `fclose'
/usr/local/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/7/libgcov.a(_gcov.o): in function `__gcov_rewrite':
(.text+0x31e): undefined reference to `fseek'
collect2: error: ld returned 1 exit status

I believe that's because of the modified libc used when compiling SGX applications.

Is it possible to diagnose coverage in SGX applications (using gcov or some other method)?

Re: Compiling SGX Applications With Coverage

Junli_S_Intel — Mon, 12 Oct 2020 03:01:08 GMT

Yes, we could get the code coverage data for SGX application. For non-enclave(non-trusted) part, just like normal application. For enclave(trusted) part, you need do some work-around to get the coverage data.

Re:Compiling Sgx Application With Coverage

JesusG_Intel — Tue, 13 Oct 2020 22:38:31 GMT

Hello Ben,

You can't use gcov to get coverage for enclave code but you can use the Intel VTune Profiler to find hotspots, or areas of high CPU utilization, in your enclave code.

Use the Intel® VTune™ Amplifier Application 2016 Update 2 and higher to measure the performance of Intel® Software Guard Extensions (Intel® SGX) applications, including the enclave.

Follow these steps:

Download and install Intel® VTune™ Profiler
Use sgx-hotspots in the VTune command line:

vtune -collect sgx-hotspots -- /home/test/myApplication

You can read more about it here: Intel SGX Developer Reference for Linux, section "Performance Measurement using Intel® VTune(TM) Amplifier"

Is this similar to what you are looking for?

Re: Compiling SGX Applications With Coverage

BenLD — Sun, 18 Oct 2020 08:56:44 GMT

Thanks, but since the vast majority of my code is in the trusted part, I won't get much benefit from instrumenting the non-trusted part. What's the work-around you're referring to? I couldn't find any.

Re: Re:Compiling Sgx Application With Coverage

BenLD — Sun, 18 Oct 2020 09:01:15 GMT

Thank you. Unfortunately, VTune won't work for me since I'm not looking to profile my project for performance issues, but rather use gcov (https://gcc.gnu.org/onlinedocs/gcc/Gcov-Intro.html#Gcov-Intro) or something similar to analyze tests coverage, e.g. which lines of code in my project is executed as result of the tests. Usually I just compile the code using GCC with the --coverage flag and execute the tests, but as stated in the original post, it doesn't work.

Re:Compiling Sgx Application With Coverage

JesusG_Intel — Thu, 22 Oct 2020 14:15:34 GMT

Hi Ben,

Good news. The open source DCAP Quote Verification Library uses Bullseye (bullseye.com) for code coverage testing: https://github.com/intel/SGXDataCenterAttestationPrimitives/tree/master/QuoteVerification/QVL.

They don't explicitly explain how to do it but you can explore the project files to see how they use Bullseye for coverage testing.

Re: Re:Compiling Sgx Application With Coverage

BenLD — Mon, 26 Oct 2020 10:18:47 GMT

Thank you, but looking at their CMakeLists.txt it seems like they compile with --coverage, same as I do, but I get errors such as: undefined reference to `fclose`, possibly because of the way sgx modify the interaction with the filesystem as in here?

https://github.com/intel/linux-sgx/blob/master/common/inc/sgx_tprotected_fs.h

I wonder how their code even compiles.

Re: Compiling SGX Applications With Coverage

JesusG_Intel — Mon, 26 Oct 2020 22:19:40 GMT

Ben,

You are trying to call the untrusted gcov library from within your enclave and that is why you are getting all those errors. In CMakeLists.txt, https://github.com/intel/SGXDataCenterAttestationPrimitives/blob/master/QuoteVerification/QVL/Src/CMakeLists.txt, they use the --coverage flag under If (NOT BUILD_ENCLAVE).

Refer to the Trusted Libraries section of the Intel SGX Developer Reference for Linux for more information on library calls you can make directly from within enclaves. Any other calls you need to make to functions outside of the enclave have to be wrapped in an OCALL.

Re: Compiling SGX Applications With Coverage

BenLD — Thu, 29 Oct 2020 10:56:43 GMT

Thank you. So, basically, it's either I'm re-writing gcov or there's no way to generate coverage reports for the enclave code? In other words, there's currently no existing solution for estimating tests' code coverage of sgx applications?

Re:Compiling Sgx Application With Coverage

JesusG_Intel — Thu, 29 Oct 2020 22:14:11 GMT

Hello Ben,

In one of my previous responses I mentioned that the engineering team uses Bullseye (bullseye.com) for measuring code coverage within enclave code.

You can see how they use this tool in the Quote Verification Library, https://github.com/intel/SGXDataCenterAttestationPrimitives/blob/ab8d31d72f842adb4b8a49eb3639f2e9a789d13b/QuoteVerification/QVL/README.md. Search for "Bullseye" and "coverage" in the Github repository.

I am not familiar with Bullseye so, unfortunately, I cannot provide more details than this.

Re: Re:Compiling Sgx Application With Coverage

BenLD — Mon, 02 Nov 2020 10:24:37 GMT

Thank you. I think bullseye is not the issue - they seem to compile with "--coverage" and then use bullseye, but I can't even get the code compiling with "--coverage" in the first place.

Re:Compiling Sgx Application With Coverage

JesusG_Intel — Wed, 04 Nov 2020 17:04:00 GMT

Hello Ben,

I'm sorry for causing confusion. It is not possible to gather coverage data from within an enclave. The workaround the engineering team uses is to use one ocall to transfer the coverage data out of the SGX enclave. I don't know if this is enough for you to figure out the solution but I am trying to get the next level of detail.

Re: Re:Compiling Sgx Application With Coverage

BenLD — Thu, 12 Nov 2020 16:09:05 GMT

Thank you. Any news on how they do it?

Re: Compiling SGX Applications With Coverage

JesusG_Intel — Thu, 12 Nov 2020 21:07:28 GMT

Hello BenLD,

I have not heard back from engineering. It may be a while until I can get a response but I will respond here as soon as I have more guidance.