topic Re: intel vtune to profile sgx application in Intel® Software Guard Extensions (Intel® SGX)

intel vtune to profile sgx application

maxdd — Sat, 04 Dec 2021 21:09:39 GMT

I want to use intel vtune to profile sgx application. The vtune version is 2021.9.0. I execute following commands as root:

export INTEL_LIBITTNOTIFY64=/opt/intel/oneapi/vtune/latest/lib64/runtime/libittnotify_collector.so ./vtune -collect sgx-hotspots -collect-with runss -target-pid 31127

The output is:

vtune: Warning: To enable hardware event-based sampling, VTune Profiler has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes. vtune: Collection started. To stop the collection, either press CTRL-C or enter from another console window: vtune -r /opt/intel/oneapi/vtune/2021.9.0/bin64/r005sgxhs -command stop. ^Cvtune: Collection detached. vtune: Collection stopped. vtune: Using result path `/opt/intel/oneapi/vtune/2021.9.0/bin64/r005sgxhs' vtune: Executing actions 21 % Resolving information for `vmlinux' vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions. vtune: Executing actions 75 % Generating a report Elapsed Time: 3.091s Precise Clockticks: 5,224,007,836 Total Thread Count: 4 Paused Time: 0s Top Hotspots Function Module Precise Clockticks ------------------------- ------------------------ ------------------ __morestack libsgx_urts.so.1.1.103.3 3,324,004,986 dp_netdev_input__ ovs-vswitchd 120,000,180 _mm256_loadu_si256 librte_vhost.so.20.0 106,000,159 restore_xregs libsgx_urts.so.1.1.103.3 104,000,156 __GI___pthread_mutex_lock libpthread-2.27.so 102,000,153 [Others] N/A 1,468,002,202 Effective Logical Core Utilization: 0.0% (0.000 out of 8) Collection and Platform Info User Name: root Operating System: 4.15.0-48-generic DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS" Computer Name: h8.trafgen.sdnnfv.beluga.pdl.cmu.local Result Size: 7.5 MB Collection start time: 16:34:19 04/12/2021 UTC Collection stop time: 16:34:22 04/12/2021 UTC Collector Type: Event-based sampling driver CPU Name: Intel(R) Processor code named Kabylake Frequency: 3.792 GHz Logical CPU Count: 8 Cache Allocation Technology Level 2 capability: not detected Level 3 capability: not detected If you want to skip descriptions of detected performance issues in the report, enter: vtune -report summary -report-knob show-issues=false -r <my_result_dir>. Alternatively, you may view the report in the csv format: vtune -report <report_name> -format=csv. vtune: Executing actions 100 % done

It does not show detailed results about how functions are executed in enclave. It only shows '__morestack' and 'restore_xregs'. Is there a way to further decompose the execution of those functions?

When compiling the SGX application, I am using SGX_DEBUG mode.

This is the config I use for enclave.

<EnclaveConfiguration> <ProdID>0</ProdID> <ISVSVN>0</ISVSVN> <StackMaxSize>262144</StackMaxSize> <HeapMaxSize>10240000</HeapMaxSize> <TCSNum>10</TCSNum> <TCSPolicy>1</TCSPolicy>  <DisableDebug>0</DisableDebug> <MiscSelect>0</MiscSelect> <MiscMask>0xFFFFFFFF</MiscMask> </EnclaveConfiguration>

Re:intel vtune to profile sgx application

JesusG_Intel — Tue, 07 Dec 2021 23:06:28 GMT

Hello maxdd,

It took me a while to answer because I was experimenting with VTune, which I don't have much experience with.

I wasn't able to see my enclave functions in the Vtune report until I shortened the sampling interval using the sampling-interval knob.

$ vtune -collect sgx-hotspots -knob sampling-interval=.01 ./app

If using Vtune from the command line only, you can generate CSV versions of various reports by running:

$ vtune -report <reportname> -format=csv

Use vtune -help report to find available report names, e.g. callstacks, hotspots, top-down

To see data from the enclave only, I used:

$ vtune -report hotspots -format=csv -filter module=enclave.signed.so

I did not try attaching to a process, rather, I started the app using Vtune directly as shown above. I used a modified version of the SGX SDK sample, SampleEnclave, which runs very quickly.

Sincerely,

Jesus G.

Intel Customer Support

Re: intel vtune to profile sgx application

JesusG_Intel — Tue, 07 Dec 2021 23:10:25 GMT

I will help you on this thread and delete the other two threads you opened on this topic.

Re: Re:intel vtune to profile sgx application

maxdd — Wed, 08 Dec 2021 18:14:49 GMT

Thanks for the reply.

I try it again but I still can't get information about the functions in enclave. Do I need to install some debug symbols?

Also, can you show me the results you get running on SampleEnclave?

root@h8:/opt/intel/oneapi/vtune/2021.9.0/bin64# ./vtune -collect sgx-hotspots -knob sampling-interval=.01 -target-pid 16984 vtune: Warning: To enable hardware event-based sampling, VTune Profiler has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes. Warning: sav for event CPU_CLK_UNHALTED.REF_TSC is too low, value reset to 100000 vtune: Collection started. To stop the collection, either press CTRL-C or enter from another console window: vtune -r /opt/intel/oneapi/vtune/2021.9.0/bin64/r013sgxhs -command stop. ^Cvtune: Collection detached. vtune: Collection stopped. vtune: Using result path `/opt/intel/oneapi/vtune/2021.9.0/bin64/r013sgxhs' vtune: Executing actions 19 % Resolving information for `libsgx_urts.so.1.1.103 vtune: Warning: Cannot locate debugging information for file `/lib/modules/4.15.0-48-generic/kernel/arch/x86/kvm/kvm.ko'. vtune: Executing actions 19 % Resolving information for `nf_nat' vtune: Warning: Cannot locate debugging information for file `/lib/modules/4.15.0-48-generic/kernel/net/netfilter/nf_nat.ko'. vtune: Warning: Cannot locate file `sep5.ko'. vtune: Executing actions 20 % Resolving information for `tg3' vtune: Warning: Cannot locate debugging information for file `/lib/modules/4.15.0-48-generic/kernel/drivers/net/ethernet/broadcom/tg3.ko'. vtune: Executing actions 20 % Resolving information for `ip_tables' vtune: Warning: Cannot locate debugging information for file `/lib/modules/4.15.0-48-generic/kernel/net/ipv4/netfilter/ip_tables.ko'. vtune: Executing actions 20 % Resolving information for `nf_conntrack' vtune: Warning: Cannot locate debugging information for file `/lib/modules/4.15.0-48-generic/kernel/net/netfilter/nf_conntrack.ko'. vtune: Executing actions 22 % Resolving information for `vmlinux' vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions. vtune: Executing actions 75 % Generating a report Elapsed Time: 2.107s Precise Clockticks: 2,912,020,000 Total Thread Count: 8 Paused Time: 0s Top Hotspots Function Module Precise Clockticks -------------------------- ------------------------ ------------------ __morestack libsgx_urts.so.1.1.103.3 2,455,040,000 virtio_dev_tx_split ovs-vswitchd 195,320,000 dp_netdev_input__ ovs-vswitchd 90,880,000 miniflow_extract ovs-vswitchd 20,680,000 netdev_dpdk_vhost_rxq_recv ovs-vswitchd 16,480,000 [Others] N/A 133,620,000 Effective Logical Core Utilization: 0.0% (0.000 out of 8) Collection and Platform Info User Name: root Operating System: 4.15.0-48-generic DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS" Computer Name: h8.trafgen.sdnnfv.beluga.pdl.cmu.local Result Size: 79.4 MB Collection start time: 18:04:29 08/12/2021 UTC Collection stop time: 18:04:31 08/12/2021 UTC Collector Type: Event-based sampling driver CPU Name: Intel(R) Processor code named Kabylake Frequency: 3.792 GHz Logical CPU Count: 8 Cache Allocation Technology Level 2 capability: not detected Level 3 capability: not detected If you want to skip descriptions of detected performance issues in the report, enter: vtune -report summary -report-knob show-issues=false -r <my_result_dir>. Alternatively, you may view the report in the csv format: vtune -report <report_name> -format=csv. vtune: Executing actions 100 % done

Re: intel vtune to profile sgx application

JesusG_Intel — Wed, 08 Dec 2021 20:54:25 GMT

Hello maxdd,

You do not need to install extra debug symbols. Compiling with SGX_DEBUG=1 is enough to compile your enclave code with the necessary debug symbols.

Try starting the application from VTune directly instead of attaching to a process. Below I show the outputs for the VTune run of SampleEnclave followed by the hotspots report in CSV format.

VTune Run

/opt/intel/sgxsdk/SampleCode/SampleEnclave$ vtune -collect sgx-hotspots -knob sampling-interval=.01 ./app

vtune: Warning: To enable hardware event-based sampling, VTune Profiler has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.

Warning: sav for event CPU_CLK_UNHALTED.REF_TSC is too low, value reset to 100000

vtune: Collection started. To stop the collection, either press CTRL-C or enter from another console window: vtune -r /opt/intel/sgxsdk/SampleCode/SampleEnclave/r002sgxhs -command stop.

Checksum(0x0x7fffa8d23540, 100) = 0xfffd4143

Info: executing thread synchronization, please wait...

Info: SampleEnclave successfully returned.

Enter a character before exit ...

vtune: Collection stopped.

vtune: Using result path `/opt/intel/sgxsdk/SampleCode/SampleEnclave/r002sgxhs'

vtune: Executing actions 19 % Resolving information for `isgx'

vtune: Warning: Cannot locate debugging information for file `/lib/modules/5.4.0-71-generic/kernel/drivers/intel/sgx/isgx.ko'.

vtune: Warning: Cannot read load addresses of sections from `/sys/module/isgx/sections'. This may affect the correctness of symbol resolution for `isgx'. Make sure this directory exists and all files in this directory have read permissions.

vtune: Warning: Cannot locate debugging information for file `/usr/lib/x86_64-linux-gnu/libprotobuf.so.17.0.0'.

vtune: Executing actions 19 % Resolving information for `e1000e'

vtune: Warning: Cannot locate debugging information for file `/lib/modules/5.4.0-71-generic/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko'.

vtune: Warning: Cannot read load addresses of sections from `/sys/module/e1000e/sections'. This may affect the correctness of symbol resolution for `e1000e'. Make sure this directory exists and all files in this directory have read permissions.

vtune: Executing actions 20 % Resolving information for `app'

vtune: Warning: Cannot locate file `vmlinux'.

vtune: Executing actions 20 % Resolving information for `libsgx_enclave_common.

vtune: Warning: Cannot locate debugging information for file `/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28'.

vtune: Executing actions 21 % Resolving information for `libsgx_enclave_common.

vtune: Warning: Cannot locate file `sep5.ko'.

vtune: Executing actions 21 % Resolving information for `libittnotify_collector

vtune: Warning: Cannot locate debugging information for file `/opt/intel/oneapi/vtune/2021.9.0/lib64/runtime/libittnotify_collector.so'.

vtune: Executing actions 21 % Resolving information for `xt_comment'

vtune: Warning: Cannot locate debugging information for file `/lib/modules/5.4.0-71-generic/kernel/net/netfilter/xt_comment.ko'.

vtune: Warning: Cannot read load addresses of sections from `/sys/module/xt_comment/sections'. This may affect the correctness of symbol resolution for `xt_comment'. Make sure this directory exists and all files in this directory have read permissions.

vtune: Executing actions 22 % Resolving information for `xt_comment'

vtune: Warning: Function and source-level analysis for the Linux kernel will not be possible since neither debug version of the kernel nor kernel symbol tables are found. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.

vtune: Executing actions 75 % Generating a report Elapsed Time: 8.351s

Precise Clockticks: 677,040,000

Total Thread Count: 10

Paused Time: 0s

Top Hotspots

Function Module Precise Clockticks

---------------------------------- ------------------------ ------------------

[vmlinux] vmlinux 293,520,000

do_ocall enclave.signed.so 134,320,000

__morestack libsgx_urts.so.1.1.117.1 131,580,000

CTrustThreadPool::get_bound_thread libsgx_urts.so.1.1.117.1 12,360,000

__GI___pthread_mutex_lock libpthread-2.31.so 10,280,000

[Others] N/A 94,980,000

Effective Logical Core Utilization: 0.3% (0.023 out of

Collection and Platform Info

Application Command Line: ./app

User Name: el-pelon

Operating System: 5.4.0-71-generic DISTRIB_ID=Ubuntu DISTRIB_RELEASE=20.04 DISTRIB_CODENAME=focal DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"

Computer Name: del-microphone

Result Size: 29.8 MB

Collection start time: 20:25:41 08/12/2021 UTC

Collection stop time: 20:25:49 08/12/2021 UTC

Collector Type: Event-based sampling driver

CPU

Name: Intel(R) Processor code named Kabylake

Frequency: 3.096 GHz

Logical CPU Count: 8

Cache Allocation Technology

Level 2 capability: not detected

Level 3 capability: not detected

If you want to skip descriptions of detected performance issues in the report,

enter: vtune -report summary -report-knob show-issues=false -r <my_result_dir>.

Alternatively, you may view the report in the csv format: vtune -report

<report_name> -format=csv.

vtune: Executing actions 100 % done

Hotspots Report

/opt/intel/sgxsdk/SampleCode/SampleEnclave$ vtune -report hotspots -format=csv -filter module=enclave.signed.so

vtune: Using result path `/opt/intel/sgxsdk/SampleCode/SampleEnclave/r002sgxhs'

vtune: Executing actions 75 % Generating a report Function Precise Clockticks Module Function (Full) Source File Start Address

do_ocall 134320000 enclave.signed.so do_ocall [Unknown] 0x1ef94

restore_xregs 7960000 enclave.signed.so restore_xregs [Unknown] 0x1edc1

enclave_entry 6900000 enclave.signed.so enclave_entry [Unknown] 0x1ee89

sgx_thread_mutex_lock 4440000 enclave.signed.so sgx_thread_mutex_lock [Unknown] 0xd900

sgx_spin_lock 3680000 enclave.signed.so sgx_spin_lock [Unknown] 0x9fb0

asm_oret 3140000 enclave.signed.so asm_oret [Unknown] 0x1f0c4

do_oret 2200000 enclave.signed.so do_oret [Unknown] 0x8b00

save_xregs 1800000 enclave.signed.so save_xregs [Unknown] 0x1edf1

sgx_ocfree 1220000 enclave.signed.so sgx_ocfree [Unknown] 0x72e0

sgx_thread_set_untrusted_event_ocall 1100000 enclave.signed.so sgx_thread_set_untrusted_event_ocall Enclave_t.c 0x3159

sgx_thread_wait_untrusted_event_ocall 1100000 enclave.signed.so sgx_thread_wait_untrusted_event_ocall Enclave_t.c 0x30a7

ecall_consumer 1040000 enclave.signed.so ecall_consumer Thread.cpp 0x6eb4

sgx_thread_cond_wait 840000 enclave.signed.so sgx_thread_cond_wait [Unknown] 0xd6b0

sgx_thread_mutex_unlock 720000 enclave.signed.so sgx_thread_mutex_unlock [Unknown] 0xdb60

do_rdrand 640000 enclave.signed.so do_rdrand [Unknown] 0x1f1a5

__morestack 620000 enclave.signed.so __morestack [Unknown] 0x1f0b1

enter_enclave 580000 enclave.signed.so enter_enclave [Unknown] 0x1ec20

ecall_producer 480000 enclave.signed.so ecall_producer Thread.cpp 0x6d93

ecall_increase_counter 380000 enclave.signed.so ecall_increase_counter Thread.cpp 0x6d16

do_ecall 320000 enclave.signed.so do_ecall [Unknown] 0x8540

sgx_thread_setwait_untrusted_events_ocall 320000 enclave.signed.so sgx_thread_setwait_untrusted_events_ocall Enclave_t.c 0x320b

get_thread_data 300000 enclave.signed.so get_thread_data [Unknown] 0x1ee6d

sgx_ocall 300000 enclave.signed.so sgx_ocall [Unknown] 0x8a90

sgx_ocalloc 260000 enclave.signed.so sgx_ocalloc [Unknown] 0x71f0

update_ocall_lastsp 160000 enclave.signed.so update_ocall_lastsp [Unknown] 0x8ac0

sgx_is_within_enclave 140000 enclave.signed.so sgx_is_within_enclave [Unknown] 0x7140

sgx_thread_cond_signal 140000 enclave.signed.so sgx_thread_cond_signal [Unknown] 0xd840

sgx_is_enclave_crashed 120000 enclave.signed.so sgx_is_enclave_crashed [Unknown] 0x1ec00

sgx_thread_mutex_unlock_lazy 100000 enclave.signed.so sgx_thread_mutex_unlock_lazy [Unknown] 0xda70

_libc___memset 80000 enclave.signed.so _libc___memset [Unknown] 0x12340

get_enclave_state 80000 enclave.signed.so get_enclave_state [Unknown] 0x1ee2e

__memcpy 60000 enclave.signed.so __memcpy [Unknown] 0xd450

_ULx86_64_dwarf_extract_proc_info_from_fde 40000 enclave.signed.so _ULx86_64_dwarf_extract_proc_info_from_fde [Unknown] 0x1dcb0

do_init_thread 40000 enclave.signed.so do_init_thread [Unknown] 0x8360

sgx_ecall_consumer 40000 enclave.signed.so sgx_ecall_consumer Enclave_t.c 0x27da

sgx_read_rand 40000 enclave.signed.so sgx_read_rand [Unknown] 0x7340

sgx_spin_unlock 40000 enclave.signed.so sgx_spin_unlock [Unknown] 0xa020

_ULx86_64_access_reg 20000 enclave.signed.so _ULx86_64_access_reg [Unknown] 0x1c090

_ULx86_64_dwarf_callback 20000 enclave.signed.so _ULx86_64_dwarf_callback [Unknown] 0x1d2e0

_ULx86_64_dwarf_read_encoded_pointer 20000 enclave.signed.so _ULx86_64_dwarf_read_encoded_pointer [Unknown] 0x19680

__vfprintf 20000 enclave.signed.so __vfprintf [Unknown] 0xed80

create_state_record_for.part.0 20000 enclave.signed.so create_state_record_for.part.0 [Unknown] 0x185f0

dlfree 20000 enclave.signed.so dlfree [Unknown] 0xc3b0

do_init_enclave 20000 enclave.signed.so do_init_enclave [Unknown] 0x1eb00

do_relocs 20000 enclave.signed.so do_relocs [Unknown] 0x93b0

ecall_map 20000 enclave.signed.so ecall_map Libcxx.cpp 0x45c5

elf_tls_info 20000 enclave.signed.so elf_tls_info [Unknown] 0x9770

mbrtowc 20000 enclave.signed.so mbrtowc [Unknown] 0x15070

operator new[] 20000 enclave.signed.so operator new[](unsigned long) [Unknown] 0x1ac50

run_cfi_program 20000 enclave.signed.so run_cfi_program [Unknown] 0x17b50

sgx_ecall_pointer_in 20000 enclave.signed.so sgx_ecall_pointer_in Enclave_t.c 0x1645

std::__1::__tree<std::__1::__value_type<char, int>, std::__1::__map_value_compare<char, std::__1::__value_type<char, int>, std::__1::less<char>, (bool)1>, std::__1::allocator<std::__1::__value_type<char, int>>>::__construct_node<std::__1::pair<char const, int>> 20000 enclave.signed.so std::__1::__tree<std::__1::__value_type<char, int>, std::__1::__map_value_compare<char, std::__1::__value_type<char, int>, std::__1::less<char>, (bool)1>, std::__1::allocator<std::__1::__value_type<char, int>>>::__construct_node<std::__1::pair<char const, int>>(std::__1::pair<char const, int>&&) __tree 0x642c

std::__1::__tree<std::__1::__value_type<char, int>, std::__1::__map_value_compare<char, std::__1::__value_type<char, int>, std::__1::less<char>, (bool)1>, std::__1::allocator<std::__1::__value_type<char, int>>>::__root_ptr 20000 enclave.signed.so std::__1::__tree<std::__1::__value_type<char, int>, std::__1::__map_value_compare<char, std::__1::__value_type<char, int>, std::__1::less<char>, (bool)1>, std::__1::allocator<std::__1::__value_type<char, int>>>::__root_ptr(void) const __tree 0x67b8

std::exception::exception 20000 enclave.signed.so std::exception::exception(void) [Unknown] 0x1ab40

trts_ecall 20000 enclave.signed.so trts_ecall(unsigned int, void*) [Unknown] 0x8130

vtune: Executing actions 100 % done

Re:intel vtune to profile sgx application

JesusG_Intel — Wed, 15 Dec 2021 23:52:04 GMT

Hello Maxdd,

I hope the information I provided was helpful to you. Since we haven't heard from you, Intel will stop monitoring this thread. If you need further assistance, please open a new thread.

Sincerely,

Jesus G.

Intel Customer Support