- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I want to use intel vtune to profile sgx application. The vtune version is 2021.9.0. I execute following commands as root:
export INTEL_LIBITTNOTIFY64=/opt/intel/oneapi/vtune/latest/lib64/runtime/libittnotify_collector.so
./vtune -collect sgx-hotspots -collect-with runss -target-pid 31127
The output is:
vtune: Warning: To enable hardware event-based sampling, VTune Profiler has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.
vtune: Collection started. To stop the collection, either press CTRL-C or enter from another console window: vtune -r /opt/intel/oneapi/vtune/2021.9.0/bin64/r005sgxhs -command stop.
^Cvtune: Collection detached.
vtune: Collection stopped.
vtune: Using result path `/opt/intel/oneapi/vtune/2021.9.0/bin64/r005sgxhs'
vtune: Executing actions 21 % Resolving information for `vmlinux'
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
vtune: Executing actions 75 % Generating a report Elapsed Time: 3.091s
Precise Clockticks: 5,224,007,836
Total Thread Count: 4
Paused Time: 0s
Top Hotspots
Function Module Precise Clockticks
------------------------- ------------------------ ------------------
__morestack libsgx_urts.so.1.1.103.3 3,324,004,986
dp_netdev_input__ ovs-vswitchd 120,000,180
_mm256_loadu_si256 librte_vhost.so.20.0 106,000,159
restore_xregs libsgx_urts.so.1.1.103.3 104,000,156
__GI___pthread_mutex_lock libpthread-2.27.so 102,000,153
[Others] N/A 1,468,002,202
Effective Logical Core Utilization: 0.0% (0.000 out of 8)
Collection and Platform Info
User Name: root
Operating System: 4.15.0-48-generic DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
Computer Name: h8.trafgen.sdnnfv.beluga.pdl.cmu.local
Result Size: 7.5 MB
Collection start time: 16:34:19 04/12/2021 UTC
Collection stop time: 16:34:22 04/12/2021 UTC
Collector Type: Event-based sampling driver
CPU
Name: Intel(R) Processor code named Kabylake
Frequency: 3.792 GHz
Logical CPU Count: 8
Cache Allocation Technology
Level 2 capability: not detected
Level 3 capability: not detected
If you want to skip descriptions of detected performance issues in the report,
enter: vtune -report summary -report-knob show-issues=false -r <my_result_dir>.
Alternatively, you may view the report in the csv format: vtune -report
<report_name> -format=csv.
vtune: Executing actions 100 % done
It does not show detailed results about how functions are executed in enclave. It only shows '__morestack' and 'restore_xregs'. Is there a way to further decompose the execution of those functions?
When compiling the SGX application, I am using SGX_DEBUG mode.
This is the config I use for enclave.
<EnclaveConfiguration>
<ProdID>0</ProdID>
<ISVSVN>0</ISVSVN>
<StackMaxSize>262144</StackMaxSize>
<HeapMaxSize>10240000</HeapMaxSize>
<TCSNum>10</TCSNum>
<TCSPolicy>1</TCSPolicy>
<!-- Recommend changing 'DisableDebug' to 1 to make the enclave undebuggable for enclave release -->
<DisableDebug>0</DisableDebug>
<MiscSelect>0</MiscSelect>
<MiscMask>0xFFFFFFFF</MiscMask>
</EnclaveConfiguration>
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello maxdd,
It took me a while to answer because I was experimenting with VTune, which I don't have much experience with.
I wasn't able to see my enclave functions in the Vtune report until I shortened the sampling interval using the sampling-interval knob.
$ vtune -collect sgx-hotspots -knob sampling-interval=.01 ./app
If using Vtune from the command line only, you can generate CSV versions of various reports by running:
$ vtune -report <reportname> -format=csv
Use vtune -help report to find available report names, e.g. callstacks, hotspots, top-down
To see data from the enclave only, I used:
$ vtune -report hotspots -format=csv -filter module=enclave.signed.so
I did not try attaching to a process, rather, I started the app using Vtune directly as shown above. I used a modified version of the SGX SDK sample, SampleEnclave, which runs very quickly.
Sincerely,
Jesus G.
Intel Customer Support
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello maxdd,
It took me a while to answer because I was experimenting with VTune, which I don't have much experience with.
I wasn't able to see my enclave functions in the Vtune report until I shortened the sampling interval using the sampling-interval knob.
$ vtune -collect sgx-hotspots -knob sampling-interval=.01 ./app
If using Vtune from the command line only, you can generate CSV versions of various reports by running:
$ vtune -report <reportname> -format=csv
Use vtune -help report to find available report names, e.g. callstacks, hotspots, top-down
To see data from the enclave only, I used:
$ vtune -report hotspots -format=csv -filter module=enclave.signed.so
I did not try attaching to a process, rather, I started the app using Vtune directly as shown above. I used a modified version of the SGX SDK sample, SampleEnclave, which runs very quickly.
Sincerely,
Jesus G.
Intel Customer Support
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for the reply.
I try it again but I still can't get information about the functions in enclave. Do I need to install some debug symbols?
Also, can you show me the results you get running on SampleEnclave?
root@h8:/opt/intel/oneapi/vtune/2021.9.0/bin64# ./vtune -collect sgx-hotspots -knob sampling-interval=.01 -target-pid 16984
vtune: Warning: To enable hardware event-based sampling, VTune Profiler has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.
Warning: sav for event CPU_CLK_UNHALTED.REF_TSC is too low, value reset to 100000
vtune: Collection started. To stop the collection, either press CTRL-C or enter from another console window: vtune -r /opt/intel/oneapi/vtune/2021.9.0/bin64/r013sgxhs -command stop.
^Cvtune: Collection detached.
vtune: Collection stopped.
vtune: Using result path `/opt/intel/oneapi/vtune/2021.9.0/bin64/r013sgxhs'
vtune: Executing actions 19 % Resolving information for `libsgx_urts.so.1.1.103
vtune: Warning: Cannot locate debugging information for file `/lib/modules/4.15.0-48-generic/kernel/arch/x86/kvm/kvm.ko'.
vtune: Executing actions 19 % Resolving information for `nf_nat'
vtune: Warning: Cannot locate debugging information for file `/lib/modules/4.15.0-48-generic/kernel/net/netfilter/nf_nat.ko'.
vtune: Warning: Cannot locate file `sep5.ko'.
vtune: Executing actions 20 % Resolving information for `tg3'
vtune: Warning: Cannot locate debugging information for file `/lib/modules/4.15.0-48-generic/kernel/drivers/net/ethernet/broadcom/tg3.ko'.
vtune: Executing actions 20 % Resolving information for `ip_tables'
vtune: Warning: Cannot locate debugging information for file `/lib/modules/4.15.0-48-generic/kernel/net/ipv4/netfilter/ip_tables.ko'.
vtune: Executing actions 20 % Resolving information for `nf_conntrack'
vtune: Warning: Cannot locate debugging information for file `/lib/modules/4.15.0-48-generic/kernel/net/netfilter/nf_conntrack.ko'.
vtune: Executing actions 22 % Resolving information for `vmlinux'
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
vtune: Executing actions 75 % Generating a report Elapsed Time: 2.107s
Precise Clockticks: 2,912,020,000
Total Thread Count: 8
Paused Time: 0s
Top Hotspots
Function Module Precise Clockticks
-------------------------- ------------------------ ------------------
__morestack libsgx_urts.so.1.1.103.3 2,455,040,000
virtio_dev_tx_split ovs-vswitchd 195,320,000
dp_netdev_input__ ovs-vswitchd 90,880,000
miniflow_extract ovs-vswitchd 20,680,000
netdev_dpdk_vhost_rxq_recv ovs-vswitchd 16,480,000
[Others] N/A 133,620,000
Effective Logical Core Utilization: 0.0% (0.000 out of 8)
Collection and Platform Info
User Name: root
Operating System: 4.15.0-48-generic DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
Computer Name: h8.trafgen.sdnnfv.beluga.pdl.cmu.local
Result Size: 79.4 MB
Collection start time: 18:04:29 08/12/2021 UTC
Collection stop time: 18:04:31 08/12/2021 UTC
Collector Type: Event-based sampling driver
CPU
Name: Intel(R) Processor code named Kabylake
Frequency: 3.792 GHz
Logical CPU Count: 8
Cache Allocation Technology
Level 2 capability: not detected
Level 3 capability: not detected
If you want to skip descriptions of detected performance issues in the report,
enter: vtune -report summary -report-knob show-issues=false -r <my_result_dir>.
Alternatively, you may view the report in the csv format: vtune -report
<report_name> -format=csv.
vtune: Executing actions 100 % done
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I will help you on this thread and delete the other two threads you opened on this topic.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello maxdd,
You do not need to install extra debug symbols. Compiling with SGX_DEBUG=1 is enough to compile your enclave code with the necessary debug symbols.
Try starting the application from VTune directly instead of attaching to a process. Below I show the outputs for the VTune run of SampleEnclave followed by the hotspots report in CSV format.
VTune Run
/opt/intel/sgxsdk/SampleCode/SampleEnclave$ vtune -collect sgx-hotspots -knob sampling-interval=.01 ./app
vtune: Warning: To enable hardware event-based sampling, VTune Profiler has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.
Warning: sav for event CPU_CLK_UNHALTED.REF_TSC is too low, value reset to 100000
vtune: Collection started. To stop the collection, either press CTRL-C or enter from another console window: vtune -r /opt/intel/sgxsdk/SampleCode/SampleEnclave/r002sgxhs -command stop.
Checksum(0x0x7fffa8d23540, 100) = 0xfffd4143
Info: executing thread synchronization, please wait...
Info: SampleEnclave successfully returned.
Enter a character before exit ...
vtune: Collection stopped.
vtune: Using result path `/opt/intel/sgxsdk/SampleCode/SampleEnclave/r002sgxhs'
vtune: Executing actions 19 % Resolving information for `isgx'
vtune: Warning: Cannot locate debugging information for file `/lib/modules/5.4.0-71-generic/kernel/drivers/intel/sgx/isgx.ko'.
vtune: Warning: Cannot read load addresses of sections from `/sys/module/isgx/sections'. This may affect the correctness of symbol resolution for `isgx'. Make sure this directory exists and all files in this directory have read permissions.
vtune: Warning: Cannot locate debugging information for file `/usr/lib/x86_64-linux-gnu/libprotobuf.so.17.0.0'.
vtune: Executing actions 19 % Resolving information for `e1000e'
vtune: Warning: Cannot locate debugging information for file `/lib/modules/5.4.0-71-generic/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko'.
vtune: Warning: Cannot read load addresses of sections from `/sys/module/e1000e/sections'. This may affect the correctness of symbol resolution for `e1000e'. Make sure this directory exists and all files in this directory have read permissions.
vtune: Executing actions 20 % Resolving information for `app'
vtune: Warning: Cannot locate file `vmlinux'.
vtune: Executing actions 20 % Resolving information for `libsgx_enclave_common.
vtune: Warning: Cannot locate debugging information for file `/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28'.
vtune: Executing actions 21 % Resolving information for `libsgx_enclave_common.
vtune: Warning: Cannot locate file `sep5.ko'.
vtune: Executing actions 21 % Resolving information for `libittnotify_collector
vtune: Warning: Cannot locate debugging information for file `/opt/intel/oneapi/vtune/2021.9.0/lib64/runtime/libittnotify_collector.so'.
vtune: Executing actions 21 % Resolving information for `xt_comment'
vtune: Warning: Cannot locate debugging information for file `/lib/modules/5.4.0-71-generic/kernel/net/netfilter/xt_comment.ko'.
vtune: Warning: Cannot read load addresses of sections from `/sys/module/xt_comment/sections'. This may affect the correctness of symbol resolution for `xt_comment'. Make sure this directory exists and all files in this directory have read permissions.
vtune: Executing actions 22 % Resolving information for `xt_comment'
vtune: Warning: Function and source-level analysis for the Linux kernel will not be possible since neither debug version of the kernel nor kernel symbol tables are found. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
vtune: Executing actions 75 % Generating a report Elapsed Time: 8.351s
Precise Clockticks: 677,040,000
Total Thread Count: 10
Paused Time: 0s
Top Hotspots
Function Module Precise Clockticks
---------------------------------- ------------------------ ------------------
[vmlinux] vmlinux 293,520,000
do_ocall enclave.signed.so 134,320,000
__morestack libsgx_urts.so.1.1.117.1 131,580,000
CTrustThreadPool::get_bound_thread libsgx_urts.so.1.1.117.1 12,360,000
__GI___pthread_mutex_lock libpthread-2.31.so 10,280,000
[Others] N/A 94,980,000
Effective Logical Core Utilization: 0.3% (0.023 out of
Collection and Platform Info
Application Command Line: ./app
User Name: el-pelon
Operating System: 5.4.0-71-generic DISTRIB_ID=Ubuntu DISTRIB_RELEASE=20.04 DISTRIB_CODENAME=focal DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"
Computer Name: del-microphone
Result Size: 29.8 MB
Collection start time: 20:25:41 08/12/2021 UTC
Collection stop time: 20:25:49 08/12/2021 UTC
Collector Type: Event-based sampling driver
CPU
Name: Intel(R) Processor code named Kabylake
Frequency: 3.096 GHz
Logical CPU Count: 8
Cache Allocation Technology
Level 2 capability: not detected
Level 3 capability: not detected
If you want to skip descriptions of detected performance issues in the report,
enter: vtune -report summary -report-knob show-issues=false -r <my_result_dir>.
Alternatively, you may view the report in the csv format: vtune -report
<report_name> -format=csv.
vtune: Executing actions 100 % done
Hotspots Report
/opt/intel/sgxsdk/SampleCode/SampleEnclave$ vtune -report hotspots -format=csv -filter module=enclave.signed.so
vtune: Using result path `/opt/intel/sgxsdk/SampleCode/SampleEnclave/r002sgxhs'
vtune: Executing actions 75 % Generating a report Function Precise Clockticks Module Function (Full) Source File Start Address
do_ocall 134320000 enclave.signed.so do_ocall [Unknown] 0x1ef94
restore_xregs 7960000 enclave.signed.so restore_xregs [Unknown] 0x1edc1
enclave_entry 6900000 enclave.signed.so enclave_entry [Unknown] 0x1ee89
sgx_thread_mutex_lock 4440000 enclave.signed.so sgx_thread_mutex_lock [Unknown] 0xd900
sgx_spin_lock 3680000 enclave.signed.so sgx_spin_lock [Unknown] 0x9fb0
asm_oret 3140000 enclave.signed.so asm_oret [Unknown] 0x1f0c4
do_oret 2200000 enclave.signed.so do_oret [Unknown] 0x8b00
save_xregs 1800000 enclave.signed.so save_xregs [Unknown] 0x1edf1
sgx_ocfree 1220000 enclave.signed.so sgx_ocfree [Unknown] 0x72e0
sgx_thread_set_untrusted_event_ocall 1100000 enclave.signed.so sgx_thread_set_untrusted_event_ocall Enclave_t.c 0x3159
sgx_thread_wait_untrusted_event_ocall 1100000 enclave.signed.so sgx_thread_wait_untrusted_event_ocall Enclave_t.c 0x30a7
ecall_consumer 1040000 enclave.signed.so ecall_consumer Thread.cpp 0x6eb4
sgx_thread_cond_wait 840000 enclave.signed.so sgx_thread_cond_wait [Unknown] 0xd6b0
sgx_thread_mutex_unlock 720000 enclave.signed.so sgx_thread_mutex_unlock [Unknown] 0xdb60
do_rdrand 640000 enclave.signed.so do_rdrand [Unknown] 0x1f1a5
__morestack 620000 enclave.signed.so __morestack [Unknown] 0x1f0b1
enter_enclave 580000 enclave.signed.so enter_enclave [Unknown] 0x1ec20
ecall_producer 480000 enclave.signed.so ecall_producer Thread.cpp 0x6d93
ecall_increase_counter 380000 enclave.signed.so ecall_increase_counter Thread.cpp 0x6d16
do_ecall 320000 enclave.signed.so do_ecall [Unknown] 0x8540
sgx_thread_setwait_untrusted_events_ocall 320000 enclave.signed.so sgx_thread_setwait_untrusted_events_ocall Enclave_t.c 0x320b
get_thread_data 300000 enclave.signed.so get_thread_data [Unknown] 0x1ee6d
sgx_ocall 300000 enclave.signed.so sgx_ocall [Unknown] 0x8a90
sgx_ocalloc 260000 enclave.signed.so sgx_ocalloc [Unknown] 0x71f0
update_ocall_lastsp 160000 enclave.signed.so update_ocall_lastsp [Unknown] 0x8ac0
sgx_is_within_enclave 140000 enclave.signed.so sgx_is_within_enclave [Unknown] 0x7140
sgx_thread_cond_signal 140000 enclave.signed.so sgx_thread_cond_signal [Unknown] 0xd840
sgx_is_enclave_crashed 120000 enclave.signed.so sgx_is_enclave_crashed [Unknown] 0x1ec00
sgx_thread_mutex_unlock_lazy 100000 enclave.signed.so sgx_thread_mutex_unlock_lazy [Unknown] 0xda70
_libc___memset 80000 enclave.signed.so _libc___memset [Unknown] 0x12340
get_enclave_state 80000 enclave.signed.so get_enclave_state [Unknown] 0x1ee2e
__memcpy 60000 enclave.signed.so __memcpy [Unknown] 0xd450
_ULx86_64_dwarf_extract_proc_info_from_fde 40000 enclave.signed.so _ULx86_64_dwarf_extract_proc_info_from_fde [Unknown] 0x1dcb0
do_init_thread 40000 enclave.signed.so do_init_thread [Unknown] 0x8360
sgx_ecall_consumer 40000 enclave.signed.so sgx_ecall_consumer Enclave_t.c 0x27da
sgx_read_rand 40000 enclave.signed.so sgx_read_rand [Unknown] 0x7340
sgx_spin_unlock 40000 enclave.signed.so sgx_spin_unlock [Unknown] 0xa020
_ULx86_64_access_reg 20000 enclave.signed.so _ULx86_64_access_reg [Unknown] 0x1c090
_ULx86_64_dwarf_callback 20000 enclave.signed.so _ULx86_64_dwarf_callback [Unknown] 0x1d2e0
_ULx86_64_dwarf_read_encoded_pointer 20000 enclave.signed.so _ULx86_64_dwarf_read_encoded_pointer [Unknown] 0x19680
__vfprintf 20000 enclave.signed.so __vfprintf [Unknown] 0xed80
create_state_record_for.part.0 20000 enclave.signed.so create_state_record_for.part.0 [Unknown] 0x185f0
dlfree 20000 enclave.signed.so dlfree [Unknown] 0xc3b0
do_init_enclave 20000 enclave.signed.so do_init_enclave [Unknown] 0x1eb00
do_relocs 20000 enclave.signed.so do_relocs [Unknown] 0x93b0
ecall_map 20000 enclave.signed.so ecall_map Libcxx.cpp 0x45c5
elf_tls_info 20000 enclave.signed.so elf_tls_info [Unknown] 0x9770
mbrtowc 20000 enclave.signed.so mbrtowc [Unknown] 0x15070
operator new[] 20000 enclave.signed.so operator new[](unsigned long) [Unknown] 0x1ac50
run_cfi_program 20000 enclave.signed.so run_cfi_program [Unknown] 0x17b50
sgx_ecall_pointer_in 20000 enclave.signed.so sgx_ecall_pointer_in Enclave_t.c 0x1645
std::__1::__tree<std::__1::__value_type<char, int>, std::__1::__map_value_compare<char, std::__1::__value_type<char, int>, std::__1::less<char>, (bool)1>, std::__1::allocator<std::__1::__value_type<char, int>>>::__construct_node<std::__1::pair<char const, int>> 20000 enclave.signed.so std::__1::__tree<std::__1::__value_type<char, int>, std::__1::__map_value_compare<char, std::__1::__value_type<char, int>, std::__1::less<char>, (bool)1>, std::__1::allocator<std::__1::__value_type<char, int>>>::__construct_node<std::__1::pair<char const, int>>(std::__1::pair<char const, int>&&) __tree 0x642c
std::__1::__tree<std::__1::__value_type<char, int>, std::__1::__map_value_compare<char, std::__1::__value_type<char, int>, std::__1::less<char>, (bool)1>, std::__1::allocator<std::__1::__value_type<char, int>>>::__root_ptr 20000 enclave.signed.so std::__1::__tree<std::__1::__value_type<char, int>, std::__1::__map_value_compare<char, std::__1::__value_type<char, int>, std::__1::less<char>, (bool)1>, std::__1::allocator<std::__1::__value_type<char, int>>>::__root_ptr(void) const __tree 0x67b8
std::exception::exception 20000 enclave.signed.so std::exception::exception(void) [Unknown] 0x1ab40
trts_ecall 20000 enclave.signed.so trts_ecall(unsigned int, void*) [Unknown] 0x8130
vtune: Executing actions 100 % done
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Maxdd,
I hope the information I provided was helpful to you. Since we haven't heard from you, Intel will stop monitoring this thread. If you need further assistance, please open a new thread.
Sincerely,
Jesus G.
Intel Customer Support
![](/skins/images/D3C0B914909A6564BBB97F4AD1ED1973/responsive_peak/images/icon_anonymous_message.png)
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page