Intel® Software Guard Extensions (Intel® SGX)
Use hardware-based isolation and memory encryption to provide more code protection in your solutions.

intel vtune to profile sgx application

maxdd
Beginner
659 Views

I want to use intel vtune to profile sgx application. The vtune version is 2021.9.0. I execute following commands as root:

 

export INTEL_LIBITTNOTIFY64=/opt/intel/oneapi/vtune/latest/lib64/runtime/libittnotify_collector.so
./vtune -collect sgx-hotspots -collect-with runss -target-pid 31127


The output is:

 

vtune: Warning: To enable hardware event-based sampling, VTune Profiler has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.
vtune: Collection started. To stop the collection, either press CTRL-C or enter from another console window: vtune -r /opt/intel/oneapi/vtune/2021.9.0/bin64/r005sgxhs -command stop.
^Cvtune: Collection detached.
vtune: Collection stopped.
vtune: Using result path `/opt/intel/oneapi/vtune/2021.9.0/bin64/r005sgxhs'
vtune: Executing actions 21 % Resolving information for `vmlinux'
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
vtune: Executing actions 75 % Generating a report Elapsed Time: 3.091s
Precise Clockticks: 5,224,007,836
Total Thread Count: 4
Paused Time: 0s

Top Hotspots
Function Module Precise Clockticks
------------------------- ------------------------ ------------------
__morestack libsgx_urts.so.1.1.103.3 3,324,004,986
dp_netdev_input__ ovs-vswitchd 120,000,180
_mm256_loadu_si256 librte_vhost.so.20.0 106,000,159
restore_xregs libsgx_urts.so.1.1.103.3 104,000,156
__GI___pthread_mutex_lock libpthread-2.27.so 102,000,153
[Others] N/A 1,468,002,202
Effective Logical Core Utilization: 0.0% (0.000 out of 8)
Collection and Platform Info
User Name: root
Operating System: 4.15.0-48-generic DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
Computer Name: h8.trafgen.sdnnfv.beluga.pdl.cmu.local
Result Size: 7.5 MB
Collection start time: 16:34:19 04/12/2021 UTC
Collection stop time: 16:34:22 04/12/2021 UTC
Collector Type: Event-based sampling driver
CPU
Name: Intel(R) Processor code named Kabylake
Frequency: 3.792 GHz
Logical CPU Count: 8
Cache Allocation Technology
Level 2 capability: not detected
Level 3 capability: not detected

If you want to skip descriptions of detected performance issues in the report,
enter: vtune -report summary -report-knob show-issues=false -r <my_result_dir>.
Alternatively, you may view the report in the csv format: vtune -report
<report_name> -format=csv.
vtune: Executing actions 100 % done


It does not show detailed results about how functions are executed in enclave. It only shows '__morestack' and 'restore_xregs'. Is there a way to further decompose the execution of those functions?


When compiling the SGX application, I am using SGX_DEBUG mode.

 

This is the config I use for enclave.

<EnclaveConfiguration>
  <ProdID>0</ProdID>
  <ISVSVN>0</ISVSVN>
  <StackMaxSize>262144</StackMaxSize>
  <HeapMaxSize>10240000</HeapMaxSize>
  <TCSNum>10</TCSNum>
  <TCSPolicy>1</TCSPolicy>
  <!-- Recommend changing 'DisableDebug' to 1 to make the enclave undebuggable for enclave release -->
  <DisableDebug>0</DisableDebug>
  <MiscSelect>0</MiscSelect>
  <MiscMask>0xFFFFFFFF</MiscMask>
</EnclaveConfiguration>

 

0 Kudos
1 Solution
JesusG_Intel
Moderator
654 Views

Hello maxdd,


It took me a while to answer because I was experimenting with VTune, which I don't have much experience with.


I wasn't able to see my enclave functions in the Vtune report until I shortened the sampling interval using the sampling-interval knob.

 

$ vtune -collect sgx-hotspots -knob sampling-interval=.01 ./app

 

If using Vtune from the command line only, you can generate CSV versions of various reports by running:

$ vtune -report <reportname> -format=csv

 

Use vtune -help report to find available report names, e.g. callstacks, hotspots, top-down

 

To see data from the enclave only, I used:

$ vtune -report hotspots -format=csv -filter module=enclave.signed.so

 

I did not try attaching to a process, rather, I started the app using Vtune directly as shown above. I used a modified version of the SGX SDK sample, SampleEnclave, which runs very quickly.


Sincerely,

Jesus G.

Intel Customer Support


View solution in original post

5 Replies
JesusG_Intel
Moderator
655 Views

Hello maxdd,


It took me a while to answer because I was experimenting with VTune, which I don't have much experience with.


I wasn't able to see my enclave functions in the Vtune report until I shortened the sampling interval using the sampling-interval knob.

 

$ vtune -collect sgx-hotspots -knob sampling-interval=.01 ./app

 

If using Vtune from the command line only, you can generate CSV versions of various reports by running:

$ vtune -report <reportname> -format=csv

 

Use vtune -help report to find available report names, e.g. callstacks, hotspots, top-down

 

To see data from the enclave only, I used:

$ vtune -report hotspots -format=csv -filter module=enclave.signed.so

 

I did not try attaching to a process, rather, I started the app using Vtune directly as shown above. I used a modified version of the SGX SDK sample, SampleEnclave, which runs very quickly.


Sincerely,

Jesus G.

Intel Customer Support


maxdd
Beginner
630 Views

Thanks for the reply.

I try it again but I still can't get information about the functions in enclave. Do I need to install some debug symbols?

Also, can you show me the results you get running on SampleEnclave?

 

root@h8:/opt/intel/oneapi/vtune/2021.9.0/bin64# ./vtune -collect sgx-hotspots -knob sampling-interval=.01 -target-pid 16984
vtune: Warning: To enable hardware event-based sampling, VTune Profiler has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.
Warning: sav for event CPU_CLK_UNHALTED.REF_TSC is too low, value reset to 100000
vtune: Collection started. To stop the collection, either press CTRL-C or enter from another console window: vtune -r /opt/intel/oneapi/vtune/2021.9.0/bin64/r013sgxhs -command stop.
^Cvtune: Collection detached.
vtune: Collection stopped.
vtune: Using result path `/opt/intel/oneapi/vtune/2021.9.0/bin64/r013sgxhs'
vtune: Executing actions 19 % Resolving information for `libsgx_urts.so.1.1.103
vtune: Warning: Cannot locate debugging information for file `/lib/modules/4.15.0-48-generic/kernel/arch/x86/kvm/kvm.ko'.
vtune: Executing actions 19 % Resolving information for `nf_nat'
vtune: Warning: Cannot locate debugging information for file `/lib/modules/4.15.0-48-generic/kernel/net/netfilter/nf_nat.ko'.
vtune: Warning: Cannot locate file `sep5.ko'.
vtune: Executing actions 20 % Resolving information for `tg3'
vtune: Warning: Cannot locate debugging information for file `/lib/modules/4.15.0-48-generic/kernel/drivers/net/ethernet/broadcom/tg3.ko'.
vtune: Executing actions 20 % Resolving information for `ip_tables'
vtune: Warning: Cannot locate debugging information for file `/lib/modules/4.15.0-48-generic/kernel/net/ipv4/netfilter/ip_tables.ko'.
vtune: Executing actions 20 % Resolving information for `nf_conntrack'
vtune: Warning: Cannot locate debugging information for file `/lib/modules/4.15.0-48-generic/kernel/net/netfilter/nf_conntrack.ko'.
vtune: Executing actions 22 % Resolving information for `vmlinux'
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
vtune: Executing actions 75 % Generating a report                              Elapsed Time: 2.107s
    Precise Clockticks: 2,912,020,000
    Total Thread Count: 8
    Paused Time: 0s

Top Hotspots
Function                    Module                    Precise Clockticks
--------------------------  ------------------------  ------------------
__morestack                 libsgx_urts.so.1.1.103.3       2,455,040,000
virtio_dev_tx_split         ovs-vswitchd                     195,320,000
dp_netdev_input__           ovs-vswitchd                      90,880,000
miniflow_extract            ovs-vswitchd                      20,680,000
netdev_dpdk_vhost_rxq_recv  ovs-vswitchd                      16,480,000
[Others]                    N/A                              133,620,000
    Effective Logical Core Utilization: 0.0% (0.000 out of 8)
Collection and Platform Info
    User Name: root
    Operating System: 4.15.0-48-generic DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
    Computer Name: h8.trafgen.sdnnfv.beluga.pdl.cmu.local
    Result Size: 79.4 MB
    Collection start time: 18:04:29 08/12/2021 UTC
    Collection stop time: 18:04:31 08/12/2021 UTC
    Collector Type: Event-based sampling driver
    CPU
        Name: Intel(R) Processor code named Kabylake
        Frequency: 3.792 GHz
        Logical CPU Count: 8
        Cache Allocation Technology
            Level 2 capability: not detected
            Level 3 capability: not detected

If you want to skip descriptions of detected performance issues in the report,
enter: vtune -report summary -report-knob show-issues=false -r <my_result_dir>.
Alternatively, you may view the report in the csv format: vtune -report
<report_name> -format=csv.
vtune: Executing actions 100 % done

 

JesusG_Intel
Moderator
652 Views

I will help you on this thread and delete the other two threads you opened on this topic.

 

JesusG_Intel
Moderator
622 Views

Hello maxdd,

 

You do not need to install extra debug symbols. Compiling with SGX_DEBUG=1 is enough to compile your enclave code with the necessary debug symbols.

Try starting the application from VTune directly instead of attaching to a process. Below I show the outputs for the VTune run of SampleEnclave followed by the hotspots report in CSV format.

 

VTune Run

 

/opt/intel/sgxsdk/SampleCode/SampleEnclave$ vtune -collect sgx-hotspots -knob sampling-interval=.01 ./app

vtune: Warning: To enable hardware event-based sampling, VTune Profiler has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.

Warning: sav for event CPU_CLK_UNHALTED.REF_TSC is too low, value reset to 100000

vtune: Collection started. To stop the collection, either press CTRL-C or enter from another console window: vtune -r /opt/intel/sgxsdk/SampleCode/SampleEnclave/r002sgxhs -command stop.

Checksum(0x0x7fffa8d23540, 100) = 0xfffd4143

Info: executing thread synchronization, please wait...

Info: SampleEnclave successfully returned.

Enter a character before exit ...

 

 

vtune: Collection stopped.

vtune: Using result path `/opt/intel/sgxsdk/SampleCode/SampleEnclave/r002sgxhs'

vtune: Executing actions 19 % Resolving information for `isgx'

vtune: Warning: Cannot locate debugging information for file `/lib/modules/5.4.0-71-generic/kernel/drivers/intel/sgx/isgx.ko'.

vtune: Warning: Cannot read load addresses of sections from `/sys/module/isgx/sections'. This may affect the correctness of symbol resolution for `isgx'. Make sure this directory exists and all files in this directory have read permissions.

vtune: Warning: Cannot locate debugging information for file `/usr/lib/x86_64-linux-gnu/libprotobuf.so.17.0.0'.

vtune: Executing actions 19 % Resolving information for `e1000e'

vtune: Warning: Cannot locate debugging information for file `/lib/modules/5.4.0-71-generic/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko'.

vtune: Warning: Cannot read load addresses of sections from `/sys/module/e1000e/sections'. This may affect the correctness of symbol resolution for `e1000e'. Make sure this directory exists and all files in this directory have read permissions.

vtune: Executing actions 20 % Resolving information for `app'

vtune: Warning: Cannot locate file `vmlinux'.

vtune: Executing actions 20 % Resolving information for `libsgx_enclave_common.

vtune: Warning: Cannot locate debugging information for file `/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28'.

vtune: Executing actions 21 % Resolving information for `libsgx_enclave_common.

vtune: Warning: Cannot locate file `sep5.ko'.

vtune: Executing actions 21 % Resolving information for `libittnotify_collector

vtune: Warning: Cannot locate debugging information for file `/opt/intel/oneapi/vtune/2021.9.0/lib64/runtime/libittnotify_collector.so'.

vtune: Executing actions 21 % Resolving information for `xt_comment'

vtune: Warning: Cannot locate debugging information for file `/lib/modules/5.4.0-71-generic/kernel/net/netfilter/xt_comment.ko'.

vtune: Warning: Cannot read load addresses of sections from `/sys/module/xt_comment/sections'. This may affect the correctness of symbol resolution for `xt_comment'. Make sure this directory exists and all files in this directory have read permissions.

vtune: Executing actions 22 % Resolving information for `xt_comment'

vtune: Warning: Function and source-level analysis for the Linux kernel will not be possible since neither debug version of the kernel nor kernel symbol tables are found. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.

vtune: Executing actions 75 % Generating a report                             Elapsed Time: 8.351s

   Precise Clockticks: 677,040,000

   Total Thread Count: 10

   Paused Time: 0s

 

Top Hotspots

Function                           Module                   Precise Clockticks

---------------------------------- ------------------------ ------------------

[vmlinux]                          vmlinux                         293,520,000

do_ocall                           enclave.signed.so               134,320,000

__morestack                        libsgx_urts.so.1.1.117.1        131,580,000

CTrustThreadPool::get_bound_thread libsgx_urts.so.1.1.117.1         12,360,000

__GI___pthread_mutex_lock          libpthread-2.31.so               10,280,000

[Others]                           N/A                              94,980,000

   Effective Logical Core Utilization: 0.3% (0.023 out of

Collection and Platform Info

   Application Command Line: ./app

   User Name: el-pelon

   Operating System: 5.4.0-71-generic DISTRIB_ID=Ubuntu DISTRIB_RELEASE=20.04 DISTRIB_CODENAME=focal DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"

   Computer Name: del-microphone

   Result Size: 29.8 MB

   Collection start time: 20:25:41 08/12/2021 UTC

   Collection stop time: 20:25:49 08/12/2021 UTC

   Collector Type: Event-based sampling driver

   CPU

       Name: Intel(R) Processor code named Kabylake

       Frequency: 3.096 GHz

       Logical CPU Count: 8

       Cache Allocation Technology

           Level 2 capability: not detected

           Level 3 capability: not detected

 

If you want to skip descriptions of detected performance issues in the report,

enter: vtune -report summary -report-knob show-issues=false -r <my_result_dir>.

Alternatively, you may view the report in the csv format: vtune -report

<report_name> -format=csv.

vtune: Executing actions 100 % done

 

 

Hotspots Report

 

/opt/intel/sgxsdk/SampleCode/SampleEnclave$ vtune -report hotspots -format=csv -filter module=enclave.signed.so

vtune: Using result path `/opt/intel/sgxsdk/SampleCode/SampleEnclave/r002sgxhs'

vtune: Executing actions 75 % Generating a report                             Function Precise Clockticks     Module Function (Full) Source File    Start Address

do_ocall       134320000      enclave.signed.so      do_ocall       [Unknown]      0x1ef94

restore_xregs  7960000 enclave.signed.so      restore_xregs  [Unknown]      0x1edc1

enclave_entry  6900000 enclave.signed.so      enclave_entry  [Unknown]      0x1ee89

sgx_thread_mutex_lock  4440000 enclave.signed.so      sgx_thread_mutex_lock  [Unknown]      0xd900

sgx_spin_lock  3680000 enclave.signed.so      sgx_spin_lock  [Unknown]      0x9fb0

asm_oret       3140000 enclave.signed.so      asm_oret       [Unknown]      0x1f0c4

do_oret 2200000 enclave.signed.so      do_oret [Unknown]      0x8b00

save_xregs     1800000 enclave.signed.so      save_xregs     [Unknown]      0x1edf1

sgx_ocfree     1220000 enclave.signed.so      sgx_ocfree     [Unknown]      0x72e0

sgx_thread_set_untrusted_event_ocall   1100000 enclave.signed.so      sgx_thread_set_untrusted_event_ocall   Enclave_t.c    0x3159

sgx_thread_wait_untrusted_event_ocall  1100000 enclave.signed.so      sgx_thread_wait_untrusted_event_ocall  Enclave_t.c    0x30a7

ecall_consumer 1040000 enclave.signed.so      ecall_consumer Thread.cpp     0x6eb4

sgx_thread_cond_wait   840000 enclave.signed.so      sgx_thread_cond_wait   [Unknown]      0xd6b0

sgx_thread_mutex_unlock 720000 enclave.signed.so      sgx_thread_mutex_unlock [Unknown]      0xdb60

do_rdrand      640000 enclave.signed.so      do_rdrand      [Unknown]      0x1f1a5

__morestack    620000 enclave.signed.so      __morestack    [Unknown]      0x1f0b1

enter_enclave  580000 enclave.signed.so      enter_enclave  [Unknown]      0x1ec20

ecall_producer 480000 enclave.signed.so      ecall_producer Thread.cpp     0x6d93

ecall_increase_counter 380000 enclave.signed.so      ecall_increase_counter Thread.cpp     0x6d16

do_ecall       320000 enclave.signed.so      do_ecall       [Unknown]      0x8540

sgx_thread_setwait_untrusted_events_ocall      320000 enclave.signed.so      sgx_thread_setwait_untrusted_events_ocall      Enclave_t.c    0x320b

get_thread_data 300000 enclave.signed.so      get_thread_data [Unknown]      0x1ee6d

sgx_ocall      300000 enclave.signed.so      sgx_ocall      [Unknown]      0x8a90

sgx_ocalloc    260000 enclave.signed.so      sgx_ocalloc    [Unknown]      0x71f0

update_ocall_lastsp    160000 enclave.signed.so      update_ocall_lastsp    [Unknown]      0x8ac0

sgx_is_within_enclave  140000 enclave.signed.so      sgx_is_within_enclave  [Unknown]      0x7140

sgx_thread_cond_signal 140000 enclave.signed.so      sgx_thread_cond_signal [Unknown]      0xd840

sgx_is_enclave_crashed 120000 enclave.signed.so      sgx_is_enclave_crashed [Unknown]      0x1ec00

sgx_thread_mutex_unlock_lazy   100000 enclave.signed.so      sgx_thread_mutex_unlock_lazy   [Unknown]      0xda70

_libc___memset 80000  enclave.signed.so      _libc___memset [Unknown]      0x12340

get_enclave_state      80000  enclave.signed.so      get_enclave_state      [Unknown]      0x1ee2e

__memcpy       60000  enclave.signed.so      __memcpy       [Unknown]      0xd450

_ULx86_64_dwarf_extract_proc_info_from_fde     40000  enclave.signed.so      _ULx86_64_dwarf_extract_proc_info_from_fde     [Unknown]      0x1dcb0

do_init_thread 40000  enclave.signed.so      do_init_thread [Unknown]      0x8360

sgx_ecall_consumer     40000  enclave.signed.so      sgx_ecall_consumer     Enclave_t.c    0x27da

sgx_read_rand  40000  enclave.signed.so      sgx_read_rand  [Unknown]      0x7340

sgx_spin_unlock 40000  enclave.signed.so      sgx_spin_unlock [Unknown]      0xa020

_ULx86_64_access_reg   20000  enclave.signed.so      _ULx86_64_access_reg   [Unknown]      0x1c090

_ULx86_64_dwarf_callback       20000  enclave.signed.so      _ULx86_64_dwarf_callback       [Unknown]      0x1d2e0

_ULx86_64_dwarf_read_encoded_pointer   20000  enclave.signed.so      _ULx86_64_dwarf_read_encoded_pointer   [Unknown]      0x19680

__vfprintf     20000  enclave.signed.so      __vfprintf     [Unknown]      0xed80

create_state_record_for.part.0 20000  enclave.signed.so      create_state_record_for.part.0 [Unknown]      0x185f0

dlfree 20000  enclave.signed.so      dlfree [Unknown]      0xc3b0

do_init_enclave 20000  enclave.signed.so      do_init_enclave [Unknown]      0x1eb00

do_relocs      20000  enclave.signed.so      do_relocs      [Unknown]      0x93b0

ecall_map      20000  enclave.signed.so      ecall_map      Libcxx.cpp     0x45c5

elf_tls_info   20000  enclave.signed.so      elf_tls_info   [Unknown]      0x9770

mbrtowc 20000  enclave.signed.so      mbrtowc [Unknown]      0x15070

operator new[] 20000  enclave.signed.so      operator new[](unsigned long)  [Unknown]      0x1ac50

run_cfi_program 20000  enclave.signed.so      run_cfi_program [Unknown]      0x17b50

sgx_ecall_pointer_in   20000  enclave.signed.so      sgx_ecall_pointer_in   Enclave_t.c    0x1645

std::__1::__tree<std::__1::__value_type<char, int>, std::__1::__map_value_compare<char, std::__1::__value_type<char, int>, std::__1::less<char>, (bool)1>, std::__1::allocator<std::__1::__value_type<char, int>>>::__construct_node<std::__1::pair<char const, int>>       20000  enclave.signed.so      std::__1::__tree<std::__1::__value_type<char, int>, std::__1::__map_value_compare<char, std::__1::__value_type<char, int>, std::__1::less<char>, (bool)1>, std::__1::allocator<std::__1::__value_type<char, int>>>::__construct_node<std::__1::pair<char const, int>>(std::__1::pair<char const, int>&&)    __tree 0x642c

std::__1::__tree<std::__1::__value_type<char, int>, std::__1::__map_value_compare<char, std::__1::__value_type<char, int>, std::__1::less<char>, (bool)1>, std::__1::allocator<std::__1::__value_type<char, int>>>::__root_ptr 20000  enclave.signed.so   std::__1::__tree<std::__1::__value_type<char, int>, std::__1::__map_value_compare<char, std::__1::__value_type<char, int>, std::__1::less<char>, (bool)1>, std::__1::allocator<std::__1::__value_type<char, int>>>::__root_ptr(void) const  __tree 0x67b8

std::exception::exception      20000  enclave.signed.so      std::exception::exception(void) [Unknown]      0x1ab40

trts_ecall     20000  enclave.signed.so      trts_ecall(unsigned int, void*) [Unknown]      0x8130

vtune: Executing actions 100 % done

 

JesusG_Intel
Moderator
566 Views

Hello Maxdd,


I hope the information I provided was helpful to you. Since we haven't heard from you, Intel will stop monitoring this thread. If you need further assistance, please open a new thread.


Sincerely,

Jesus G.

Intel Customer Support


Reply