[check_symbol_table elfparser.cpp:253] symbol 'cblas_gemm_s16s16s32' is undefined

[qianbao]

Hi, I want to use intel mkl in my enclave.

I have installed intel mkl in my Dockerfile and added

-I/opt/intel/oneapi/mkl/latest/include to Enclave_Include_Paths
-L/opt/intel/oneapi/mkl/latest/lib -lmkl_rt to Enclave_Link_Flags between --start-group and --end-group flags

I am getting the errors

CXX <= App/App.cpp
LINK => app
CXX <= Enclave/TrustedLibrary/Libcxx.cpp
LINK => enclave.so

[check_symbol_table elfparser.cpp:253] symbol 'cblas_gemm_s16s16s32' is undefined
Symbol table incorrect
The input enclave file is not correct.
Error happened while signing the enclave.
make[1]: *** [Makefile:288: enclave.signed.so] Error 255
make[1]: Leaving directory '/home/admin/dev/linux-sgx/SampleCode/Cxx11SGXDemo_bak'
make: *** [Makefile:203: all] Error 2

I was expecting errors since it is not ported for sgx but the function it is talking about is just the libmkl_rt api function. Any advice is appreciated.

if I use static library

add -I/opt/intel/oneapi/mkl/latest/include to Enclave_Include_Paths

add -L/opt/intel/oneapi/mkl/latest/lib -lmkl_gf_lp64 -lmkl_core -lmkl_gnu_thread to Enclave_Link_Flags between --start-group and --end-group flags

then, I got

CXX <= App/App.cpp
LINK => app
CXX <= Enclave/TrustedLibrary/Libcxx.cpp
/usr/bin/ld: enclave.so: local symbol `cos' in /opt/intel/sgxsdk/lib64/libsgx_tstdc.a(cos_iface_c99.o) is referenced by DSO
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:284: enclave.so] Error 1
make[1]: Leaving directory '/home/admin/dev/linux-sgx/SampleCode/Cxx11SGXDemo_bak'
make: *** [Makefile:203: all] Error 2

[qianbao]

I usr the SampleCOde and modify the Makfile

######## SGX SDK Settings ########

SGX_SDK ?= /opt/intel/sgxsdk
SGX_MODE ?= HW
SGX_ARCH ?= x64
SGX_DEBUG ?= 1

ifeq ($(shell getconf LONG_BIT), 32)
SGX_ARCH := x86
else ifeq ($(findstring -m32, $(CXXFLAGS)), -m32)
SGX_ARCH := x86
endif

ifeq ($(SGX_MODE), HYPER)
SGX_SIGN = sgx_sign_hyper
else
SGX_SIGN = sgx_sign
endif

ifeq ($(SGX_ARCH), x86)
SGX_COMMON_FLAGS := -m32
SGX_LIBRARY_PATH := $(SGX_SDK)/lib
SGX_ENCLAVE_SIGNER := $(SGX_SDK)/bin/x86/$(SGX_SIGN)
SGX_EDGER8R := $(SGX_SDK)/bin/x86/sgx_edger8r
else
SGX_COMMON_FLAGS := -m64
SGX_LIBRARY_PATH := $(SGX_SDK)/lib64
SGX_ENCLAVE_SIGNER := $(SGX_SDK)/bin/x64/$(SGX_SIGN)
SGX_EDGER8R := $(SGX_SDK)/bin/x64/sgx_edger8r
endif

ifeq ($(SGX_DEBUG), 1)
ifeq ($(SGX_PRERELEASE), 1)
$(error Cannot set SGX_DEBUG and SGX_PRERELEASE at the same time!!)
endif
endif

ifeq ($(SGX_DEBUG), 1)
SGX_COMMON_FLAGS += -O0 -g
else

SGX_COMMON_FLAGS += -O2
endif

SGX_COMMON_FLAGS += -Wall -Wextra -Winit-self -Wpointer-arith -Wreturn-type \
-Waddress -Wsequence-point -Wformat-security \
-Wmissing-include-dirs -Wfloat-equal -Wundef -Wshadow \
-Wcast-align -Wcast-qual -Wconversion -Wredundant-decls
SGX_COMMON_CFLAGS := $(SGX_COMMON_FLAGS) -Wjump-misses-init -Wstrict-prototypes -Wunsuffixed-float-constants
SGX_COMMON_CXXFLAGS := $(SGX_COMMON_FLAGS) -Wnon-virtual-dtor -std=c++11

######## App Settings ########

ifeq ($(SGX_MODE), SIM)
Urts_Library_Name := sgx_urts_sim
else ifeq ($(SGX_MODE), HYPER)
Urts_Library_Name := sgx_urts_hyper
else
Urts_Library_Name := sgx_urts
endif

App_Cpp_Files := App/App.cpp $(wildcard App/TrustedLibrary/*.cpp)
App_Include_Paths := -IApp -I$(SGX_SDK)/include -I/opt/intel/oneapi/mkl/latest/include/

App_C_Flags := -fPIC -Wno-attributes $(App_Include_Paths)

# Three configuration modes - Debug, prerelease, release
# Debug - Macro DEBUG enabled.
# Prerelease - Macro NDEBUG and EDEBUG enabled.
# Release - Macro NDEBUG enabled.
ifeq ($(SGX_DEBUG), 1)
App_C_Flags += -DDEBUG -UNDEBUG -UEDEBUG
else ifeq ($(SGX_PRERELEASE), 1)
App_C_Flags += -DNDEBUG -DEDEBUG -UDEBUG
else
App_C_Flags += -DNDEBUG -UEDEBUG -UDEBUG
endif

App_Cpp_Flags := $(App_C_Flags)
App_Link_Flags := -L$(SGX_LIBRARY_PATH) -l$(Urts_Library_Name) -lpthread -lmkl_rt

App_Cpp_Objects := $(App_Cpp_Files:.cpp=.o)

App_Name := app

######## Enclave Settings ########

Enclave_Version_Script := Enclave/Enclave_debug.lds
ifeq ($(SGX_MODE), HW)
ifneq ($(SGX_DEBUG), 1)
ifneq ($(SGX_PRERELEASE), 1)
# Choose to use 'Enclave.lds' for HW release mode
Enclave_Version_Script = Enclave/Enclave.lds
endif
endif
endif

ifeq ($(SGX_MODE), SIM)
Trts_Library_Name := sgx_trts_sim
Service_Library_Name := sgx_tservice_sim
else ifeq ($(SGX_MODE), HYPER)
Trts_Library_Name := sgx_trts_hyper
Service_Library_Name := sgx_tservice_hyper
else
Trts_Library_Name := sgx_trts
Service_Library_Name := sgx_tservice
endif
Crypto_Library_Name := sgx_tcrypto

Enclave_Cpp_Files := Enclave/Enclave.cpp $(wildcard Enclave/TrustedLibrary/*.cpp)
Enclave_Include_Paths := -IEnclave -I$(SGX_SDK)/include -I$(SGX_SDK)/include/libcxx -I$(SGX_SDK)/include/tlibc -I/opt/intel/oneapi/mkl/latest/include/

Enclave_C_Flags := -nostdinc -fvisibility=hidden -fpie -fstack-protector $(Enclave_Include_Paths)
Enclave_Cpp_Flags := $(Enclave_C_Flags) -nostdinc++

# Enable the security flags
Enclave_Security_Link_Flags := -Wl,-z,relro,-z,now,-z,noexecstack

# To generate a proper enclave, it is recommended to follow below guideline to link the trusted libraries:
# 1. Link sgx_trts with the `--whole-archive' and `--no-whole-archive' options,
# so that the whole content of trts is included in the enclave.
# 2. For other libraries, you just need to pull the required symbols.
# Use `--start-group' and `--end-group' to link these libraries.
# Do NOT move the libraries linked with `--start-group' and `--end-group' within `--whole-archive' and `--no-whole-archive' options.
# Otherwise, you may get some undesirable errors.
Enclave_Link_Flags := $(Enclave_Security_Link_Flags) \
-Wl,--no-undefined -nostdlib -nodefaultlibs -nostartfiles -L$(SGX_LIBRARY_PATH) -L/opt/intel/oneapi/mkl/latest/lib \
-Wl,--whole-archive -l$(Trts_Library_Name) -Wl,--no-whole-archive \
-Wl,--start-group -lmkl_gf_lp64 -lmkl_core -lmkl_gnu_thread -lsgx_tstdc -lsgx_tcxx -l$(Crypto_Library_Name) -l$(Service_Library_Name) -Wl,--end-group \
-Wl,-Bstatic -Wl,-Bsymbolic -Wl,--no-undefined \

-Wl,-pie,-eenclave_entry -Wl,--export-dynamic \
-Wl,--defsym,__ImageBase=0 \
-Wl,--version-script=$(Enclave_Version_Script)

Enclave_Cpp_Objects := $(Enclave_Cpp_Files:.cpp=.o)

Enclave_Name := enclave.so
Signed_Enclave_Name := enclave.signed.so
Enclave_Config_File := Enclave/Enclave.config.xml

ifeq ($(SGX_MODE), HW)
ifeq ($(SGX_DEBUG), 1)
Build_Mode = HW_DEBUG
else ifeq ($(SGX_PRERELEASE), 1)
Build_Mode = HW_PRERELEASE
else
Build_Mode = HW_RELEASE
endif
else ifeq ($(SGX_MODE), HYPER)
ifeq ($(SGX_DEBUG), 1)
Build_Mode = HYPER_DEBUG
else ifeq ($(SGX_PRERELEASE), 1)
Build_Mode = HYPER_PRERELEASE
else
Build_Mode = HYPER_RELEASE
endif
else
ifeq ($(SGX_DEBUG), 1)
Build_Mode = SIM_DEBUG
else ifeq ($(SGX_PRERELEASE), 1)
Build_Mode = SIM_PRERELEASE
else
Build_Mode = SIM_RELEASE
endif
endif

LIB_DIR=/opt/intel/oneapi/mkl/latest/lib/
SO_FILES = $(wildcard $(LIB_DIR)/libmkl_rt.so)
A_FILES = $(wildcard $(LIB_DIR)/*.a)

.PHONY: all run target
all: .config_$(Build_Mode)_$(SGX_ARCH)
@$(MAKE) target

ifeq ($(Build_Mode), HW_RELEASE)

arget: $(App_Name) $(Enclave_Name)
@Echo "The project has been built in release hardware mode."
@Echo "Please sign the $(Enclave_Name) first with your signing key before you run the $(App_Name) to launch and access the enclave."
@Echo "To sign the enclave use the command:"
@Echo " $(SGX_ENCLAVE_SIGNER) sign -key <your key> -enclave $(Enclave_Name) -out <$(Signed_Enclave_Name)> -config $(Enclave_Config_File)"
@Echo "You can also sign the enclave using an external signing tool."
@Echo "To build the project in simulation mode set SGX_MODE=SIM. To build the project in prerelease mode set SGX_PRERELEASE=1 and SGX_MODE=HW."
else
target: $(App_Name) $(Signed_Enclave_Name)
ifeq ($(Build_Mode), HW_DEBUG)
@Echo "The project has been built in debug hardware mode."
else ifeq ($(Build_Mode), SIM_DEBUG)
@Echo "The project has been built in debug simulation mode."
else ifeq ($(Build_Mode), HYPER_DEBUG)
@Echo "The project has been built in debug hyper mode."
else ifeq ($(Build_Mode), HW_PRERELEASE)
@Echo "The project has been built in pre-release hardware mode."
else ifeq ($(Build_Mode), SIM_PRERELEASE)
@Echo "The project has been built in pre-release simulation mode."
else ifeq ($(Build_Mode), HYPER_PRERELEASE)
@Echo "The project has been built in pre-release hyper mode."
else ifeq ($(Build_Mode), SIM_RELEASE)
@Echo "The project has been built in release simulation mode."
else
@Echo "The project has been built in release hyper mode."
endif
endif

run: all
ifneq ($(Build_Mode), HW_RELEASE)
@$(CURDIR)/$(App_Name)
@Echo "RUN => $(App_Name) [$(SGX_MODE)|$(SGX_ARCH), OK]"
endif

.config_$(Build_Mode)_$(SGX_ARCH):
@RM -f .config_* $(App_Name) $(Enclave_Name) $(Signed_Enclave_Name) $(App_Cpp_Objects) App/Enclave_u.* $(Enclave_Cpp_Objects) Enclave/Enclave_t.*
@touch .config_$(Build_Mode)_$(SGX_ARCH)

######## App Objects ########

App/Enclave_u.h: $(SGX_EDGER8R) Enclave/Enclave.edl
@cd App && $(SGX_EDGER8R) --sgx-mode $(SGX_MODE) --untrusted ../Enclave/Enclave.edl --search-path ../Enclave --search-path $(SGX_SDK)/include
@Echo "GEN => $@"

App/Enclave_u.c: App/Enclave_u.h

App/Enclave_u.o: App/Enclave_u.c
@$(CC) $(SGX_COMMON_CFLAGS) $(App_C_Flags) -c $< -o $@@
@Echo "CC <= $<"

App/%.o: App/%.cpp App/Enclave_u.h
@$(CXX) $(SGX_COMMON_CXXFLAGS) $(App_Cpp_Flags) -c $< -o $@@
@Echo "CXX <= $<"

$(App_Name): App/Enclave_u.o $(App_Cpp_Objects)
@$(CXX) $^ -o $@ $(App_Link_Flags)
@Echo "LINK => $@"

######## Enclave Objects ########
LIBSS = $(SO_FILES)

Enclave/Enclave_t.h: $(SGX_EDGER8R) Enclave/Enclave.edl
@cd Enclave && $(SGX_EDGER8R) --sgx-mode $(SGX_MODE) --trusted ../Enclave/Enclave.edl --search-path ../Enclave --search-path $(SGX_SDK)/include
@Echo "GEN => $@"

Enclave/Enclave_t.c: Enclave/Enclave_t.h

Enclave/Enclave_t.o: Enclave/Enclave_t.c
@$(CC) $(SGX_COMMON_CFLAGS) $(Enclave_C_Flags) -c $< -o $@@
@Echo "CC <= $<"

Enclave/%.o: Enclave/%.cpp
@$(CXX) $(SGX_COMMON_CXXFLAGS) $(Enclave_Cpp_Flags) -c $< -o $@@
@Echo "CXX <= $<"

$(Enclave_Cpp_Objects): Enclave/Enclave_t.h

$(Enclave_Name): Enclave/Enclave_t.o $(Enclave_Cpp_Objects)
@$(CXX) $^ -o $@ $(Enclave_Link_Flags)
@Echo "LINK => $@"

$(Signed_Enclave_Name): $(Enclave_Name)
@$(SGX_ENCLAVE_SIGNER) sign -key Enclave/Enclave_private_test.pem -enclave $(Enclave_Name) -out $@ -config $(Enclave_Config_File)
@Echo "SIGN => $@"

.PHONY: clean

clean:
@RM -f .config_* $(App_Name) $(Enclave_Name) $(Signed_Enclave_Name) $(App_Cpp_Objects) App/Enclave_u.* $(Enclave_Cpp_Objects) Enclave/Enclave_t.*

[Wan_Intel]

Hi Qianbao,

Thanks for reaching out to us.

Let me check with relevant team and I'll update you as soon as possible.

Regards,

Wan

[qianbao]

When I made, I found that enclave.so had been generated, but the cblas_gemm_s16s16s32 function symbol in enclave.so was U (undefined)

nm -D enclave.so | grep "cblas_gemm_s16s16s32"
          U cblas_gemm_s16s16s32

and then the sign failed. What should I do? Should I link more .so, or other .a

[qianbao]

I want to know how to solve all the unresolved symbols from the library by myself to make it workable inside SGX enclaves.

[Wan_Intel]

Hi Qianbao,

Thanks for your patience.

Referring to GitHub thread: Does Intel® SGX work with Intel® MKL, developer lzha101 mentioned that Intel® Math Kernel Library is not supported inside Intel® SGX Enclave. Alternative way is to use Intel® SGX Deep Neural Network Library (DNNL) inside Intel® SGX Enclave. Hope it helps.

Regards,

Wan

[Wan_Intel]

Hi Qianbao,

Thanks for your question.

If you need any additional information from Intel, please submit a new question as this thread will no longer be monitored.

Regards,

Wan