MKL with Spark

Jacques_J_ · ‎08-10-2017

Hello,

I'm trying to use MKL with Spark using netlib-java. I included the folder containing the dlls in the Path variable and specified the following option to the JVM : -Dcom.github.fommil.netlib.BLAS=mkl_rt.dll

However, it doesn't work and I still get the following warnings :

17/08/10 14:22:09 WARN BLAS: Failed to load implementation from: mkl_rt.dll
17/08/10 14:22:09 WARN BLAS: Using the fallback implementation.

Any help would be greatly appreciated

Ying_H_Intel · ‎08-15-2017

Hi Jacques,

Do you use spark under windows? mkl_rt.dll is window library, where mkl_rt.so is Linux library. As it is dynamic library, you may add some environment LD_LIBRARY_PATH or run script source /opt/intel/mkl/mklvars.sh intel64 before run your spark application

what kind of application do you like to run based on MKL? I may recommend one of data analysis accelerate library DAAL , which is based on MKL, have java /c/python interface, and also work on Spark and MLlib.

get free DAAL from https://software.intel.com/en-us/intel-daal

DAAL training webpage:

https://software.intel.com/en-us/intel-daal-support/training

DAAL Scala Spark sample download from

https://software.intel.com/en-us/product-code-samples

Best Regards,

Ying