Optimization Sklearn (svm.SVC) not working

[cuongnguyengit]

Hi all

I have a issue in following when i try to run my code to check speed IntelPython making.

My system:

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 30
On-line CPU(s) list: 0-29
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 30
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 42
Model name: Intel Xeon E312xx (Sandy Bridge, IBRS update)
Stepping: 1
CPU MHz: 2194.710
BogoMIPS: 4389.42
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 4096K
L3 cache: 16384K
NUMA node0 CPU(s): 0-29

My code:

from sklearn.svm import SVC
from sklearn.datasets import load_digits
from time import time

svm_sklearn = SVC(kernel="rbf", gamma="scale", C=0.5, probability=True)

digits = load_digits()
X, y = digits.data, digits.target

start = time()
svm_sklearn = svm_sklearn.fit(X, y)
end = time()
print(end - start) # output: 0.141261...
t = time()
print(svm_sklearn.score(X, y)) # output: 0.9905397885364496
print(svm_sklearn.score(X, y)) # output: 0.9905397885364496
print(svm_sklearn.score(X, y)) # output: 0.9905397885364496
print(time() - t, '(s)')

from daal4py.sklearn import patch_sklearn
patch_sklearn() # <-- apply patch
from sklearn.svm import SVC

svm_d4p = SVC(kernel="rbf", gamma="scale", C=0.5, probability=True)

start = time()
svm_d4p = svm_d4p.fit(X, y)
end = time()
print(end - start) # output: 0.032536...
t = time()
print(svm_d4p.score(X, y)) # output: 0.9905397885364496
print(svm_d4p.score(X, y)) # output: 0.9905397885364496
print(svm_d4p.score(X, y)) # output: 0.9905397885364496
print(time() - t, '(s)')

and its result

1.0682997703552246
0.9905397885364496
0.9905397885364496
0.9905397885364496
0.6014969348907471 (s)
Intel(R) Data Analytics Acceleration Library (Intel(R) DAAL) solvers for sklearn enabled: https://intelpython.github.io/daal4py/sklearn.html
0.9806723594665527
0.9905397885364496
0.9905397885364496
0.9905397885364496
0.6032438278198242 (s)

I cant understand this. Speeds are similar but i hava 30 cores.

conda list:

# Name Version Build Channel
absl-py 0.10.0 py37hc8dfbb8_1 <unknown>
aiohttp 3.6.3 py38h1e0a361_2 <unknown>
asn1crypto 1.3.0 py37_0 intel
astunparse 1.6.3 py_0 <unknown>
async-timeout 3.0.1 py37_0 <unknown>
attrs 20.2.0 py_0 <unknown>
blinker 1.4 py37_0 <unknown>
boto 2.49.0 py_0 <unknown>
boto3 1.16.9 pyhd8ed1ab_0 <unknown>
botocore 1.19.9 pyhd3deb0d_0 <unknown>
bzip2 1.0.8 h14c3975_2 intel
c-ares 1.16.1 h516909a_3 <unknown>
cachetools 4.1.1 py_0 <unknown>
certifi 2020.4.5.2 py37_0 intel
cffi 1.14.0 py37h14c3975_2 intel
chardet 3.0.4 py37_3 intel
click 7.1.2 py_0 <unknown>
conda 4.8.3 py37_0 intel
conda-package-handling 1.4.1 py37_2 intel
cryptography 2.9.2 py37_0 intel
cycler 0.10.0 py37_7 intel
cython 0.29.17 py37h6ebd63d_0 intel
daal 2020.2 intel_254 <unknown>
daal4py 0.2020.2 py37h533f8aa_7 intel
docutils 0.16 pypi_0 pypi
et-xmlfile 1.0.1 py_0 <unknown>
fastapi 0.60.1 pypi_0 pypi
fastbpe 0.1.0 pypi_0 pypi
flit 2.3.0 pypi_0 pypi
flit-core 2.3.0 pypi_0 pypi
freetype 2.10.2 0 intel
funcsigs 1.0.2 py37_7 intel
gast 0.3.3 py_0 <unknown>
gensim 3.7.3 py37he1b5a44_1 <unknown>
google-auth 1.22.0 py_0 <unknown>
google-auth-oauthlib 0.4.1 py_2 <unknown>
google-pasta 0.2.0 pyh8c360ce_0 <unknown>
grpcio 1.31.0 py37hb0870dc_0 <unknown>
h11 0.9.0 pypi_0 pypi
h5py 2.10.0 nompi_py37hf7afa78_105 <unknown>
hdf5 1.10.6 hb1b8bf9_0 <unknown>
httptools 0.1.1 pypi_0 pypi
icc_rt 2020.2 intel_254 <unknown>
idna 2.9 py37_0 intel
impi_rt 2019.8 intel_254 <unknown>
importlib-metadata 2.0.0 py37hc8dfbb8_0 <unknown>
intel-openmp 2020.2 intel_254 <unknown>
intelpython 2020.2 0 intel
ipp 2020.2 intel_254 <unknown>
jdcal 1.4.1 py_0 <unknown>
jmespath 0.10.0 py_0 <unknown>
joblib 0.15.1 py37_0 intel
keras 2.4.3 py_0 <unknown>
keras-base 2.4.3 py_0 <unknown>
keras-preprocessing 1.1.0 py_0 <unknown>
kiwisolver 1.2.0 py37hf484d3e_0 intel
libarchive 3.4.2 h62408e4_0 <unknown>
libffi 3.3 11 intel
libgcc-ng 9.1.0 hdf63c60_0 intel
libgfortran-ng 7.3.0 hdf63c60_0 <unknown>
libpng 1.6.37 4 intel
libprotobuf 3.13.0.1 h200bbdf_0 <unknown>
libstdcxx-ng 9.1.0 hdf63c60_0 intel
libxml2 2.9.10 h14c3975_0 intel
llvmlite 0.32.1 py37h75308e0_0 intel
lz4-c 1.9.2 hf484d3e_1 intel
lzo 2.10 h14c3975_4 intel
markdown 3.3.2 py37_0 <unknown>
matplotlib 3.1.2 py37hf484d3e_5 intel
mkl 2020.2 intel_254 <unknown>
mkl-service 2.3.0 py37_4 intel
mkl_fft 1.1.0 py37h6ebd63d_3 intel
mkl_random 1.1.1 py37h6ebd63d_3 intel
mpi4py 3.0.3 py37hf484d3e_7 intel
multidict 4.7.6 py37h7b6447c_1 <unknown>
mysql-connector-python 8.0.22 pypi_0 pypi
nltk 3.5 pypi_0 pypi
numba 0.49.1 np118py37hf484d3e_2 intel
numexpr 2.7.0 py37_2 intel
numpy 1.18.5 py37h6ebd63d_5 intel
numpy-base 1.18.5 py37_5 intel
oauthlib 3.1.0 py_0 <unknown>
openpyxl 3.0.5 pypi_0 pypi
openssl 1.1.1g h14c3975_1 intel
opt_einsum 3.1.0 py_0 <unknown>
pandas 0.25.3 py37hf484d3e_6 intel
pip 20.1 py37_0 intel
protobuf 3.13.0.1 py37hb809cae_1 <unknown>
pyasn1 0.4.8 py_0 <unknown>
pyasn1-modules 0.2.8 py_0 <unknown>
pycosat 0.6.3 py37_5 intel
pycparser 2.20 py37_1 intel
pydantic 1.6.1 pypi_0 pypi
pyeditline 2.0.1 py37_0 intel
pyjwt 1.7.1 py_0 <unknown>
pymysql 0.10.1 pypi_0 pypi
pymysql-pool 0.3.4 pypi_0 pypi
pyopenssl 19.1.0 py37_1 intel
pyparsing 2.4.7 py37_1 intel
pysocks 1.7.0 py37_1 intel
python 3.7.7 hf484d3e_13 intel
python-crfsuite 0.9.7 py37h99015e2_1 <unknown>
python-dateutil 2.8.1 py37_1 intel
python-libarchive-c 2.8 py37_13 intel
pytoml 0.1.21 pypi_0 pypi
pytz 2020.1 py37_0 intel
pyvi 0.1 pypi_0 pypi
pyyaml 5.3 py37_0 intel
regex 2019.8.19 pypi_0 pypi
requests 2.23.0 py37_4 intel
requests-oauthlib 1.3.0 pyh9f0ad1d_0 <unknown>
rsa 4.6 pyh9f0ad1d_0 <unknown>
ruamel_yaml 0.15.99 py37_5 intel
scikit-learn 0.23.1 py37h6ebd63d_0 intel
scipy 1.4.1 py37h6ebd63d_7 intel
setuptools 47.3.0 py37_0 intel
six 1.15.0 py37_0 intel
sklearn-crfsuite 0.3.6 pyh9f0ad1d_0 <unknown>
smart_open 2.1.0 py_0 <unknown>
smp 0.1.4 py37_0 intel
sqlite 3.32.1 h14c3975_1 intel
starlette 0.13.6 pypi_0 pypi
tabulate 0.8.7 pyh9f0ad1d_0 <unknown>
tbb 2020.3 intel_254 <unknown>
tbb4py 2020.3 py37_intel_0 <unknown>
tcl 8.6.9 h14c3975_2 intel
tensorboard 2.2.1 pyh532a8cf_0 <unknown>
tensorboard-plugin-wit 1.6.0 py_0 <unknown>
tensorflow 2.2.0 py37_0 <unknown>
tensorflow-base 2.2.0 0 <unknown>
tensorflow-estimator 2.2.0 pyh208ff02_0 <unknown>
tensorflow-mkl 2.2.0 h4fcabd2_0 <unknown>
termcolor 1.1.0 py37_1 <unknown>
threadpoolctl 2.1.0 pyh5ca1d4c_0 <unknown>
tk 8.6.9 6 intel
tqdm 4.39.0 py37_2 intel
unidecode 1.1.1 pypi_0 pypi
urllib3 1.25.9 py37_0 intel
uvicorn 0.11.6 pypi_0 pypi
uvloop 0.14.0 pypi_0 pypi
websockets 8.1 pypi_0 pypi
werkzeug 1.0.1 pyh9f0ad1d_0 <unknown>
wheel 0.34.2 py37_4 intel
wrapt 1.11.2 py37h8f50634_1 <unknown>
xgboost 1.1.1 497_gcd3d14apy37_0 intel
xlrd 1.2.0 pypi_0 pypi
xz 5.2.5 h14c3975_0 intel
yaml 0.1.7 6 intel
yarl 1.6.2 py37h8f50634_0 <unknown>
zipp 3.3.1 py_0 <unknown>
zlib 1.2.11.1 h14c3975_1 intel
zstd 1.4.4 hf484d3e_1 intel

i thinks its useful

Can you give me a advice?? Plz

Thanks for reading.

[AthiraM_Intel]

Hi,

Thanks for reaching out to us.

Could you please try the same after exporting the below command and check whether there is any improvement in time.

export USE_DAAL4PY_SKLEARN=1

We could see that you are using an old intel hardware.

We tried the same in latest intel architecture (cascade lake).

After exporting the above command , we observed an improvement in time.

We got the following result :

Intel(R) Data Analytics Acceleration Library (Intel(R) DAAL) solvers for sklearn enabled: https://intelpython.github.io/daal4py/sklearn.html

0.41217684745788574

0.9905397885364496

0.9905397885364496

0.9905397885364496

0.42245054244995117 (s)

Intel(R) Data Analytics Acceleration Library (Intel(R) DAAL) solvers for sklearn enabled: https://intelpython.github.io/daal4py/sklearn.html

0.39907169342041016

0.9910962715637173

0.9910962715637173

0.9910962715637173

0.3754143714904785 (s)

Also you can try with a bigger dataset to check if there is significant improvement in time.

Thanks.

[cuongnguyengit]

Hi,

Thanks for your reply.

I did as below:

export USE_DAAL4PY_SKLEARN=1

source activate root

python3.7 test.py

and output is:

Intel(R) Data Analytics Acceleration Library (Intel(R) DAAL) solvers for sklearn enabled: https://intelpython.github.io/daal4py/sklearn.html
1.0459799766540527
0.9905397885364496
0.9905397885364496
0.9905397885364496
0.631990909576416 (s)
Intel(R) Data Analytics Acceleration Library (Intel(R) DAAL) solvers for sklearn enabled: https://intelpython.github.io/daal4py/sklearn.html
1.02339506149292
0.9905397885364496
0.9905397885364496
0.9905397885364496
0.6148135662078857 (s))

I think they have no change in time. When i dont use command "export USE_DAAL4PY_SKLEARN=1", my training svm time is the same or better sometime.

The program would be better than it?

My code is from a sample in your github running very good.

I look forward to be more.

Thanks!

[AthiraM_Intel]

Hi,

No improvement in time may be because you are using old Intel architecture. We will check with the concerned team regarding this issue and get back to you soon.

Thanks

[Rachel_O_Intel]

Hi,

I am an engineer looking into your case. Please also let me know which OS system you have if the solutions I outline below do not work.

There are a few options you can try:

1) Ensure that environment variable is turned on only when you run accelerated scikit-learn. If you turn on the environment variable before running the whole script, it will turn on Intel accelerations for both versions, regardless of whether you've included the monkey-patch programmatically or not (you can see it's turned on for both sklearn versions in the script as it indicates from the print out saying "solvers enabled" twice).

If you separate the non-accelerated scikit-learn and the accelerated scikit-learn into two scripts, then only turn on the environment variable or monkey-patch only with the Intel-accelerated one, you should be able to see a difference. Also note that the toy scikit-learn dataset used in this sample is fairly small, which can also explain why you may not see a difference (it may not be large enough to show a difference on your hardware).

You can also separate the non-accelerated and accelerated scikit-learn into different environments and then try running them (put the non-accelerated scikit-learn in a separate environment without IDP). 

2) I notice that you are using IDP/daal4py 2020 update 2. Please consider updated IDP to version 2020 update 4 or daal4py/Intel scikit-learn to version 2020 update 3 directly. There were updates done to the scikit-learn accelerations in this, so this should help. This may also allow you to gain more performance.

3) You can also check if Intel accelerated scikit-learn is being utilized with verbose mode (it will show you a series of print statements which should indicate which implementation is being called). I suggest trying this while the non-accelerated scikit-learn is commented out.

In your command line shell, use the following command: 

#for linux or macOS
export IDP_SKLEARN_VERBOSE=INFO

#for windows
set IDP_SKLEARN_VERBOSE=INFO

Please let me know if this helps, also refer here to the daal4py scikit-learn documentaion for more information.

Best,

Rachel

[cuongnguyengit]

Hi,

I will try your solutions soon.

I have a more question. I installed the Intel distribution for Python for ambitions possible to speed up my machine learning code (Sklearn) and deep learning code ( with Tensorflow)

but my intel hardware system is old and not working really as my wish.

So i want to ask whether my deep learning code is better when my system doesnt have a GPU for running it.

Thanks 

[Rachel_O_Intel]

Hi,

Please let me know if my solutions helped resolve your scikit-learn issues or if you have any other problems with viewing scikit-learn performance related to this issue.

I suggest posting your deep learning performance inquiries separately on the Intel Optimized AI Frameworks Forum. That forum will have better equipped experts to assist you on optimizing your deep learning code.

Please note that Intel Distribution for Python includes Intel optimizations for NumPy, SciPy, Scikit-Learn, and daal4py. Intel Optimized TensorFlow is considered a separate product that is not included or optimized by Intel Distribution of Python.

Best,

Rachel

[cuongnguyengit]

Hi,

Thanks for your support.

I am trying to update IDP to version 2020 update 4 but it spent more time i think.

[cuongnguyengit]

Hi,

After i update the software i think it worked. Time for training in my example have only 0.58 s

and time for testing is 0.6s.

It is significantly great for my task.

Thanks.

[Rachel_O_Intel]

Hi,

I am glad to hear that you can now see the performance speedup in your code.

Please also look at the daal4py and accelerated scikit-learn API documentation for reference.

Is your classical ML issue resolved?

Best,

Rachel

[cuongnguyengit]

Hi,

my issue is solved by update Intel distribution for python version.

Thank you very much for your supports 

[Rachel_O_Intel]

Hi,

No problem, I am glad your issue was resolved.

Thanks for the confirmation. This thread will be no longer supported by Intel, please raise a new thread if you have any further issues.

Best,

Rachel