Intel® Distribution for Python*
Engage in discussions with community peers related to Python* applications and core computational packages.
424 Discussions

Python Jupyter Notebook: Kernel Dies and OMP: Error #15

Jon_1607
Novice
6,886 Views

I am trying to run a Python script under Jupyter Notebook ( Anaconda3-2023.03-1-Windows-10_Pro x86_64).

I import the following Python Libraries and Data:

import tensorflow

from tensorflow.keras import layers, models

import matplotlib.pyplot as mplot

from keras.datasets import cifar10

import numpy as np

 

I have attached the following text file:

Neural Networks Program 04.docx which includes both the Python Code and the Output.

 

A:  When the section involving MatPlotLib constructing a graph is executed , The Kernel Fails.

B: When I inspected the Jupyter Notebook Log; it displayed the following message:

 

C: OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5 already initialized.


OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results.

The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library.

As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results.

For more information, please see http://www.intel.com/software/products/support/.

 

I would appreciate it if someone could help me to diagnose and understand this problem so that I can prevent this error as well as the Kernel dying.

 

Thanks

0 Kudos
11 Replies
RajashekarK_Intel
Moderator
6,832 Views

Hi,

Thanks for posting in Intel Communities

 

Appreciate your detail of information by providing us with the sample code. 

 

Can you provide additional details inorder to debug this issue of multiple copies of OpenMP runtime being linked to the program.

 

1. Are you using Intel Distribution for Python? If yes, Please provide the version of the package.

2. Packages version sitting in your base conda environment/virtual environment where you are facing this issue, with pip freeze > requirements.txt. Send us your requirements.txt file as this will help us replicate your exact conda environment. 

3. Provide the screen shot of the error.

 

Regards,

Rajashekar

 

0 Kudos
Jon_1607
Novice
6,798 Views
LIBIOMP5.DLL ERROR
 
When running Jupyter Notebook outside of Anaconda Navigator I noticed an error in the log:
Error #15: Initializing libiomp5md.dll, but found libiomp5 already initialized
 
This file is stored in two places:
C:\users\jondi\anaconda3\library\bin
C:\users\jondi\anaconda3\pkgs\intel-openmp 2023.1.0-h59b6b97_46319\Library\bin
 
How can I prevent this problem from occurring? In general and when running Jupyter Notebook?
0 Kudos
RajashekarK_Intel
Moderator
6,779 Views

Hi,

Unfortunately, I'm not getting any libiomp conflict error in my environment.

 

In order to debug and replicate your issue requesting you to follow up on below questions.

  1. Are you using Intel Distribution for python (intelpython) ? check via command
    conda list​
    .
  2. To replicate we need to check packages present in your conda environment, you can do that by 
    conda list --export > requirements.txt​
    and send the requirements.txt file.
  3. As a workaround you can uninstall and install matplotlib using 
    conda uninstall matplotlib
    conda install matplotlib​
    and check if the issue still persists.

 

Regards,

Rajashekar

 

0 Kudos
Jon_1607
Novice
6,768 Views
Thank you for responding to my request for assistance w.r.t. the code in the two Jupyter Notebook programs.
 
In my previous response to the email from Intel, I sent a copy of the requirements.txt file .
 
I have uninstalled Anaconda 2023.03 and removed everything related to its installation from my computer.
 
I have downloaded and installed Anaconda 2023.07 selecting YES to the specification w.r.t. Python 3.11
 
A. I am not able to install Tensorflow into the Root Environment either from within Anaconda Navigator(GUI) 
or within the Command Prompt.
 
B. I created a new environment TFLOW, from within Anaconda Navigator; starting with tensorflow and specifying 
Python 3.10 and I then added the basic versions of the additional packages: numpy, pandas, matplotlib, sklearn...
 
I have run the Jupyter Notebook containing the code for the Neural Networks Program 03 ( sent to you ) 
This program runs successfully in the TFLOW environment.
 
I have run the Jupyter Notebook containing the code for the Neural Networks Program 04 ( sent to you).
This program was executed one cell at a time to confirm the input, process and output and it runs successfully
in the TFLOW environment but as soon as the matplotlib code cell is executed then the kernel dies....
 
I have attempted to run both programs on another platform that is available from Anaconda Navigator [ DATALORE FROM JETBRAINS ] : While both the above programs; Neural Networks Program 03  and Neural Networks Program 04, ran successfully and I produced the correct output,  I did observe some strange warning messages from this platform
relating to memory or storage issues. 
 
So I increased the Windows PageFile size and I increased the buffer size in the Jupyter Notebook Config file and then 
tried to run the two Jupyter Notebook programs in my TFLOW environment one cell at a time.
BUT I received the same results as before: 
a. program 03 ( including matplotlib code ) works fine
b. program 04 works fine (UNTIL the matplotlib code is executed ==> Kernel Error)
 
 
KERNEL DYING ERROR QUESTIONS
 
System Information:
 
 
Jon_1607_1-1689631308105.png

 


Are the Kernel problems caused by:
a  Memory / Resource issue with Jupyter Notebook?
b. A clash between various packages: MatplotLib, Numpy, Tensorflow, Python ...?
c.  A conflict between TensorFlow and the recent version of Python ?
 
Is there a way to optimize the order of imputation of packages into a Notebook?
 
Is there a way to optimize the order of installation of packages into an environment?
 
This whole saga is very frustrating and internet research on this problem (which seems to have been around in 
various forms for several years ) , is not very helpful.
 
I will try the suggestion w.r.t. uninstalling matplotlib ...
 
0 Kudos
RajashekarK_Intel
Moderator
6,729 Views

Hi,

 

Yes, Please try the uninstalling and installing method.

 

Reading your issue, you mentioned that you've sent us the requirements.txt and code related to Neural Networks Program 03, We can confirm from our end that we haven't received that yet in this Community (Intel Distribution for Python).

 

So, requesting you to send the requirements.txt file to check whether you are using intelpython and other packages version, to replicate the same.

conda list --export > requirements.txt

 

Even I've tried the same steps which you've done Installing everything from scratch, But couldn't replicate the issue as you faced with the code you sent Neural Networks Program 04.

no_warning_of_omp.png

 

 

Regards,

Rajashekar

 

0 Kudos
Jon_1607
Novice
6,707 Views

I have attached the requirements.txt.

The previous email reply was rejected because i simply replied via gmail instead of logging in?

0 Kudos
Jon_1607
Novice
6,707 Views
I installed the Intel Distribution for Python and got the same results as before w.r.t. the two programs
 
When the first program ran there was no kernel problem:
 
When the second program ran there was no Jupyter Notebook and so no kernel problem:
 
However, it crashed when executing the matplotlib code because of the following problem
 
Error #15: Initializing libiomp5md.dll, but found libiomp5 already initialized
 
Please refer to the attached screenshot
 
0 Kudos
RajashekarK_Intel
Moderator
6,673 Views

Hi,

 

We are trying hard to replicate your issue, Hence in order to debug we request your assistance in follow-up questions and follow our suggested workaround where we didn't face any issue even while having Intel Distribution for Python in the environment.

 

1. As you mentioned you are using Intel distribution for python, the requirements.txt you've sent doesn't reflect it.

2. Did you try uninstalling and installing Matplotlib?

 

 

One possible issue might be your conda environment, we suggest you to create the environment which we used as we didn't face any error with jupyter kernel and CLI execution of program.

 

Workaround

 

1. Please find the attached environment.yml file under the zip file and create a conda environment.

 

conda env create -n new_env -f environment.yml

 

2. Activate the environment using: 

 

conda activate new_env

 

3. Launch your jupyter notebook with the following command: 

 

jupyter notebook

 

4. Create a jupyter notebook and execute the program which you've provided Neural Networks Program 04 cell by cell.

 

We even executed your program on CLI without kernel, and didn't face any issue.

no_kernel_CLI_Run.png

 

If this resolves your issue, make sure to accept this as a solution. This would help others with similar issue.

 

Regards,

Rajashekar

 

0 Kudos
Jon_1607
Novice
6,570 Views

new requirements.docx file attached : Intel_Env environment based on the environment.yml file provided by Intel.

 

a. I installed Anaconda 2023.07 for Windows 10 PRO 64

b. I created the environment as per the instructions using the yml file provided by Intel

c. i activated this environment

d. I checked that all the relevant packages for my Python Code were installed in the Intel_Env environment

e. I executed the first program ( using Tensorflow and matplotlib) Neural Networks Program 03:

   1. cell by cell:

    2. it completed successfully

f. I executed the first program ( using Tensorflow and matplotlib) Neural Networks Program 04:

   1. cell by cell

    2. it crashed when executing the matplotlib code.

 

I will now uninstall numpy and matplotlib from the Intel_ENV environment

I will then install numpy and matplotlib into the Intel_ENV environment

 

I will try to repeat the two programs again using the updated Intel_ENV environment:

a. Neural Networks Program 03

b. Neural Networks Program 04

0 Kudos
RajashekarK_Intel
Moderator
6,276 Views

Hi, Good day to you.


We have not heard back from you. Is your issue resolved with the above solutions you mentioned you're going to try? If this resolves your issue, make sure to accept this as a solution. This would help others with similar issue.


FYI, we don't have the sample which you're mentioning it's working for you (Neural Networks Program 3)


If you are still facing issue, provide below details,

1) CPU Details

2) Log where the notebook is getting crashed where you've tried with environment provided by us.


Regards,

Rajashekar



0 Kudos
RajashekarK_Intel
Moderator
6,204 Views

Hi,

We assume that your issue is resolved. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.


Regards,

Rajashekar


0 Kudos
Reply