Re:Python Jupyter Notebook: Kernel Dies and OMP: Error #15

Jon_1607 · ‎07-11-2023

I am trying to run a Python script under Jupyter Notebook ( Anaconda3-2023.03-1-Windows-10_Pro x86_64).

I import the following Python Libraries and Data:

import tensorflow

from tensorflow.keras import layers, models

import matplotlib.pyplot as mplot

from keras.datasets import cifar10

import numpy as np

I have attached the following text file:

Neural Networks Program 04.docx which includes both the Python Code and the Output.

A: When the section involving MatPlotLib constructing a graph is executed , The Kernel Fails.

B: When I inspected the Jupyter Notebook Log; it displayed the following message:

C: OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5 already initialized.

OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results.

The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library.

As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results.

For more information, please see http://www.intel.com/software/products/support/.

I would appreciate it if someone could help me to diagnose and understand this problem so that I can prevent this error as well as the Kernel dying.

Thanks

RajashekarK_Intel · ‎07-14-2023

Hi,

Thanks for posting in Intel Communities

Appreciate your detail of information by providing us with the sample code.

Can you provide additional details inorder to debug this issue of multiple copies of OpenMP runtime being linked to the program.

1. Are you using Intel Distribution for Python? If yes, Please provide the version of the package.

2. Packages version sitting in your base conda environment/virtual environment where you are facing this issue, with pip freeze > requirements.txt. Send us your requirements.txt file as this will help us replicate your exact conda environment.

3. Provide the screen shot of the error.

Regards,

Rajashekar

Jon_1607 · ‎07-16-2023

LIBIOMP5.DLL ERROR

When running Jupyter Notebook outside of Anaconda Navigator I noticed an error in the log:

Error #15: Initializing libiomp5md.dll, but found libiomp5 already initialized

This file is stored in two places:

C:\users\jondi\anaconda3\library\bin

C:\users\jondi\anaconda3\pkgs\intel-openmp 2023.1.0-h59b6b97_46319\Library\bin

How can I prevent this problem from occurring? In general and when running Jupyter Notebook?

RajashekarK_Intel · ‎07-17-2023

Hi,

Unfortunately, I'm not getting any libiomp conflict error in my environment.

In order to debug and replicate your issue requesting you to follow up on below questions.

Are you using Intel Distribution for python (intelpython) ? check via command
```
conda list
```
.
To replicate we need to check packages present in your conda environment, you can do that by
```
conda list --export > requirements.txt
```
and send the requirements.txt file.
As a workaround you can uninstall and install matplotlib using
```
conda uninstall matplotlib
conda install matplotlib
```
and check if the issue still persists.

Regards,

Rajashekar

Jon_1607 · ‎07-17-2023

Thank you for responding to my request for assistance w.r.t. the code in the two Jupyter Notebook programs.

In my previous response to the email from Intel, I sent a copy of the requirements.txt file .

I have uninstalled Anaconda 2023.03 and removed everything related to its installation from my computer.

I have downloaded and installed Anaconda 2023.07 selecting YES to the specification w.r.t. Python 3.11

A. I am not able to install Tensorflow into the Root Environment either from within Anaconda Navigator(GUI)

or within the Command Prompt.

B. I created a new environment TFLOW, from within Anaconda Navigator; starting with tensorflow and specifying

Python 3.10 and I then added the basic versions of the additional packages: numpy, pandas, matplotlib, sklearn...

I have run the Jupyter Notebook containing the code for the Neural Networks Program 03 ( sent to you )

This program runs successfully in the TFLOW environment.

I have run the Jupyter Notebook containing the code for the Neural Networks Program 04 ( sent to you).

This program was executed one cell at a time to confirm the input, process and output and it runs successfully

in the TFLOW environment but as soon as the matplotlib code cell is executed then the kernel dies....

I have attempted to run both programs on another platform that is available from Anaconda Navigator [ DATALORE FROM JETBRAINS ] : While both the above programs; Neural Networks Program 03 and Neural Networks Program 04, ran successfully and I produced the correct output, I did observe some strange warning messages from this platform

relating to memory or storage issues.

So I increased the Windows PageFile size and I increased the buffer size in the Jupyter Notebook Config file and then

tried to run the two Jupyter Notebook programs in my TFLOW environment one cell at a time.

BUT I received the same results as before:

a. program 03 ( including matplotlib code ) works fine

b. program 04 works fine (UNTIL the matplotlib code is executed ==> Kernel Error)

KERNEL DYING ERROR QUESTIONS

System Information:

Are the Kernel problems caused by:

a Memory / Resource issue with Jupyter Notebook?

b. A clash between various packages: MatplotLib, Numpy, Tensorflow, Python ...?

c. A conflict between TensorFlow and the recent version of Python ?

Is there a way to optimize the order of imputation of packages into a Notebook?

Is there a way to optimize the order of installation of packages into an environment?

This whole saga is very frustrating and internet research on this problem (which seems to have been around in

various forms for several years ) , is not very helpful.

I will try the suggestion w.r.t. uninstalling matplotlib ...

RajashekarK_Intel · ‎07-18-2023

Hi,

Yes, Please try the uninstalling and installing method.

Reading your issue, you mentioned that you've sent us the requirements.txt and code related to Neural Networks Program 03, We can confirm from our end that we haven't received that yet in this Community (Intel Distribution for Python).

So, requesting you to send the requirements.txt file to check whether you are using intelpython and other packages version, to replicate the same.

conda list --export > requirements.txt

Even I've tried the same steps which you've done Installing everything from scratch, But couldn't replicate the issue as you faced with the code you sent Neural Networks Program 04.

Regards,

Rajashekar

Jon_1607 · ‎07-19-2023

I have attached the requirements.txt.

The previous email reply was rejected because i simply replied via gmail instead of logging in?

Jon_1607 · ‎07-19-2023

I installed the Intel Distribution for Python and got the same results as before w.r.t. the two programs

When the first program ran there was no kernel problem:

When the second program ran there was no Jupyter Notebook and so no kernel problem:

However, it crashed when executing the matplotlib code because of the following problem

Error #15: Initializing libiomp5md.dll, but found libiomp5 already initialized

Please refer to the attached screenshot

RajashekarK_Intel · ‎07-20-2023

Hi,

We are trying hard to replicate your issue, Hence in order to debug we request your assistance in follow-up questions and follow our suggested workaround where we didn't face any issue even while having Intel Distribution for Python in the environment.

1. As you mentioned you are using Intel distribution for python, the requirements.txt you've sent doesn't reflect it.

2. Did you try uninstalling and installing Matplotlib?

One possible issue might be your conda environment, we suggest you to create the environment which we used as we didn't face any error with jupyter kernel and CLI execution of program.

Workaround

1. Please find the attached environment.yml file under the zip file and create a conda environment.

conda env create -n new_env -f environment.yml

2. Activate the environment using:

conda activate new_env

3. Launch your jupyter notebook with the following command:

jupyter notebook

4. Create a jupyter notebook and execute the program which you've provided Neural Networks Program 04 cell by cell.

We even executed your program on CLI without kernel, and didn't face any issue.

If this resolves your issue, make sure to accept this as a solution. This would help others with similar issue.

Regards,

Rajashekar

Jon_1607 · ‎07-20-2023

new requirements.docx file attached : Intel_Env environment based on the environment.yml file provided by Intel.

a. I installed Anaconda 2023.07 for Windows 10 PRO 64

b. I created the environment as per the instructions using the yml file provided by Intel

c. i activated this environment

d. I checked that all the relevant packages for my Python Code were installed in the Intel_Env environment

e. I executed the first program ( using Tensorflow and matplotlib) Neural Networks Program 03:

1. cell by cell:

2. it completed successfully

f. I executed the first program ( using Tensorflow and matplotlib) Neural Networks Program 04:

1. cell by cell

2. it crashed when executing the matplotlib code.

I will now uninstall numpy and matplotlib from the Intel_ENV environment

I will then install numpy and matplotlib into the Intel_ENV environment

I will try to repeat the two programs again using the updated Intel_ENV environment:

a. Neural Networks Program 03

b. Neural Networks Program 04

RajashekarK_Intel · ‎07-26-2023

Hi, Good day to you.

We have not heard back from you. Is your issue resolved with the above solutions you mentioned you're going to try? If this resolves your issue, make sure to accept this as a solution. This would help others with similar issue.

FYI, we don't have the sample which you're mentioning it's working for you (Neural Networks Program 3)

If you are still facing issue, provide below details,

1) CPU Details

2) Log where the notebook is getting crashed where you've tried with environment provided by us.

Regards,

Rajashekar

RajashekarK_Intel · ‎07-31-2023

Hi,

We assume that your issue is resolved. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Regards,

Rajashekar