Getting OMP: Error #15: Initializing libiomp5.so, but found libiomp5.so already initialized.

[Erika_A_Intel]

I am running a python script which imports and uses these libraries:

pandas, keras, sklearn, numpy, math, matplotlib

It throws the following error:

OMP: Error #15: Initializing libiomp5.so, but found libiomp5.so already initialized.

OMP: Hint: This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.

Since my user code is not explicitly calling any OpenMP library, I wonder if you could help me find out which libraries may be causing this and how to solve it.

[Surya_R_Intel]

Hi Erika, Thanks for reaching out to us. From our research it is found that the error could be caused by the numpy library. For further troubleshooting, it would be better if you could share us the python script which will help us in reproducing the error. Also kindly let me know if you are running the script in conda environment. Regards, Surya

[Erika_A_Intel]

Hello Surya,

I am new to conda, but I know that some packages were installed with conda in the environment I am using.

Also, I may have narrowed down the issue while trying to give you a script and data. It can be reproduced with the script from this tutorial: https://machinelearningmastery.com/stateful-stateless-lstm-time-series-forecasting-python/ and a different data set. Under "A vs A test" section you will find the script needed to reproduce this issue. I will attach the CSV files.

I replaced the run() method from the original tutorial with these two:

def run(): # load dataset series = read_csv('sample_timeseries.csv', header=0).head(11000) # experiment repeats = 1 results = DataFrame() # run experiment results['results'] = experiment(repeats, series) # summarize results print(results.describe()) # save results results.to_csv('experiment_stateful.csv', index=False)   def run2(): # load dataset data = read_csv('sample_raw.csv', header=0).head(11000) series = DataFrame() series['timeseries'] = data['var1'] / data['var2'] # experiment repeats = 1 results = DataFrame() # run experiment results['results'] = experiment(repeats, series) # summarize results print(results.describe()) # save results results.to_csv('experiment150_stateful.csv', index=False)   # entry point #run() run2()

run2() will fail and run() will not. The difference is the division operation to create the 'series' data frame.

Also, lines 3 and 18 use method "head". I did this to get the script running faster because the data set has more than 100,000 samples. I noticed that for smaller values (I tried 200), run2() will not fail.

[Erika_A_Intel]

posted a file.

[Surya_R_Intel]

Hi Erika, Thank you for the details shared. We had tried to run the python script from our end. But as of now we couldn't recreate the error due to some unknown issue. We are trying to resolve this. Meanwhile could you please try to run the script in a conda environment. Please follow the below steps. conda create -n <env_name> -c intel python=3.6 source activate <env_name> After activating the conda environment, install the required packages and run the python script. Kindly let us know if this helps. Surya

[Erika_A_Intel]

Hi Surya,

I followed your instructions (I assumed I had to use conda to install the missing packages) but I still see the error. These are the command line inputs I used:

conda create -n experiment -c intel python=3.6 source activate experiment conda install pandas conda install scikit-learn conda install tensorflow conda install pyyaml python tutorial.py

Thanks.

[Surya_R_Intel]

Hi Erika, We tried to run the script in conda environment. For 200 samples, we are able to execute both the run() and run2() function without any error. It took approximately 6 minutes. For 11000 samples, I have submitted the script as a qsub job for both run() and run2() function. We didn't get any error and it is running in the background. Since we are not able to recreate the error, could you please share the logs obtained while running the script in conda environment. Or else kindly let us know how to recreate the exact error. Regards, Surya

[Surya_R_Intel]

Hi Erika, Hope your issue got resolved. Kindly let us know the status. Regards, Surya

[Surya_R_Intel]

We are closing this thread since we didn't get any response. Kindly open a new thread if you face further issues