Error in testing ImageNet Dataset in IntelAI models

[TaejinKim]

Hello,

I'm testing https://github.com/IntelAI/models/tree/master/datasets/imagenet this, and I've faced problem.

My error outputs are here.

2022-05-31 06:06:36.620505: F tensorflow/core/platform/default/env.cc:73] Check failed: ret == 0 (11 vs. 0)Thread tf_ForEach creation via pthread_create() failed.
Fatal Python error: Aborted

Thread 0x00007fddd0bf6b80 (most recent call first):
File "/home/mscho/tf_env/lib64/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 443 in get_matching_files_v2
File "/home/mscho/tf_env/lib64/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 383 in get_2022-05-31 06:06:36.620610: F tensorflow/core/platform/default/env.cc:73] Check failed: ret == 0 (11 vs. 0)Thread tf_ForEach creation via pthread_create() failed.
matching_files
File "/home/mscho/imagenet_dir/imagenet_to_gcs.py", line 366 in convert_to_tf_records
File "/home/mscho/imagenet_dir/imagenet_to_gcs.py", line 419 in main
File "/home/mscho/tf_env/lib64/python3.6/site-packages/absl/app.py", line 258 in _run_main
File "/home/mscho/tf_env/lib64/python3.6/site-packages/absl/app.p./imagenet_to_tfrecords.sh: line 76: 2518293 Aborted (core dumped) python3 ${WORKDIR}/imagenet_to_gcs.py --raw_data_dir=$IMAGENET_HOME --local_scratch_dir=$IMAGENET_HOME/tf_records
mv: cannot stat 'train/*': No such file or directory
mv: cannot stat 'validation/*': No such file or directory

I searched for pthread_create() failed error, but I couldn't find something to solve this.

Please let me know, if there's something to be added here.

[VaradJ_Intel]

Hi,

Thank you for posting in Intel communities.

Thanks for reporting this issue. We were able to reproduce it and we have informed the development team about it.

Thank You.

[TaejinKim]

Hi Varad,

Thank you for checking it. How will I know it is modified? Would you notice me by reply this post?

Regards,

Jason.

[Preethi_V_Intel]

Can you please provide the command you ran and understand where exactly you are facing the issue

[Preethi_V_Intel]

Can you please share a reproducer, without one I'll not be able to assist this ticket

[TaejinKim]

Hi,

Sorry for the late reply.

I'd just followed git and run the code ./imagenet_to_tfrecords.sh .

Regards,

Jason.

[Preethi_V_Intel]

this is core dump error. Can you try to fecth core dump files on your working dir if possible or may be run your code on debug mode

bash -x imagenet_to_tfrecords.sh

[Preethi_V_Intel · Accepted solution]

Core dump messages can help us understand it better. Just letting you know that this is a public imagenet script, are you getting this using from Intel packages?

[TaejinKim]

+ tar xf n15075141.tar -C n15075141
+ rm n15075141.tar
+ cd /root/mason
+ [[ ! -f /root/mason/imagenet_to_gcs.py ]]
+ wget https://raw.githubusercontent.com/IntelAI/models/master/datasets/imagenet/imagenet_to_gcs.py
--2022-11-03 19:34:59-- https://raw.githubusercontent.com/IntelAI/models/master/datasets/imagenet/imagenet_to_gcs.py
Resolving proxy-dmz.intel.com (proxy-dmz.intel.com)... 172.30.7.176
Connecting to proxy-dmz.intel.com (proxy-dmz.intel.com)|172.30.7.176|:911... connected.
Proxy request sent, awaiting response... 200 OK
Length: 15260 (15K) [text/plain]
Saving to: ‘imagenet_to_gcs.py’

imagenet_to_gcs.py 100%[==============================================================================================================>] 14.90K --.-KB/s in 0.1s

2022-11-03 19:35:00 (126 KB/s) - ‘imagenet_to_gcs.py’ saved [15260/15260]

+ python3 /root/mason/imagenet_to_gcs.py --raw_data_dir=/root/imagenet_raw_data --local_scratch_dir=/root/imagenet_raw_data/tf_records
2022-11-03 19:35:03.011210: F tensorflow/core/platform/default/env.cc:73] Check failed: ret == 0 (11 vs. 0)Thread tf_ForEach creation via pthread_create() failed.
Fatal Python error: Aborted

Thread 0x00007f20f1da2dc0 (most recent call first):
File "/usr/local/lib64/python3.6/site-packages/tensorflow/py2022-11-03 19:35:03.011261: F tensorflow/core/platform/default/env.cc:73] Check failed: ret == 0 (11 vs. 0)Thread tf_ForEach creation via pthread_create() failed.
thon/lib/io/file_io.py", line 443 in get_ma2022-11-03 19:35:03.011320: F tensorflow/core/platform/default/env.cc:73] Check failed: ret == 0 (11 vs. 0)Thread tf_ForEach creation via pthread_create() failed.
tching_files_v2
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 383 in get_matching_files
File "/root/mason/imagenet_to_gcs.py", line 370 in convert_to./imagenet_to_tfrecords.sh: line 74: 173558 Aborted (core dumped) python3 ${WORKDIR}/imagenet_to_gcs.py --raw_data_dir=$IMAGENET_HOME --local_scratch_dir=$IMAGENET_HOME/tf_records
+ rm /root/mason/imagenet_to_gcs.py
+ cd /root/imagenet_raw_data/tf_records
+ mv 'train/*' /root/imagenet_raw_data/tf_records
mv: cannot stat 'train/*': No such file or directory
+ mv 'validation/*' /root/imagenet_raw_data/tf_records
mv: cannot stat 'validation/*': No such file or directory
+ rm -rf train
+ rm -rf validation
+ cd /root/mason

[TaejinKim]

this is where the error occurs.

[TaejinKim]

Hi Preethi,

I solved this problem by editing imagenet_to_gcs.py.

There was a problem that occurred when processing images by tf.glob.

I replaced tf.glob to glob.glob function. 

Regards,

Jason.

[TaejinKim]

Hi,

I'll reply to this feed after downloading the dataset again.

Sorry for the delay.

Regards,

Jason.