Intel® DevCloud
Help for those needing help starting or connecting to the Intel® DevCloud
Announcements
The Intel sign-in experience is changing in February to support enhanced security controls. If you sign in, click here for more information.
1216 Discussions

Queries regarding walltime, multithreading, performance

PBhar
Student Ambassador
1,661 Views

Hi,

I had some queries

 

1. Whenever I use dataloader in Pytorch (multithreading), I see this error multiple times:

 

=================================================================

Traceback (most recent call last):

  File "/glob/intel-python/python3/lib/python3.6/multiprocessing/util.py", line 262, in _run_finalizers

    finalizer()

  File "/glob/intel-python/python3/lib/python3.6/multiprocessing/util.py", line 186, in __call__

    res = self._callback(*self._args, **self._kwargs)

  File "/glob/intel-python/python3/lib/python3.6/shutil.py", line 480, in rmtree

    _rmtree_safe_fd(fd, path, onerror)

  File "/glob/intel-python/python3/lib/python3.6/shutil.py", line 438, in _rmtree_safe_fd

    onerror(os.unlink, fullname, sys.exc_info())

  File "/glob/intel-python/python3/lib/python3.6/shutil.py", line 436, in _rmtree_safe_fd

    os.unlink(name, dir_fd=topfd)

OSError: [Errno 16] Device or resource busy: '.nfs00000038015dac06000002a7'

==================================================================

What can I do to solve this ?

 

2. How can I reduce training time ? Training is slow

During training I use this command

=====================================================================

qsub -I -l nodes=4:ppn=2,walltime=24:00:00,mem=196gb

=====================================================================

But this doesn't help much. What more can I do to get improved performance ?

 

3. I also want to increase the walltime. Training take weeks.

 

4. Earlier I used to use a docker and training would happen in my container. This allowed me to close my laptop. But if training takes place for multiple days, then what do you suggest for tracking the interactive session (similar to counterpart of entering into docker session) Can you suggest a way for this problem ?

 

0 Kudos
21 Replies
AthiraM_Intel
Moderator
116 Views

Hi,


We are closing this case.Please feel free to open a new thread if you have any further queries.


Thanks


Reply