- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dear list,
I want to stop the execution of the command "mpdboot" on the headnode of my cluster .Is there any way to do it so that even if i issue the command "mpdboot" on the head node for any user it will not allow its execution and show me an error message and terminate.
With Regards,
Anish
Fist of all Thanks Tim and Gergana!! for your clarifications and solutions.Tim is absolutely right in saying that I wish that all users will be able to run MPDBOOT on all the nodes in the cluster except the head node.Its the client's requirement:"if mpdboot will not be running on HEAD NODE then no matter where the users will submit the job its not going to run on the HEAD NODE."
Secondly i am not using any scheduler for submitting the jobs,just normal intel commands-mpdboot with hostfile,mpicc and mpiexec with hostfile.
lastly if am specifying -nolocal in"/opt/intel/impi/3.2.2/etc64/mpiexec.conf" then it will consider head node as local or the node from where i will fire mpiexec command as local ?? if second condition is true then in that case the compute node from where i will issue mpiexec command will not execute it on itself.
With Regards,
Anish
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I want to stop the execution of the command "mpdboot" on the headnode of my cluster .Is there any way to do it so that even if i issue the command "mpdboot" on the head node for any user it will not allow its execution and show me an error message and terminate.
With Regards,
Anish
Dear Anish,
I'm afaid I'm not sure what you mean. Could you clarify why you don't want users to start the mpdboot command? I suppose you can change the permissions on the Intel MPI Library install directory (e.g. /opt/intel/impi/) so that no user would be able to access any of the commands.
Or do you mean that you want to kill an mpdboot command that's already running? You can do a simple Ctrl+C, although that might leave some zombie processes running around. The mpdallexit command would kill any mpd daemons started from mpdboot for the given user. Or you can kill an actual MPI job via mpdkilljob. More info is available in the Intel MPI Library Reference Manual.
Or perhaps I misunderstood your question. Let me know if that's the case.
Regards,
~Gergana
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Ah, I see! Thanks for clarifying, Tim.
In this case, I guess changing the permissions would work. Although, any good job schedulershould be able to restrict access to the head node when jobs are submitted. Anish, what type of job scheduler are you using on the cluster?
In reality, what you want is not to prevent mpdboot from running on the head node, but to restrict users from running MPI jobs on the head node. Those are two different things. The mpdboot command would start the mpd daemons on all nodes you will be using, but mpiexec is the command used to run the MPI job. And it's not necessary that all nodes which have mpd daemons started be used for the MPI job.
This is easily done via the -nolocal option for the mpiexec/mpirun command which will tell the Intel MPI Library to exclude the current node when running the job. To make this a global setting for all Intel MPI jobs run on the cluster by all users, you can create an
$ cat /opt/intel/impi/3.2.2/etc64/mpiexec.conf
-nolocal
More details are in the Reference Manual, as noted in my earlier post. I hope this solves your problem. Let us know how it helps.
Regards,
~Gergana
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Ah, I see! Thanks for clarifying, Tim.
In this case, I guess changing the permissions would work. Although, any good job schedulershould be able to restrict access to the head node when jobs are submitted. Anish, what type of job scheduler are you using on the cluster?
In reality, what you want is not to prevent mpdboot from running on the head node, but to restrict users from running MPI jobs on the head node. Those are two different things. The mpdboot command would start the mpd daemons on all nodes you will be using, but mpiexec is the command used to run the MPI job. And it's not necessary that all nodes which have mpd daemons started be used for the MPI job.
This is easily done via the -nolocal option for the mpiexec/mpirun command which will tell the Intel MPI Library to exclude the current node when running the job. To make this a global setting for all Intel MPI jobs run on the cluster by all users, you can create an
$ cat /opt/intel/impi/3.2.2/etc64/mpiexec.conf
-nolocal
More details are in the Reference Manual, as noted in my earlier post. I hope this solves your problem. Let us know how it helps.
Regards,
~Gergana
First of all Thanks Tim and Gergana!! for your clarifications and solutions.Tim is absolutely right in saying that I wish that all users will be able to run MPDBOOT on all the nodes in the cluster except the head node.Its the client's requirement:"if mpdboot will not be running on HEAD NODE then no matter where the users will submit the job its not going to run on the HEAD NODE."
Secondly i am not using any scheduler for submitting the jobs,just normal intel commands-mpdboot with hostfile,mpicc and mpiexec with hostfile.
lastly if am specifying -nolocal in"/opt/intel/impi/3.2.2/etc64/mpiexec.conf" then it will consider head node as local or the node from where i will fire mpiexec command as local ?? if second condition is true then in that case the compute node from where i will issue mpiexec command will not execute it on itself.
With Regards,
Anish
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Anish,
Indeed, you are correct in your 2nd case: the -nolocal option will always exclude the node from where the mpiexec command is started. So, if you're *not* starting from the head node, then the head node will *not* be excluded.
Another suggestion I can make is using a machine file to explicitly specify where you want your job to run. In fact, there're several ways to specify which nodes to include, but no easy way to exclude a specific node. And, when you think about it, it's not really the purpose of the MPI implementation to do job scheduling. That's better left to the professional job schedulers out there :)
These are the options that I see for you right now:
- Change the permissions for the mpdboot command, so that it cannot be run from the head node. Here, though, you need to make sure that an mpdboot command started from one of the compute nodes will not be able to start an mpd daemon on the head node (even if the head node is listed in the hosts file).
- Restrict users so they *always* start mpdboot from the head node and *always* use -nolocal for their mpiexec commands. This can be done via the mpiexec.conf file I described earlier.
- Use a machine file for mpiexec to specify the list of nodes where you want your MPI job to execute. But, here, the user would have to specify the machine file himself.
- Simply install a job scheduler on the system. There are a lot of free/open source ones out there, which are fairly popular (such as Torque - an open source version of PBS*, Lava - an open source version of LSF* from Platform*, and even Sun* Grid Engine (SGE) has an open source version).
That's all I can think of right now. Does anyone else have other ideas?
Regards,
~Gergana
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
These are the options that I see for you right now:
- Change the permissions for the mpdboot command, so that it cannot be run from the head node. Here, though, you need to make sure that an mpdboot command started from one of the compute nodes will not be able to start an mpd daemon on the head node (even if the head node is listed in the hosts file).
- Restrict users so they *always* start mpdboot from the head node and *always* use -nolocal for their mpiexec commands. This can be done via the mpiexec.conf file I described earlier.
- Use a machine file for mpiexec to specify the list of nodes where you want your MPI job to execute. But, here, the user would have to specify the machine file himself.
- Simply install a job scheduler on the system. There are a lot of free/open source ones out there, which are fairly popular (such as Torque - an open source version of PBS*, Lava - an open source version of LSF* from Platform*, and even Sun* Grid Engine (SGE) has an open source version).
That's all I can think of right now. Does anyone else have other ideas?
Regards,
~Gergana
Thanx Gergana,
But i would like to share one of my earlier experience in one of the sites where i was system admin for 1 yr . There it was not possible to run mpdboot from head node for any user even for "root". I found out that the permission for mpdboot command there were the same as being used over here.Only difference is that the mpd version over there was 3.1 and that being used over here is 3.2 and the mpdboot.py for 3.1 is quite different from mpdboot.py for 3.2 .Is it possible that in this version this feature is not there.
Apart from that as you said : Change the permissions for the mpdboot command, so that it cannot be run from the head node : it is possible to change permission (rwx)for users so that it is applicable for particular/group users but how will you restrict it from running on a particular node... The package is installed on a shared directory so whatever changes will be there they will be applicable and same for all nodes.But if it is possible then can you please tell me how??
With Best Regards,
Anish
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanx Gergana,
But i would like to share one of my earlier experience in one of the sites where i was system admin for 1 yr . There it was not possible to run mpdboot from head node for any user even for "root". I found out that the permission for mpdboot command there were the same as being used over here.Only difference is that the mpd version over there was 3.1 and that being used over here is 3.2 and the mpdboot.py for 3.1 is quite different from mpdboot.py for 3.2 .Is it possible that in this version this feature is not there.
With Best Regards,
Anish
The problem has been solved by making changes to the path set in .bash_profile
With Regards,
Anish
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page