Yes, you are right. By default two jobs started on the same node (set of nodes) would have identical process layout even undersubscribed. That leads to jobs sharing same set of processor cores and half of cores standing idle - MPI processes are 'pinned' to the processor cores for performance benefit reasons.
To move apart process of two different jobs on the same node, you'd need to manually set pinning strategy for job running.
Intel MPI Library 3.1.038 provides advanced facilities for intelligent process pining described in section 3.2.2 of Library Reference Manual. The major instrument is I_MPI_PIN_PROCESSOR_LIST variable that defines pining strategy.
For particular case of two 4-rank jobs running on the same 8-core node, pinning strategy would be "allcores:grain=sock,offset=4" meaning "count all processor cores within processor socket boundaries with offset of 4 cores". Respecting command line may be like this:
> mpirun -genv I_MPI_PIN_PROCESSOR_LIST allcores:grain=sock,offset=0 -np 4 job1
> mpirun -genv I_MPI_PIN_PROCESSOR_LIST allcores:grain=sock,offset=4 -np 4 job2
To choose best pinning strategy for your application basing on processor cache sharing, you can use 'cpuinfo' utility shipping with Intel MPI Library and variety of intelligent pinning strategies built in the Library.
You may control actual process pining enabling level 3 of library debug messaging, for example:
> mpirun -genv I_MPI_DEBUG 2 -np 4 job1 | grep CPU
PS: as process pinning interface for older releases differs from the one described, please let me know the version you use so I can provide respective update.
PPS: we do recommend Intel MPI Library build 3.1.038 as it offers higher application performance then ever.
As far as I know, PBS-like job managers can't schedule for cores. They define node count and processes per nodes, but not cores per processes.
In general, MPD daemons can do this cores load balancing for two and more jobs running same node. We'd take this as feature request for Inetl MPI Library.
I am having the same problem. You say:
"The solution you proposed that lock out a job from sharing the nodes assigned to a job is actually possible, but, waste to my point of view".
Could you tell me how to do that?