Intel® Fortran Compiler
Build applications that can scale for the future with optimized code designed for Intel® Xeon® and compatible processors.
29276 Discussions

submitting single-threaded executables via LSF

mad3000
Beginner
‎12-10-2008 06:07 AM
810 Views
Solved Jump to solution

Hi

I'm trying to compile legacy codes with -"nothreads -noreentrancy".

It is to be runned on workstation with 2 Xeon's mounted, so the compiler defaults to "-threads -reentrancy threaded" i suppose.

The reason why i want it single-threaded is because some of the codes requires massive i/o read/writes to the hard drive.

Although the codes seems to be compiled as single-threaded executables, it terminates with "TERM_THREADLIMIT" error after i submit the job via LSF platform:

bsub -T 1 -n 1 ./someshellscript

where someshellscript tells to execute some of the executables, and among those executables only the program called "gtable" succeeds.

Following "gtable" comes "argos" and terminates the whole job with the error.

I'm not sure whether lsf is trying to execute "argos" with multithread but the job ends up with 4 threads running which is exceeding the given limit("-T 1").

Am i missing something?

Here's the makefile i use and etc.

$ cat makefile
IFC32 = /opt/intel/fc/9.1.032/bin/ifort
ICC32 = /opt/intel/cc/9.1.042/bin/icc
HPCFLAG = -nothreads -noreentrancy -align commons,nodcommons,rec4byte -stand f90 -i4 -r8 -double-size 64
BLASFLG = -lblas
all: mophase gij ovlp gtable ciinput cidrt ciftp argos cnvrt scf lstrn cidiag cosoci

mophase:; $(IFC32) -O3 $(HPCFLAG) -o bin/mophase mophase0.f

gij:; $(IFC32) -O3 $(HPCFLAG) -o bin/gij gij.f

ovlp:; $(IFC32) -O3 $(HPCFLAG) -o bin/ovlp ovlp.f

gtable:; $(IFC32) -O3 $(HPCFLAG) -o bin/gtable gtable.f

ciinput:; $(IFC32) -O3 $(HPCFLAG) -o bin/ciinput ciinput.f

cidrt:; $(IFC32) -O3 $(HPCFLAG) -o bin/cidrt cidrt.f

ciftp:; $(IFC32) -O3 $(HPCFLAG) -o bin/ciftp ciftp.f

argos: argos.o mclockintel.o
$(IFC32) $(HPCFLAG) -o bin/argos argos.o mclockintel.o

cnvrt: cnvrt.o mclockintel.o
$(IFC32) $(HPCFLAG) -o bin/cnvrt cnvrt.o mclockintel.o

scf:; $(IFC32) -O1 $(HPCFLAG) -o bin/scf scf.f

lstrn: lstrn.o falloc.o mclockintel.o
$(IFC32) $(HPCFLAG) -o bin/lstrn lstrn.o falloc.o mclockintel.o

cidiag: ciintel.o cimath.o cimiscn.o cinex.o cispob.o cidensi.o iolib.o cimain.o falloc.o mclockintel.o
$(IFC32) -O3 $(BLASFLG) $(HPCFLAG) -o bin/cidiag \
cimain.o ciintel.o cimath.o cimiscn.o cinex.o cispob.o cidensi.o iolib.o falloc.o mclockintel.o

cosoci: ciintel.o cimath.o cimiscn.o cinex.o cispob.o cidensi.o iolib.o cosoci.o falloc.o mclockintel.o
$(IFC32) -O3 $(BLASFLG) $(HPCFLAG) -o bin/cosoci \
cosoci.o ciintel.o cimath.o cimiscn.o cinex.o cispob.o cidensi.o iolib.o falloc.o mclockintel.o

cimain.o:; $(IFC32) -O3 \
-Wl,'-bmaxdata:0x80000000 -bmaxstack:0x40000000' $(HPCFLAG) -c cimain.f

cosoci.o:; $(IFC32) -O3 \
-Wl,'-bmaxdata:0x80000000 -bmaxstack:0x40000000' $(HPCFLAG) -c cosoci.f

ciintel.o:; $(IFC32) -O3 \
-Wl,'-bmaxdata:0x80000000 -bmaxstack:0x40000000' $(HPCFLAG) -c ciintel.f

cimath.o:; $(IFC32) -O3 -Wl,'-bmaxdata:0x80000000 -bmaxstack:0x40000000' $(BLASFLG) $(HPCFLAG) -c cimath.f

cimiscn.o:; $(IFC32) -O3 \
-Wl,'-bmaxdata:0x80000000 -bmaxstack:0x40000000' $(HPCFLAG) -c cimiscn.f

cinex.o:; $(IFC32) -O3 -Wl,'-bmaxdata:0x80000000 -bmaxstack:0x40000000' $(HPCFLAG) -c cinex.f

cispob.o:; $(IFC32) -O3 -Wl,'-bmaxdata:0x80000000 -bmaxstack:0x40000000' $(HPCFLAG) -c cispob.f

cidensi.o:; $(IFC32) -O3 -Wl,'-bmaxdata:0x80000000 -bmaxstack:0x40000000' $(HPCFLAG) -c cidensi.f

iolib.o:; $(IFC32) -O3 \
-Wl,'-bmaxdata:0x80000000 -bmaxstack:0x40000000' $(HPCFLAG) -c iolib.f

argos.o:; $(IFC32) -O1 -Wl,'-bmaxdata:0x80000000' $(HPCFLAG) -c argos.f

cnvrt.o:; $(IFC32) -O3 -Wl,'-bmaxdata:0x80000000' $(HPCFLAG) -c cnvrt.f

scf.o:; $(IFC32) -O3 -Wl,'-bmaxdata:0x80000000' $(HPCFLAG) -c scf.f

falloc.o:; $(ICC32) -us -Wl,'-bmaxdata:0x80000000 -bmaxstack:0x40000000' -c falloc.c

lstrn.o:; $(IFC32) -O3 -Wl,'-bmaxdata:0x80000000' $(HPCFLAG) -c lstrn.f

mclockintel.o:; $(IFC32) -ip -Wl,'-bmaxdata:0x80000000' -c mclockintel.f

$ /opt/intel/fc/9.1.032/bin/ifort -V
Intel Fortran Compiler for 32-bit applications, Version 9.1 Build 20060323Z Package ID: l_fc_p_9.1.032
Copyright (C) 1985-2006 Intel Corporation. All rights reserved.

$ /opt/intel/cc/9.1.042/bin/icc -V
Intel C Compiler for 32-bit applications, Version 9.1 Build 20060706Z Package ID: l_cc_c_9.1.042
Copyright (C) 1985-2006 Intel Corporation. All rights reserved.

$ ldd *
argos:
linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/tls/libm.so.6 (0x002cb000)
libc.so.6 => /lib/tls/libc.so.6 (0x00198000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x002f0000)
libdl.so.2 => /lib/libdl.so.2 (0x002c5000)
/lib/ld-linux.so.2 (0x0017f000)
cidiag:
linux-gate.so.1 => (0xffffe000)
libblas.so.3 => /usr/lib/libblas.so.3 (0x5558a000)
libm.so.6 => /lib/tls/libm.so.6 (0x002cb000)
libc.so.6 => /lib/tls/libc.so.6 (0x00198000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x002f0000)
libdl.so.2 => /lib/libdl.so.2 (0x002c5000)
libg2c.so.0 => /usr/lib/libg2c.so.0 (0x555dc000)
/lib/ld-linux.so.2 (0x0017f000)
cidrt:
linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/tls/libm.so.6 (0x002cb000)
libc.so.6 => /lib/tls/libc.so.6 (0x00198000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x002f0000)
libdl.so.2 => /lib/libdl.so.2 (0x002c5000)
/lib/ld-linux.so.2 (0x0017f000)
ciftp:
linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/tls/libm.so.6 (0x002cb000)
libc.so.6 => /lib/tls/libc.so.6 (0x00198000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x002f0000)
libdl.so.2 => /lib/libdl.so.2 (0x002c5000)
/lib/ld-linux.so.2 (0x0017f000)
ciinput:
linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/tls/libm.so.6 (0x002cb000)
libc.so.6 => /lib/tls/libc.so.6 (0x00198000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x002f0000)
libdl.so.2 => /lib/libdl.so.2 (0x002c5000)
/lib/ld-linux.so.2 (0x0017f000)
cnvrt:
linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/tls/libm.so.6 (0x002cb000)
libc.so.6 => /lib/tls/libc.so.6 (0x00198000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x002f0000)
libdl.so.2 => /lib/libdl.so.2 (0x002c5000)
/lib/ld-linux.so.2 (0x0017f000)
cosoci:
linux-gate.so.1 => (0xffffe000)
libblas.so.3 => /usr/lib/libblas.so.3 (0x5558a000)
libm.so.6 => /lib/tls/libm.so.6 (0x002cb000)
libc.so.6 => /lib/tls/libc.so.6 (0x00198000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x002f0000)
libdl.so.2 => /lib/libdl.so.2 (0x002c5000)
libg2c.so.0 => /usr/lib/libg2c.so.0 (0x555dc000)
/lib/ld-linux.so.2 (0x0017f000)
gij:
linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/tls/libm.so.6 (0x002cb000)
libc.so.6 => /lib/tls/libc.so.6 (0x00198000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x002f0000)
libdl.so.2 => /lib/libdl.so.2 (0x002c5000)
/lib/ld-linux.so.2 (0x0017f000)
gtable:
linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/tls/libm.so.6 (0x002cb000)
libc.so.6 => /lib/tls/libc.so.6 (0x00198000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x002f0000)
libdl.so.2 => /lib/libdl.so.2 (0x002c5000)
/lib/ld-linux.so.2 (0x0017f000)
lstrn:
linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/tls/libm.so.6 (0x002cb000)
libc.so.6 => /lib/tls/libc.so.6 (0x00198000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x002f0000)
libdl.so.2 => /lib/libdl.so.2 (0x002c5000)
/lib/ld-linux.so.2 (0x0017f000)
mophase:
linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/tls/libm.so.6 (0x002cb000)
libc.so.6 => /lib/tls/libc.so.6 (0x00198000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x002f0000)
libdl.so.2 => /lib/libdl.so.2 (0x002c5000)
/lib/ld-linux.so.2 (0x0017f000)
ovlp:
not a dynamic executable
scf:
linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/tls/libm.so.6 (0x002cb000)
libc.so.6 => /lib/tls/libc.so.6 (0x00198000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x002f0000)
libdl.so.2 => /lib/libdl.so.2 (0x002c5000)
/lib/ld-linux.so.2 (0x0017f000)

$ bsub -V
Platform LSF HPC 6.1 for Linux, Jun 15 2005
Copyright 1992-2005 Platform Computing Corporation

$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel Xeon CPU E5345 @ 2.33GHz
stepping : 7
cpu MHz : 2327.506
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl est tm2 cx16 xtpr
bogomips : 4658.28
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel Xeon CPU E5345 @ 2.33GHz
stepping : 7
cpu MHz : 2327.506
cache size : 4096 KB
physical id : 1
siblings : 4
core id : 4
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl est tm2 cx16 xtpr
bogomips : 4655.03
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel Xeon CPU E5345 @ 2.33GHz
stepping : 7
cpu MHz : 2327.506
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl est tm2 cx16 xtpr
bogomips : 4654.98
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel Xeon CPU E5345 @ 2.33GHz
stepping : 7
cpu MHz : 2327.506
cache size : 4096 KB
physical id : 1
siblings : 4
core id : 5
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl est tm2 cx16 xtpr
bogomips : 4655.06
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel Xeon CPU E5345 @ 2.33GHz
stepping : 7
cpu MHz : 2327.506
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 2
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl est tm2 cx16 xtpr
bogomips : 4655.02
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel Xeon CPU E5345 @ 2.33GHz
stepping : 7
cpu MHz : 2327.506
cache size : 4096 KB
physical id : 1
siblings : 4
core id : 6
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl est tm2 cx16 xtpr
bogomips : 4655.04
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel Xeon CPU E5345 @ 2.33GHz
stepping : 7
cpu MHz : 2327.506
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl est tm2 cx16 xtpr
bogomips : 4655.02
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel Xeon CPU E5345 @ 2.33GHz
stepping : 7
cpu MHz : 2327.506
cache size : 4096 KB
physical id : 1
siblings : 4
core id : 7
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl est tm2 cx16 xtpr
bogomips : 4655.04
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

$cat queue.output

Sender: LSF System
Subject: Job 14019: Exited

Job was submitted from host by user .
Job was executed on host(s) , in queue , as user .
 was used as the home directory.
 was used as the working directory.
Started at Wed Dec 10 21:36:38 2008
Results reported at Wed Dec 10 21:37:10 2008

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
#!/bin/csh -f
#BSUB -n 1
#BSUB -T 1
#BSUB -o queue.output
#BSUB -e queue.errors
#BSUB -m "node01 node02 node03 node04 node05 node06"
#BSUB -J rbi-stoll.580
set GEOM=580
set SDIR=rbi-stoll
set EXEC=${HOME}/bin
set MOLD=rbi
set CDIR=${HOME}/${MOLD}/${SDIR}
set SCFD=${CDIR}/scfvecs
set NSYM=( 1 2 4 )
if (! -d /scr/${USER} ) mkdir /scr/${USER}
if (! -d /scr/${USER}/${MOLD} ) mkdir /scr/${USER}/${MOLD}
set OUT=${CDIR}/out${GEOM}
set WORK=/scr/${USER}/${MOLD}/${SDIR}.${GEOM}
if (! -d ${OUT} ) mkdir ${OUT}
if (! -d ${WORK} ) mkdir ${WORK}
rm ${WORK}/*
cp ${EXEC}/* ${WORK}
cp ${CDIR}/*.in ${WORK}
cd ${WORK}
cp ${SCFD}/SCFVECIN.${GEOM} SCFVECIN
echo > dum.in
echo ${GEOM} > GEOM
echo "gtable"
ln -s -f gtable.out tailme
./gtable < gtable.in > gtable.out
foreach SYM ( ${NSYM} )
cat ciinp${SYM}.in ciinp.in > ciinpt${SYM}.in
end
ln -s -f argos.out tailme
echo "argos"
./argos < argos.in > argos.out
ln -s -f cnvrt.out tailme
echo "cnvrt"
./cnvrt < dum.in > cnvrt.out -m
ln -s -f scf.out tailme
echo "scf"
./scf < scf.in > scf.out
ln MOCOEF MOCOEF1
@ INIT = 1
foreach SYM ( ${NSYM} )
ln -s -f ciinp${SYM}.out tailme
echo "ciinput"
./ciinput < ciinpt${SYM}.in > ciinp${SYM}.out
ln -s -f cidrt${SYM}.out tailme
echo "cidrt"

(... more ...)
------------------------------------------------------------

TERM_THREADLIMIT: job killed after reaching LSF thread limit.
Exited with exit code 130.

Resource usage summary:

CPU time : 26.81 sec.
Max Memory : 5 MB
Max Swap : 120 MB

Max Processes : 4
Max Threads : 4

The output (if any) follows:

gtable
argos


PS:

Read file for stderr output of this job.

$cat queue.errors

forrtl: error (69): process interrupted (SIGINT)

0 Kudos
1 Solution
TimP
Honored Contributor III
‎12-10-2008 09:24 AM
810 Views
Solved Jump to solution

While your dual quad core may be suitable for running multiple single threaded jobs, all will slow down progressively, the more jobs you add. If you are trying to maximize throughput, one might guess that it would handle at least 2 jobs, if you have sufficient RAM, noting that all jobs under a 32-bit OS are sharing the same 1GB address segments, As you might guess, a 32-bit OS is unlikely to take full advantage of a dual quad core, even if you have lots of RAM. In any case, there is no guarantee that the jobs would get equal shares of execution time.

I suppose LSF has schemes for queuing up work until sufficient resources are available, and it looks like it puts limits on each job, where you may easily exceed the defaults.

I still believe you're barking up the wrong tree in believing that there must be compile options which would improve the ability to run multiple jobs, or affect the LSF limits.

View solution in original post

0 Kudos
Copy link

Link Copied

5 Replies
TimP
Honored Contributor III
‎12-10-2008 06:29 AM
810 Views
Solved Jump to solution
Quoting - mad3000

I'm trying to compile legacy codes with -"nothreads -noreentrancy".

It is to be runned on workstation with 2 Xeon's mounted, so the compiler defaults to "-threads -reentrancy threaded" i suppose.

The reason why i want it single-threaded is because some of the codes requires massive i/o read/writes to the hard drive.

Although the codes seems to be compiled as single-threaded executables, it terminates with "TERM_THREADLIMIT" error after i submit the job via LSF platform:

No, ifort doesn't make re-entrant code, unless you set an option which implies it, such as -openmp or -parallel. You are grasping at straws, or you're not telling us something important. If your application is running incorrectly, because of bugs due to uninitialized data, and you think it might be papered over by such options, you could set -zero -save. You might get help finding them for correction, by setting -g -O -check -traceback instead.

I can't see how that is specific to LSF, nor how you expect to get advice about LSF here. When LSF says thread, it may be referring to each job as a thread, perhaps an unfortunate choice of terminology. It might well be useful to thread your application, after you correct bugs.

0 Kudos
Copy link
mad3000
Beginner
‎12-10-2008 07:33 AM
810 Views
Solved Jump to solution
tim18, thank you for the reply.

I never thought the meaning of "thread" in LSF would be different until your reply.

And sorry for the long post.

As you mentioned, i don't expect to get any answers about LSF, although i would be glad to get any.

What I was concerned is that the codes might not be compiled as single-threaded executables.

I had tried compiling the codes without "-nothreads -noreentrancy" and it worked.

It worked like a charm when the job is submitted exclusively on the workstation.

But when i submit say 4 jobs on single machine, some of the jobs slows down like 1/10 the speed.

Maybe lacked tolerence but those jobs didn't finish after 30 days while others finished in 2.

The compiler i'm using is 32-bit ifort (the one under /opt/intel/fc not /opt/intel/fce) and the manual of ifort says:

-threads
Specifies that multithreaded libraries should be linked. This is the default on Intel EM64T systems. This option sets the -reentrancy threaded option. On IA-32 and Intel Itanium sys-
tems, the default is -nothreads

I believe EM64T includes Xeon which my workstation has.

Not sure but doesn't this mean that the codes would be compiled with -threads options on?

I just wanted to be sure the codes are being compiled as single-thread.

0 Kudos
Copy link
TimP
Honored Contributor III
‎12-10-2008 09:24 AM
811 Views
Solved Jump to solution

While your dual quad core may be suitable for running multiple single threaded jobs, all will slow down progressively, the more jobs you add. If you are trying to maximize throughput, one might guess that it would handle at least 2 jobs, if you have sufficient RAM, noting that all jobs under a 32-bit OS are sharing the same 1GB address segments, As you might guess, a 32-bit OS is unlikely to take full advantage of a dual quad core, even if you have lots of RAM. In any case, there is no guarantee that the jobs would get equal shares of execution time.

I suppose LSF has schemes for queuing up work until sufficient resources are available, and it looks like it puts limits on each job, where you may easily exceed the defaults.

I still believe you're barking up the wrong tree in believing that there must be compile options which would improve the ability to run multiple jobs, or affect the LSF limits.

0 Kudos
Copy link
mad3000
Beginner
‎12-10-2008 02:11 PM
810 Views
Solved Jump to solution

These codes come from an old IBM machine and gives electronic states of molecules.

There were some hardware restrictions of usable memory, hence it was impossible to calculate molecule with large number of basis sets.

Currently running systems are compatible with 64-bit applications:

$ uname -a
Linux node02 2.6.9-42.ELsmp #1 SMP Tue Aug 15 10:35:26 BST 2006 x86_64 x86_64 x86_64 GNU/Linux

but modifying the codes into 64-bit mode would be a job to be paid for.

The stack memory is resitricted to 2GB when compiled as 32-bit executable but still it means a lot to us.

Maybe the problem lies upon how lsf is running the jobs, as you pointed out.

I'd like to post results when confirmed.

Thanks

0 Kudos
Copy link
mad3000
Beginner
‎12-12-2008 07:41 AM
810 Views
Solved Jump to solution

tim18, you were right.

The thread number of jobs run by LSF seems different to the actual number of threads of executables.

The thread limit -T 1 for LSF job was unnecessary.

Total 8 jobs on same machine were done recording Hog Factor around 0.8, which is quite high comparing to the earlier jobs running executables without -nothreads -noentrancy options.

Thanks again!!

Job <13819>, Job Name , User , Project , Stat
us , Queue , Command <#!/bin/csh -f;#BSUB -o
queue.output;#BSUB -e queue.errors;#BSUB -m node07;#BSUB -
J rbi-stoll.550;set GEOM=550;set SDIR=rbi-stoll;set EXEC=$
{HOME}/bin;set MOLD=rbi;set CDIR=${HOME}/${MOLD}/${SDIR};s
et SCFD=${CDIR}/scfvecs;set NSYM=( >
Thu Dec 11 18:50:19: Submitted from host , CWD <$
HOME/rbi/rbi-stoll>, Output File , Error Fil
e ;
Thu Dec 11 18:51:43: Dispatched to ;
Fri Dec 12 17:17:00: Completed .

Accounting information about this job:
CPU_T WAIT TURNAROUND STATUS HOG_FACTOR MEM SWAP
65496.03 84 80801 done 0.8106 1166M 2899M
------------------------------------------------------------------------------

Job <13818>, Job Name , User , Project , Stat
us , Queue , Command <#!/bin/csh -f;#BSUB -o
queue.output;#BSUB -e queue.errors;#BSUB -m node07;#BSUB -
J rbi-stoll.540;set GEOM=540;set SDIR=rbi-stoll;set EXEC=$
{HOME}/bin;set MOLD=rbi;set CDIR=${HOME}/${MOLD}/${SDIR};s
et SCFD=${CDIR}/scfvecs;set NSYM=( >
Thu Dec 11 18:50:19: Submitted from host , CWD <$
HOME/rbi/rbi-stoll>, Output File , Error Fil
e ;
Thu Dec 11 18:51:22: Dispatched to ;
Fri Dec 12 18:00:29: Completed .

Accounting information about this job:
CPU_T WAIT TURNAROUND STATUS HOG_FACTOR MEM SWAP
68254.70 63 83410 done 0.8183 1166M 2899M
------------------------------------------------------------------------------

Job <13816>, Job Name , User , Project , Stat
us , Queue , Command <#!/bin/csh -f;#BSUB -o
queue.output;#BSUB -e queue.errors;#BSUB -m node07;#BSUB -
J rbi-stoll.520;set GEOM=520;set SDIR=rbi-stoll;set EXEC=$
{HOME}/bin;set MOLD=rbi;set CDIR=${HOME}/${MOLD}/${SDIR};s
et SCFD=${CDIR}/scfvecs;set NSYM=( >
Thu Dec 11 18:50:19: Submitted from host , CWD <$
HOME/rbi/rbi-stoll>, Output File , Error Fil
e ;
Thu Dec 11 18:50:42: Dispatched to ;
Fri Dec 12 18:13:22: Completed .

Accounting information about this job:
CPU_T WAIT TURNAROUND STATUS HOG_FACTOR MEM SWAP
68328.21 23 84183 done 0.8117 1166M 2899M
------------------------------------------------------------------------------

Job <13821>, Job Name , User , Project , Stat
us , Queue , Command <#!/bin/csh -f;#BSUB -o
queue.output;#BSUB -e queue.errors;#BSUB -m node07;#BSUB -
J rbi-stoll.570;set GEOM=570;set SDIR=rbi-stoll;set EXEC=$
{HOME}/bin;set MOLD=rbi;set CDIR=${HOME}/${MOLD}/${SDIR};s
et SCFD=${CDIR}/scfvecs;set NSYM=( >
Thu Dec 11 18:50:19: Submitted from host , CWD <$
HOME/rbi/rbi-stoll>, Output File , Error Fil
e ;
Thu Dec 11 18:52:23: Dispatched to ;
Fri Dec 12 18:18:20: Completed .

Accounting information about this job:
CPU_T WAIT TURNAROUND STATUS HOG_FACTOR MEM SWAP
67642.54 124 84481 done 0.8007 1166M 2899M
------------------------------------------------------------------------------

Job <13822>, Job Name , User , Project , Stat
us , Queue , Command <#!/bin/csh -f;#BSUB -o
queue.output;#BSUB -e queue.errors;#BSUB -m node07;#BSUB -
J rbi-stoll.580;set GEOM=580;set SDIR=rbi-stoll;set EXEC=$
{HOME}/bin;set MOLD=rbi;set CDIR=${HOME}/${MOLD}/${SDIR};s
et SCFD=${CDIR}/scfvecs;set NSYM=( >
Thu Dec 11 18:50:19: Submitted from host , CWD <$
HOME/rbi/rbi-stoll>, Output File , Error Fil
e ;
Thu Dec 11 18:52:43: Dispatched to ;
Fri Dec 12 18:19:09: Completed .

Accounting information about this job:
CPU_T WAIT TURNAROUND STATUS HOG_FACTOR MEM SWAP
67075.42 144 84530 done 0.7935 1166M 2899M
------------------------------------------------------------------------------

Job <13820>, Job Name , User , Project , Stat
us , Queue , Command <#!/bin/csh -f;#BSUB -o
queue.output;#BSUB -e queue.errors;#BSUB -m node07;#BSUB -
J rbi-stoll.560;set GEOM=560;set SDIR=rbi-stoll;set EXEC=$
{HOME}/bin;set MOLD=rbi;set CDIR=${HOME}/${MOLD}/${SDIR};s
et SCFD=${CDIR}/scfvecs;set NSYM=( >
Thu Dec 11 18:50:19: Submitted from host , CWD <$
HOME/rbi/rbi-stoll>, Output File , Error Fil
e ;
Thu Dec 11 18:52:03: Dispatched to ;
Fri Dec 12 18:21:32: Completed .

Accounting information about this job:
CPU_T WAIT TURNAROUND STATUS HOG_FACTOR MEM SWAP
68631.46 104 84673 done 0.8105 1166M 2899M
------------------------------------------------------------------------------

Job <13815>, Job Name , User , Project , Stat
us , Queue , Command <#!/bin/csh -f;#BSUB -o
queue.output;#BSUB -e queue.errors;#BSUB -m node07;#BSUB -
J rbi-stoll.510;set GEOM=510;set SDIR=rbi-stoll;set EXEC=$
{HOME}/bin;set MOLD=rbi;set CDIR=${HOME}/${MOLD}/${SDIR};s
et SCFD=${CDIR}/scfvecs;set NSYM=( >
Thu Dec 11 18:50:19: Submitted from host , CWD <$
HOME/rbi/rbi-stoll>, Output File , Error Fil
e ;
Thu Dec 11 18:50:22: Dispatched to ;
Fri Dec 12 19:19:53: Completed .

Accounting information about this job:
CPU_T WAIT TURNAROUND STATUS HOG_FACTOR MEM SWAP
68996.01 3 88174 done 0.7825 1166M 2899M
------------------------------------------------------------------------------

Job <13817>, Job Name , User , Project , Stat
us , Queue , Command <#!/bin/csh -f;#BSUB -o
queue.output;#BSUB -e queue.errors;#BSUB -m node07;#BSUB -
J rbi-stoll.530;set GEOM=530;set SDIR=rbi-stoll;set EXEC=$
{HOME}/bin;set MOLD=rbi;set CDIR=${HOME}/${MOLD}/${SDIR};s
et SCFD=${CDIR}/scfvecs;set NSYM=( >
Thu Dec 11 18:50:19: Submitted from host , CWD <$
HOME/rbi/rbi-stoll>, Output File , Error Fil
e ;
Thu Dec 11 18:51:02: Dispatched to ;
Fri Dec 12 19:24:58: Completed .

Accounting information about this job:
CPU_T WAIT TURNAROUND STATUS HOG_FACTOR MEM SWAP
72868.61 43 88479 done 0.8236 1166M 2899M
------------------------------------------------------------------------------

0 Kudos
Copy link
Reply

Community support is provided Monday to Friday. Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.