Scalapack and several tasks

[semyon]

Hello!

While reading MKL Scalapack manual I haven't understood how I can solve several
tasks simultaniously. If I have n processors, I'd like
(1..n/2) processors to solve one linear equation and (n/2+1..n) processors to solve another
equation. I should setup several icontxt? How can I do that?

And more complex situation:
If I have 4 processors, I'd like on the first step solve 2 tasks
Processors No 1,2 - task 1
Processors No 3,4 - task 2
and on the second step next 2 tasks
Processors No 1,3 - task 3
Processors No 2,4 - task 4

Thanks, Semyon.

[Alexander_K_Intel3]

Hello, Semyon,

I'd like to suggest youan example locatedhere. Usinga functionlike in theexample you could form different contexts over different set of processes and then use these contexts to share different tasks. For example:

PROCMAP( ICNTX1, 2, 0, 2, 1, IWRK ) - Creates a context which involves first two processes,
PROCMAP( ICNTX2, 2, 2, 2, 1, IWRK ) - Creates a context which involves 3rd and 4th processes.

Then you could use for example ICNTX1with DESCINIT to create a matrix for the first group of processes.
Sowhen youcall computational routine with the matrix, only processesfrom the contextwill be involved for operations over it.

The PROCMAP from the example doesn't support stride over process to make processes 1 and 3 work on one task while 2 and 4 on another, but you could simply addthis by yourself for example by replacing K = K + 1 with K = K + STRIDE and correcting the check in the beginning: IF (NPROCS-BEGPROC .LT. NPROW*NPCOL) to IF (NPROCS-BEGPROC*STRIDE .LT. NPROW*NPCOL) with obviously adding STRIDE as input parameter.

W.B.R.
Alexander

[semyon]

Quoting - Alexander Kobotov (Intel)

Hello, Semyon,

I'd like to suggest youan example locatedhere. Usinga functionlike in theexample you could form different contexts over different set of processes and then use these contexts to share different tasks. For example:

PROCMAP( ICNTX1, 2, 0, 2, 1, IWRK ) - Creates a context which involves first two processes,
PROCMAP( ICNTX2, 2, 2, 2, 1, IWRK ) - Creates a context which involves 3rd and 4th processes.

Then you could use for example ICNTX1with DESCINIT to create a matrix for the first group of processes.
Sowhen youcall computational routine with the matrix, only processesfrom the contextwill be involved for operations over it.

The PROCMAP from the example doesn't support stride over process to make processes 1 and 3 work on one task while 2 and 4 on another, but you could simply addthis by yourself for example by replacing K = K + 1 with K = K + STRIDE and correcting the check in the beginning: IF (NPROCS-BEGPROC .LT. NPROW*NPCOL) to IF (NPROCS-BEGPROC*STRIDE .LT. NPROW*NPCOL) with obviously adding STRIDE as input parameter.

W.B.R.
Alexander

I tried to use PROCMAP, but found out that blacs_pnum doesn't work, a simple program

program main
integer :: ictxt, nprow, npcol, myrow, mycol
integer :: iam, nprocs
integer :: pnum
call blacs_pinfo(iam, nprocs)
nprow=1
npcol=nprocs
call blacs_get(-1,0,ictxt)
call blacs_gridinit(ictxt,'R',nprow,npcol)
call blacs_gridinfo(ictxt,nprow,npcol,myrow,mycol)
pnum=blacs_pnum(ictxt, myrow, mycol)
write (*,*) 'myrow=',myrow,'mycol=',mycol,'pnum=',pnum
call blacs_gridexit(ictxt)
call blacs_exit(0)
end program main

Results to:

semyon@kepler ~/scalapack $ mpirun -n 4 a.out
myrow= 0 mycol= 0 pnum= 0
myrow= 0 mycol= 1 pnum= 0
myrow= 0 mycol= 2 pnum= 0
myrow= 0 mycol= 3 pnum= 0

I wrote a program without blacs_pnum() but it tells me:

PDDTTRF parameter number 606 had an illegal value

and info=-606

PDDTTRS parameter number 1103 had an illegal value

info=-1103

what does it mean? I expected info variable to be from 1 to 10.

Semyon.

program main
integer :: iam, nprocs
integer :: ictxt, ictxt_temp, ictxt1, ictxt2
integer :: usermap(1,10)
integer :: i, j
integer :: nprow, npcol, nprow0, npcol0
integer :: myrow, mycol
integer :: mr, mc
integer :: GNx, GNy, Nx, Ny
double precision :: phi(100,100)
double precision :: boundl(100), boundr(100), boundu(100), boundd(100)
double precision :: dl(100), du(100), d(100), f(100)
double precision :: dx, dy, tau
integer :: desca(7), descf(7)
double precision :: af(1000), work(1000)
integer :: laf, lwork, Nrhs

call blacs_pinfo(iam, nprocs)

nprow0=1
npcol0=nprocs

nprow = int(sqrt(real(nprocs)))
npcol = nprocs/nprow

mycol=mod(iam,npcol)
myrow=iam/npcol

do i=1,nprow
call blacs_get(0,0,ictxt)
call blacs_gridinit(ictxt, 'R', nprow, npcol)
call blacs_gridinfo(ictxt, nprow, npcol, mr, mc)
! write (*,*) ictxt
do j=1,npcol
usermap(1,j)=npcol*(i-1)+j-1
end do
call blacs_get(ictxt, 10, ictxt_temp)
call blacs_gridmap(ictxt_temp, usermap, 1, 1, npcol)
if ((i-1)==myrow) then
ictxt1=ictxt_temp
end if
end do

do i=1,npcol
call blacs_get(0,0,ictxt)
call blacs_gridinit(ictxt, 'R', nprow, npcol)
call blacs_gridinfo(ictxt, nprow, npcol, mr, mc)
do j=1,nprow
usermap(1,j)=npcol*(j-1)+i-1
end do
call blacs_get(ictxt, 10, ictxt_temp)
call blacs_gridmap(ictxt_temp, usermap, 1, 1, nprow)
if ((i-1)==mycol) then
ictxt2=ictxt_temp
end if
end do

write (*,*) 'iam=',iam,'ictxt1=',ictxt1,'ictxt2=',ictxt2

GNx=10
GNy=10
Nx=GNx/npcol
Ny=GNx/nprow
if(mycol==(npcol-1)) then
Nx=GNx-(npcol-1)*Nx
end if
if(myrow==(nprow-1)) then
Ny=GNy-(nprow-1)*Ny
end if

write (*,*) 'iam=',iam,'Nx=',Nx,'Ny=',Ny

tau=1.0d0
dx=1.0d0
dy=1.0d0

do i=1,Nx
do j=1,Ny
phi(i,j)=0.0
end do
end do

if(mycol==0) then
do j=1,Ny
boundl(j)=1.0
end do
end if

if(mycol==(npcol-1)) then
do j=1,Ny
boundr(j)=0.0
end do
end if

if(myrow==0) then
do i=1,Nx
j=mycol*Nx+i
boundd(i)=1.0-(j-1)*dx*1.0/((NGx-1)*dx)
end do
end if

if(myrow==(nprow-1)) then
do i=1,Nx
j=mycol*Nx+i
boundu(i)=1.0-(j-1)*dx*1.0/((NGx-1)*dx)
end do
end if

desca(1)=501
desca(2)=ictxt1
desca(3)=NGx
desca(4)=Nx
desca(5)=0
desca(6)=0
desca(7)=0

descf(1)=502
descf(2)=ictxt1
descf(3)=NGx
descf(4)=Nx
descf(5)=0
descf(6)=Nx
descf(7)=0

do i=1,Nx
dl(i)=-1.0/(2.0*dx*dx)
d(i)=1/tau+1/(dx*dx)
du(i)=-1.0d0/(2.0d0*dx*dx)
f(i)=0.0
end do

if(myrow==0) then
if(mycol==0) then
d(1)=1.0
du(1)=0.0
f(1)=1.0
end if
else
if(mycol==0) then
d(1)=1.0
du(1)=0.0
f(1)=2.0
end if
end if
if(mycol==(npcol-1)) then
d(Nx)=1.0
dl(Nx)=0.0
f(Nx)=0.0
end if

laf=1000
lwork=1000
Nrhs=1

call pddttrf(GNx, dl, d, du, 1, desca, af, laf, work, lwork, info)
if (info.ne.0) then
write (*,*) 'Error: info=',info
end if

call pddttrs('N', GNx, Nrhs, dl, d, du, 1, desca, f, 1, descf, af, laf, work, lwork, info)
if (info.ne.0) then
write (*,*) 'Error: info=',info
end if

do i=1,Nx
phi(i,myrow)=f(i)
end do

write(*,*) (phi(i,myrow),i=1,Nx)

call blacs_exit(0)

end program main