Re: Understanding the process numbers returned by BLACS_PNUM (ScaLAPACK)

OP1 · ‎09-24-2025

The following code (built with ifx 2025.2.1 on Windows and run with 6 processes) produce somewhat puzzling information, and we would like to get some clarity on the behavior we observe.

The code uses BLACS_GRIDMAP to create a custom 1x4 grid from processes 2, 3, 4 and 5 (thus, process 0 and 1 are not part of that grid). Then a process that is in that grid (process 2) calls BLACS_PNUM to retrieve the numbers of all the processes that are in that grid.

Instead of the expected output of -1, -1, 2, 3, 4, 5, we get 0, 1, 2, 3, -1, -1.

Can somebody explain this? Are process numbers defined within the context of a particular grid, or are they global?

PROGRAM P
IMPLICIT NONE (TYPE, EXTERNAL)
EXTERNAL :: BLACS_GET,  BLACS_PINFO, BLACS_PNUM, BLACS_GRIDMAP, BLACS_GRIDINFO
INTEGER(KIND = 8) :: BLACS_PNUM, CTXT_SYS, C_TXT, MY_PROC, N_PROCS, UMAP(1, 4), I, NPROW, NPCOL, MYROW, MYCOL
CALL BLACS_PINFO(MY_PROC, N_PROCS)
IF (N_PROCS /= 6) THEN
    WRITE(*, '(A)') 'Run this program with 6 processes!'
    STOP
END IF
CALL BLACS_GET(0, 0, CTXT_SYS)
C_TXT = CTXT_SYS
UMAP(1, :) = [2, 3, 4, 5]
CALL BLACS_GRIDMAP(C_TXT, UMAP, 1, 1, 4)
IF (MY_PROC == 2) THEN
    CALL BLACS_GRIDINFO(C_TXT, NPROW, NPCOL, MYROW, MYCOL)
    WRITE(*, *) 'Process number : ', MY_PROC, 'Process row: ', MYROW, 'Process column: ', MYCOL
    WRITE(*, *) 'Other processes: ', (BLACS_PNUM(C_TXT, 0, I), I = 0, N_PROCS - 1)
END IF
END PROGRAM P

sc_Intel · ‎09-25-2025

The output 0, 1, 2, 3, -1, -1 is correct. According to the oneMKL BLACS documentation (https://www.intel.com/content/www/us/en/docs/onemkl/developer-reference-fortran/2025-2/blacs-pnum.html), BLACS_PNUM returns "the system process number of the process at {PROW, PCOL} in the process grid" - meaning the numbering is grid-relative, not global ones.

When you call BLACS_GRIDMAP with processes [2, 3, 4, 5], it creates a new context and remaps them: global process 2 → grid process 0, 3 → 1, and so on. The -1 values indicate invalid positions (your grid is 1×4, so columns 4–5 don’t exist).

This abstraction ensures BLACS operations use clean 0-based grid numbering regardless of the underlying layout.

One correction to the minor issue in the code: INTEGER, EXTERNAL :: BLACS_PNUM instead of INTEGER(KIND=8) :: BLACS_PNUM.

If you need to map between grid and global process numbers, you’ll need to maintain your own mapping array.

OP1 · ‎09-26-2025

Thanks @sc_Intel for the clarification. We will create our own process map to describe the relationship between global process numbers (on which we distribute various kind of data) and local grid(s) process numbers.

Regarding the INTEGER(KIND = declaration, we use the ilp64 interface to the ScaLAPACK library, so isn't it necessary in this case?

sc_Intel · ‎09-27-2025

Thank you for clarifying. My earlier suggestion was based on an LP64 build. For ILP64, 64-bit integers are required, so your original declaration using INTEGER(KIND=8) is correct.

OP1 · ‎09-26-2025

@sc_Intel , in addition to my question above regarding the ilp64 interface, can you explain the logic of the following example code found in the Intel MKL ScaLAPACK documentation here and shown below:

SUBROUTINE PROCMAP(CONTEXT, MAPPING, BEGPROC, NPROW, NPCOL, IMAP) 
*     
*     -- BLACS example code --

*     Written by Clint Whaley 7/26/94 
*     .. 
*     .. Scalar Arguments ..
      INTEGER CONTEXT, MAPPING, BEGPROC, NPROW, NPCOL

*     .. 
*     .. Array Arguments ..
      INTEGER IMAP(NPROW, *) 
*     .. 
*     
*  Purpose 
*  ======= 
*  PROCMAP maps NPROW*NPCOL processes starting from process BEGPROC to   
*  the grid in a variety of ways depending on the parameter MAPPING.

*     
*  Arguments

*  ========= 
*     
*  CONTEXT      (output) INTEGER 
*               This integer is used by the BLACS to indicate a context. 
*               A context is a universe where messages exist and do not 
*               interact with other context's messages.  The context 
*               includes the definition of a grid, and each process's 
*               coordinates in it. 
*     
*  MAPPING      (input) INTEGER 
*               Way to map processes to grid.  Choices are: 
*               1 : row-major natural ordering 
*               2 : column-major natural ordering 
*     
*  BEGPROC      (input) INTEGER 
*               The process number (between 0 and NPROCS-1) to use as 

*               {0,0}. From this process, processes will be assigned 
*               to the grid as indicated by MAPPING. 
*     
*  NPROW        (input) INTEGER 
*               The number of process rows the created grid 

*               should have. 
*                
*  NPCOL        (input) INTEGER 
*               The number of process columns the created grid 

*               should have. 
*     
*  IMAP         (workspace) INTEGER array of dimension (NPROW, NPCOL) 
*               Workspace, where the array which maps the  

*               processes to the grid will be stored for the 
*               call to GRIDMAP. 
*     
*     =============================================================== 
*     
*     .. 
*     .. External Functions ..
      INTEGER  BLACS_PNUM

      EXTERNAL BLACS_PNUM

*     .. 
*     .. External Subroutines ..
      EXTERNAL BLACS_PINFO, BLACS_GRIDINIT, BLACS_GRIDMAP 
*     .. 
*     .. Local Scalars ..
      INTEGER TMPCONTXT, NPROCS, I, J, K

*     .. 
*     .. Executable Statements .. 
*     
*     See how many processes there are in the system 
*     
      CALL BLACS_PINFO( I, NPROCS )

      IF (NPROCS-BEGPROC .LT. NPROW*NPCOL) THEN
         WRITE(*,*) 'Not enough processes for grid'
         STOP
      END IF 
*     
*     Temporarily map all processes into 1 x NPROCS grid


*     
      CALL BLACS_GET( 0, 0, TMPCONTXT )
      CALL BLACS_GRIDINIT( TMPCONTXT, 'Row', 1, NPROCS )
      K = BEGPROC


*     
*     If we want a row-major natural ordering


*     
      IF (MAPPING .EQ. 1) THEN

         DO I = 1, NPROW
            DO J = 1, NPCOL
               IMAP(I, J) = BLACS_PNUM(TMPCONTXT, 0, K)
               K = K + 1W
            END DO
         END DO 
*     
*     If we want a column-major natural ordering


*     
      ELSE IF (MAPPING .EQ. 2) THEN

         DO J = 1, NPCOL
            DO I = 1, NPROW
               IMAP(I, J) = BLACS_PNUM(TMPCONTXT, 0, K)

               K = K + 1

            END DO
         END DO
      ELSE

         WRITE(*,*) 'Unknown mapping.'
         STOP
      END IF

*     
*     Free temporary context 
*     
      CALL BLACS_GRIDEXIT(TMPCONTXT) 
*     
*     Apply the new mapping to form desired context 
*     
      CALL BLACS_GET( 0, 0, CONTEXT )
      CALL BLACS_GRIDMAP( CONTEXT, IMAP, NPROW, NPROW, NPCOL )

      


      RETURN
      END

Ignoring the fact that this code does not compile because of the typo on line 105, could you explain why it is necessary to gather all process numbers into a temporary 1xNPROCS grid? Is it because there is no guarantee that the system process numbers start at 0 and increment linearly from there? Otherwise, IMAP could be trivially constructed, right?

sc_Intel · ‎09-29-2025

Yes, you are right, the temporary 1 × NPROCS grid is needed because BLACS process numbers may not be contiguous or linearly ordered. In many MPI setups, ranks can be assigned arbitrarily, especially in non-default or heterogeneous environments. By creating a temporary grid (TMPCONTXT) with all NPROCS processes mapped to (0, k), we can reliably query each system process number using BLACS_PNUM. This ensures IMAP is correctly populated for BLACS_GRIDMAP. The typo on line 105 (K = K + 1W should be K = K + 1) is a syntax error and doesn't affect this logic.