Sparse Matrix Multiplication Failed

[Ahmadi__Afshin]

Hello,

I am trying to develop a function that multiplies matrix A and B which are in general format but sparse in nature. These matrices contain complex numbers. My issue is, when I do not use the function and I write everything in the main(), the multiplication works perfectly for any size of array. But when I am using my own function, result is corrupt and most of the time I get random error messages.

Here is what this function does:

1. Convert A and B to CSR format (mkl_zdnscsr).

2. Create CSR handles for A and B using data in step 1 (mkl_sparse_z_create_csr).

3. Multiply A and B using handles in step 2 (mkl_sparse_spmm) and store the output in "result".

My guess is the way I return 'result'  from the function is somehow incorrect because I have checked the output of steps 1 & 2 and they produce correct results.

Any idea what the problem is? I am gonna include the summarized version of my code below for your reference.

Thank you very much in advance.

Afshin

#define ALIGN 128

/* To avoid constantly repeating the part of code that checks different functions' status, using the below macros */

// Check CBLAS function
#define CHECK_SPARSE(function) do { \
          if(function != SPARSE_STATUS_SUCCESS)             \
          {                                                 \
          status = 2;                                       \
          goto memory_free;                                 \
          }                                                 \
} while(0)





int main()
{

... matrices A and B are generated using some data

    MKL_INT stat = 0;

    // This part calls the function to multiply matrices as I discussed.

    // A x B --> csrC
    sparse_matrix_t  csrC = NULL;
    stat = dnmm_sp_CSR_handle(MatrixA, A_Num_of_Rows, A_Num_of_Cols, MatrixA_nonzero, MatrixB, B_Num_of_rows, B_Num_of_cols, MatrixB_nonzero, &csrC);
    printf("\nstat = %i", stat);

    // Now I convert the csrC to 4-array version of CSR.
    MKL_INT rows, cols;
    sparse_index_base_t indexing = 0;
    MKL_INT *columns_C = NULL, *pointerB_C = NULL, *pointerE_C = NULL;
    MKL_Complex16  *values_C = NULL;
    mkl_sparse_z_export_csr(csrC, &indexing, &rows, &cols, &pointerB_C, &pointerE_C, &columns_C, &values_C);

    // Print the number of rows and columns of converted matrix (which are incorrect sizes)
    printf("\nrows = %i , cols = %i", rows, cols);
}



// This function receives two dense matrice, convert them to sparse CSR format, multiply them, and returns the result in CSR handle
int dnmm_sp_CSR_handle(MKL_Complex16 *A, MKL_INT A_rownum, MKL_INT A_colnum, MKL_INT A_nnz, MKL_Complex16 *B, MKL_INT B_rownum, MKL_INT B_colnum, MKL_INT B_nnz, sparse_matrix_t *result) {
	
	// A : Matrix A
	// A_rownum : Number of rows in matrix A
	// A_colnum : Number of columns in matrix A
	// A_nnz : Number of nonzero elements in matrix A
	// B : Matrix B
	// B_rownum : Number of rows in matrix B
	// B_colnum : Number of columns in matrix B
	// B_nnz : Number of nonzero elements in matrix B
	// result : return CSR handle for A x B

	MKL_INT job[8];
	job[0] = 0; // the rectangular matrix A is converted to the CSR format;
	job[1] = 0; // zero-based indexing for the rectangular matrix A is used;
	job[2] = 0; // zero-based indexing for the matrix in CSR format is used;
	job[3] = 2; // whole matrix
	//job[4] // maximum number of the non-zero elements allowed if job[0] = 0
	job[5] = 5; // If job[5]>0, arrays acsr, ia, ja are generated for the output storage. If job[5]=0, only array ia is generated for the output storage.
	MKL_INT info = 0; // If info = 0, execution of mkl_zdnscsr was successful.


	MKL_INT status = 1; // return this value to check the execution status  
	//(1 : successfull, 2: error in sparse functions, 3: error in deallocating memory)
	
	MKL_Complex16 *A_val = (MKL_Complex16 *)mkl_malloc(A_nnz * sizeof(MKL_Complex16), ALIGN);
	MKL_INT *A_col = (MKL_INT *)mkl_malloc(A_nnz * sizeof(MKL_INT), ALIGN);
	MKL_INT *A_row = (MKL_INT *)mkl_malloc( (A_rownum + 1) * sizeof(MKL_INT), ALIGN); // +1 is because we are using 3-aaray variation
	job[4] = A_nnz;
	mkl_zdnscsr(job, &A_rownum, &A_colnum, A, &A_colnum, A_val, A_col, A_row, &info);
		
	MKL_Complex16 *B_val = (MKL_Complex16 *)mkl_malloc(B_nnz * sizeof(MKL_Complex16), ALIGN);
	MKL_INT *B_col = (MKL_INT *)mkl_malloc(B_nnz * sizeof(MKL_INT), ALIGN);
	MKL_INT *B_row = (MKL_INT *)mkl_malloc((B_rownum + 1) * sizeof(MKL_INT), ALIGN); // +1 is because we are using 3-aaray variation
	job[4] = B_nnz;
	mkl_zdnscsr(job, &B_rownum, &B_colnum, B, &B_colnum, B_val, B_col, B_row, &info);
			
	sparse_matrix_t   csrA = NULL, csrB = NULL;
	CHECK_SPARSE( mkl_sparse_z_create_csr(&csrA, SPARSE_INDEX_BASE_ZERO, A_rownum, A_colnum, A_row, A_row + 1, A_col, A_val) );
	CHECK_SPARSE( mkl_sparse_z_create_csr(&csrB, SPARSE_INDEX_BASE_ZERO, B_rownum, B_colnum, B_row, B_row + 1, B_col, B_val) );
	CHECK_SPARSE( mkl_sparse_spmm(SPARSE_OPERATION_NON_TRANSPOSE, csrA, csrB, result) );
	
memory_free:
	
	//Release matrix handle and deallocate arrays for which we allocate memory ourselves.
	if (mkl_sparse_destroy(csrA) != SPARSE_STATUS_SUCCESS) status = 3;
	if (mkl_sparse_destroy(csrB) != SPARSE_STATUS_SUCCESS) status = 3;
	
	//Deallocate arrays for which we allocate memory ourselves.
	mkl_free(A_val); mkl_free(A_col); mkl_free(A_row);
	mkl_free(B_val); mkl_free(B_col); mkl_free(B_row);
	
	return status;
}

[Gennady_F_Intel]

it looks you reused examples from spblasc\sourse folder. At the first glance everything looks correct...  Do you see all functions retutn SPARSE_STATUS_SUCCESS status?  

[Ahmadi__Afshin]

It is not exactly like the examples from the spblas, but I used parts of those codes. Anyway, yes I checked the statuses and they all execute successfully. I also printed out the matrices A and B after line 82 and the values are correct. The problem is in line 86 I guess. The interesting thing is that if I don't use a function and just write everything in the main() routine, it works!

[Ahmadi__Afshin]

I can confirm the problem now. It is the way "result" is returned from the function. If I use mkl_sparse_z_export_csr to export the CSR vectors of "result" inside the function (after line 87), everything looks fine. But if I do the same think inside the main() and right after calling the function (line 37), I get corrupt data.

[MariaZh]

Hello Afshin,

I guess your issue can be resolved be passing pointer to handle for C matrix into dnmm_sp_CSR_handle function, as we allocated memory needed for csrC internally during the call of mkl_sparse_spmm routine. This can also explain why the code was working correctly when placed in main().
Hope this will help!

Best regards,
Maria

[Ahmadi__Afshin]

Hello Maria,

I tried that as well but it still does not work. I modified the code above to show what I did (lines 29 and 46). The spmm operation was unsuccessful in this case.

If I define csrC as a global variable instead of passing it to the dnmm_sp_CSR_handle, the program works just fine.

Any idea what is happening?

Best,

Afshin

[MariaZh]

Well, this works fine for me.

Have you checked info status after mkl_zdnscsr?
Also, can you please clarify what was the returned status for spmm and provide MKL version and at least sizes/nnz for matrices A and B, so I will try to reproduce the behaviour?

Best regards,
Maria

[Ahmadi__Afshin]

Hi Maria,

I was able to find the bug. At line 87, I had a copy of lines 33 to 37 to check where the problem is and apparently this is why the code was not working after I passed pointer to handle for C matrix into dnmm_sp_CSR_handle function. Anyways, I really appreciate your help. Could not resolve this without your guide.
 

Best,

Afshin