A strange segment fault of mkl_sparse_spmm.

[xulei_ict]

I am new to using MKL. Maybe my call to the relevant interface is wrong.

The following code results in a segment error and I cannot understand it. The code is mainly testing the multiplication of two sparse matrices. This matrix is of size 181314*181314. It serves as the input matrix A. The input matrix B also comes from it. But I cut it and selected only some of the columns to form input matrix B. As in lines 67-68 of the code, I reduced the number of columns. If I understand the CSC format correctly, this should be the correct way to do it. The code on line 67 even works correctly when the B matrix is reduced to just one column. However, if I use the code in line 68, I get an error.

My compile command is:

/opt/intel/oneapi/compiler/2023.0.0/linux/bin/intel64/icpc  -I/opt/intel/oneapi/vtune/latest/sdk/include -isystem /opt/intel/oneapi/mkl/2023.0.0/include  -D_INTEL_ITT -diag-disable=10441 -O3 -DNDEBUG -DMKL_ILP64  ./mkl_spmm_test.cpp -Wl,-rpath,/usr/local/lib:/opt/intel/oneapi/mkl/2023.0.0/lib/intel64:/opt/intel/oneapi/compiler/latest/linux/compiler/lib/intel64 /usr/local/lib/libgsl.so -Wl,-rpath=/opt/intel/oneapi/mkl/2023.0.0/lib/intel64 /opt/intel/oneapi/mkl/2023.0.0/lib/intel64/libmkl_intel_ilp64.so /opt/intel/oneapi/mkl/2023.0.0/lib/intel64/libmkl_intel_thread.so /opt/intel/oneapi/mkl/2023.0.0/lib/intel64/libmkl_core.so /opt/intel/oneapi/compiler/latest/linux/compiler/lib/intel64/libiomp5.so -lm -ldl -lpthread /usr/local/lib/libgslcblas.so

If you are missing ohne2.mtx file, this code may not work. Please download the matrix from https://sparse.tamu.edu/Schenk_ISEI/ohne2.

Or just run the command to get the .tar file:

wget https://suitesparse-collection-website.herokuapp.com/MM/Schenk_ISEI/ohne2.tar.gz

I would like to know why this is so, any useful information would be important to me.

Thank you very much!

#include <iostream>
#include <iomanip>
#include <vector>

#include <gtest/gtest.h>

#include <mkl_types.h>
#include <mkl_spblas.h>
#include <gsl/gsl_spmatrix.h>
#include <gsl/gsl_spblas.h>

const float m_alpha = 1.0;
const float m_beta = 0.0;
uint32_t print_upto = 5;

int main()
{
    std::string graph_name = "./ohne2.mtx";
    FILE *load_mtx;
    load_mtx = fopen64(graph_name.c_str(), "r");
    gsl_spmatrix *gsl_mat_A = gsl_spmatrix_fscanf(load_mtx);
    fclose(load_mtx);

    uint32_t num_row = gsl_mat_A->size1;
    uint32_t num_col = gsl_mat_A->size2;
    std::cout << "This a " << num_row << "(m) x " << num_col << "(n) Matrix !" << std::endl;
    uint32_t num_nz = gsl_mat_A->nz;

    // build Vector x and Vector b for Ax = b
    double *m_vec_x = new double[num_col];
    double *m_vec_b = new double[num_col];
    for (uint32_t idx_vec = 0; idx_vec < num_col; ++idx_vec)
    {
        m_vec_x[idx_vec] = 1.0;
        m_vec_b[idx_vec] = 0.0;
    }

    MKL_INT *m_row_idx;
    MKL_INT *m_col_idx;
    double *m_mat_ele;
    sparse_matrix_t m_mat_A;
    sparse_status_t m_status;
    matrix_descr m_descr_A;

    // For CSC
    gsl_spmatrix *in_CSC = gsl_spmatrix_ccs(gsl_mat_A);
    MKL_INT *m_idx_ptr = new MKL_INT[num_col + 1];
    for (uint32_t idx_ptr = 0; idx_ptr < num_col + 1; ++idx_ptr)
    {
        m_idx_ptr[idx_ptr] = in_CSC->p[idx_ptr];
    }

    m_row_idx = new MKL_INT[num_nz];
    m_mat_ele = new double[num_nz];
    for (uint32_t idx_nz = 0; idx_nz < num_nz; ++idx_nz)
    {
        m_row_idx[idx_nz] = in_CSC->i[idx_nz];
        m_mat_ele[idx_nz] = in_CSC->data[idx_nz];
    }

    m_status = mkl_sparse_d_create_csc(&m_mat_A, SPARSE_INDEX_BASE_ZERO, num_row, num_col, m_idx_ptr, m_idx_ptr + 1, m_row_idx, m_mat_ele);

    sparse_matrix_t m_mat_B;
    sparse_matrix_t m_mat_C;
    for (size_t i = 0; i <= 10; i++)
    {
        // u_int32_t num_col_B = i == 10 ? 1 : (num_col - i * 20000);
        u_int32_t num_col_B = i == 10 ? 1 : (1343 + i * 20000);
        std::cout << "The num_col of Matrix B = " << num_col_B << std::endl;
        m_status = mkl_sparse_d_create_csc(&m_mat_B, SPARSE_INDEX_BASE_ZERO, num_col, num_col_B, m_idx_ptr, m_idx_ptr + 1, m_row_idx, m_mat_ele);

        std::cout << "MKL CSC\n";
        m_status = mkl_sparse_spmm(sparse_operation_t::SPARSE_OPERATION_NON_TRANSPOSE, m_mat_A, m_mat_B, &m_mat_C);

        // Print some element results.
        MKL_INT output_nrow, output_ncol;
        MKL_INT *output_cols_start, *output_cols_end;
        MKL_INT *output_row_indx;
        double *output_values;
        sparse_index_base_t output_index_base;
        m_status = mkl_sparse_d_export_csc(m_mat_C, &output_index_base, &output_nrow, &output_ncol, &output_cols_start, &output_cols_end, &output_row_indx, &output_values);
        std::cout << "SpMM result:    " << std::endl;
        for (uint32_t iter = 0; iter < print_upto; ++iter)
        {
            std::cout << std::right << std::setw(10) << output_row_indx[iter] << "\t";
        }
        std::cout << std::endl;
        for (uint32_t iter = 0; iter < print_upto; ++iter)
        {
            std::cout << std::right << std::setw(10) << std::scientific << std::setprecision(2) << output_values[iter] << "\t";
        }
        std::cout << std::endl;
        mkl_sparse_destroy(m_mat_C);
    }
    mkl_sparse_destroy(m_mat_A);
    mkl_sparse_destroy(m_mat_B);

    delete[] m_idx_ptr;
    delete[] m_row_idx;
    delete[] m_mat_ele;
    gsl_spmatrix_free(in_CSC);
}

[ShanmukhS_Intel]

Hi XuLei,

Thanks for posting on Intel Communities.

We have tried compiling the sample source code at our environment. However, we are facing a few issues which are related to missing header files as mentioned below.

gtest/gtest.h

gsl/gsl_spblas.h

gsl/gsl_spmatrix.h

We would like to inform you to kindly share the same as it helps us in reproducing the issue at our end and help you further.

Best Regards,

Shanmukh.SS

[xulei_ict]

Hi ShanmukhS_Intel,

Thanks for your reply.

gtest/gtest.h is the header file for the Google Test.  In fact, this code is not involved the Google Test and I forgot to remove it. You can just delete it.

gsl/gsl_spblas.h and gsl/gsl_spmatrix.h are header file of the GNU Scientific Library (GSL). Attached is the GSL that I use. It is very easy to install. Execute the following commands in turn:

./configure
make -j
sudo make install

The compile command above contains the link to the library file.

/usr/local/lib/libgslcblas.so

I hope this will help you to compile successfully.

Best Regards,

XuLei

[ShanmukhS_Intel]

Hi Xu lei,

Thanks for sharing the inputs.  

gtest/gtest.h is the header file for the Google Test. In fact, this code is not involved the Google Test and I forgot to remove it. You can just delete it.

>> We have removed the same

./configure

make -j

sudo make install

>>We'd like to inform you that we have followed the above 2 configuration and make steps. However, we are facing a few issues during the installation step as it requires /usr/local/include/gsl path for installation. We'll let you know in case of any issues despite any workarounds.

Best Regards,

Shanmukh.SS

[ShanmukhS_Intel]

Hi Xu Lei,

The origin of this problem lies in the fact that GSL comes with its own implementation of BLAS routines, but so does Intel oneMKL. When compiling, the GLS and oneMKL headers each declare the BLAS routines. GLS always uses plain int, oneMKL uses MKL_INT. On GNU/Linux, MKL_INT is same as plain int in the LP64 mode, in ILP64 they differ.

When using the ILP64 mode, we end up with two versions of the BLAS functions: one with plain int and one with long int. In a C++ library, this would not be a problem, as overloading function names are allowed. However, the cblas header uses extern "C", so overloading is not allowed and we end up with name clashes which go by unnoticed.

When using LP64 mode, at least the function declarations are identical and there is no ambiguity. However in the linking stage there are two implementations of the functions now, one from GSL and one from oneMKL. It seems that the linker links to the GSL versions, and it seems that these are not thread-safe. So using the multi-threaded oneMKL then calls the non-threadsafe GSL routines and the programme crashes.

The only way to get both working together is using LP64 mode and the sequential version of oneMKL. Could you please try as suggested and let us know in case of any issues?

Best Regards,

Shanmukh.SS

[ShanmukhS_Intel]

Hi XuLei,

A gentle reminder:

Has the information provided helped? Could you please let us know if we could close this case on our end?

Best Regards,

Shanmukh.SS

[ShanmukhS_Intel]

Hi Xu Lei,

We assume that your issue is resolved. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Best Regards,

Shanmukh.SS