Two Large coexisting Pardiso matrixs crashed

[sapcad]

Dear Sir,

I'm using the dss routines in version 10.0.3.021 of the intel mkl solver with VS2005(C++). My sparse matrix is fairly large and it is needed that two sparse matrixes coexist, which have the same structure. I have calculated several examples without problem, and the maximum rows of the sparse matrixes in these examples are about 70 thousand. However, these days I calculate the example, which sparse matrix has about 210 thousand rows, and the program crashes. I trace the code, and found the factorization of the first matrix is OK, and when computing the factorization of the second, program crashes immediately. Another program with only one sparse matrix at same time can calculate the example correctly. It should not be the problem of the memory. When 1G memory stick is used, the program with one sparse matrix can calculate the example correctly. I have added 4G memory stick on my computer and the problem still exists.
If I solve one matrix and delete it, and then solve another matrix, the program can go on. However, my program calls the dss_solve_real routine several hundred times during execution, deleting, rebuilding and factorizing again and again will be time consuming.
My system is XP 32 bit operating system. 64 bit PARDISO could solve this problem?

Any help would be greatly appreciated. Thank you!

[Sergey_P_Intel2]

Hi!

Your question is not so easy. Despite of PARDISO, DSS doesn't support simultaneous processing of several matrices with one and the same sparse structure. So, it is not possible to examine your situation without working example. Could you provide us with your test case?
If I am not mistaken you want to solve two matrices with one and the same sparse structure at the same time. There are two possible ways for doing this.
First, different matrices with their LU factors can be stored in separate handles. So, you have to reorder, factorize and solve each matrix separately. Of course, in this case memory is used not effectively, because many internal PARDISO arrays are one and the same in both cases.
Second, this problem can be solved by PARDISO which support simultaneous solving of several matrices which have the same sparse structure. This is the best way from the memory utilization point of view. Please see MKL reference manual for details (namely pay attention to maxfct and mnum parameters of PARDISO).

With best regards,
Sergey

[sapcad]

Quoting - Sergey Pudov (Intel)

Hi!

Your question is not so easy. Despite of PARDISO, DSS doesn't support simultaneous processing of several matrices with one and the same sparse structure. So, it is not possible to examine your situation without working example. Could you provide us with your test case?
If I am not mistaken you want to solve two matrices with one and the same sparse structure at the same time. There are two possible ways for doing this.
First, different matrices with their LU factors can be stored in separate handles. So, you have to reorder, factorize and solve each matrix separately. Of course, in this case memory is used not effectively, because many internal PARDISO arrays are one and the same in both cases.
Second, this problem can be solved by PARDISO which support simultaneous solving of several matrices which have the same sparse structure. This is the best way from the memory utilization point of view. Please see MKL reference manual for details (namely pay attention to maxfct and mnum parameters of PARDISO).

With best regards,
Sergey

Dear Sir,

Thank you for your response and proposal.
My main part of the code for sparse matrix solving is as follows. The dss routines are encapsulated in class CSMat. The function defines sparse matrix object and calls the member function of the object to carry out the sparse matrix calculation. It could be seen in the function that two sparse matrixes, smatA and smatB, are defined at same time. When smatA and smatB are not so large, the function passed. When smatA and smatB are large enough, program crashes at the factorization of the second matrixe.
Is there a possibility to get the LU factorization of the input matrix? So I could store it else where, and reset it when doing solving. Only one sparse matrix would exists in this case.

Thank you very much for your help!


////////////////////////////////////////////
class CSMat
{
public:
CSMat(void);
public:
~CSMat(void);

bool Create8DefineStructure(void);
bool Reorder(void);
bool Factor(void);
bool Solve(int nVec,double* adVec);
bool Solve(CMatrix& m);
bool Delete();

CArray m_adEle;
CArray m_aiRow;
CArray m_aiCol;
private:
_MKL_DSS_HANDLE_t m_handle;
bool m_bDefined;
};


bool CSMat::Create8DefineStructure(void)
{
int nRow,nNonZeros;
int opt = MKL_DSS_DEFAULTS;
int sym = MKL_DSS_SYMMETRIC;
int type = MKL_DSS_POSITIVE_DEFINITE;
int error;

nRow=m_aiRow.GetSize()-1;
nNonZeros=m_aiRow[nRow-1];
error = dss_create(m_handle, opt );
if ( error != MKL_DSS_SUCCESS ) return false;

error = dss_define_structure(
m_handle, sym, m_aiRow.GetData(), nRow, nRow,
m_aiCol.GetData(), nNonZeros );
if ( error != MKL_DSS_SUCCESS ) return false;

m_adEle.SetSize(nNonZeros);

m_bDefined=true;
return true;
}



bool CSMat::Reorder(void)
{
int opt = MKL_DSS_DEFAULTS;
int error;

error = dss_reorder(m_handle, opt, 0);
if ( error != MKL_DSS_SUCCESS ) return false;
return true;
}

bool CSMat::Factor(void)
{
int type = MKL_DSS_POSITIVE_DEFINITE;
int error;

error = dss_factor_real(m_handle, type, m_adEle.GetData());
if ( error == MKL_DSS_OUT_OF_MEMORY ){
AfxMessageBox(_T("OUT_OF_MEMORY "),MB_OK,0);
return false;
}

if ( error != MKL_DSS_SUCCESS ){
AfxMessageBox(_T("ERROR"),MB_OK,0);
return false;
}

return true;
}


bool Function( )
{
double *adGK;
CSMat smatA;
CSMat smatB; //smatA and smatB coexist

.

if(!smatA.Create8DefineStructure()){
return false;
}

if(!smatB.Create8DefineStructure()){
return false;
}


//Calculate elements of smatA
adGK=smatA.m_adEle.GetData();
for(loop=0;loop apEle[loop]->AsmbStf(adGK);
}

//Calculate elements of smatB
adGK=smatB.m_adEle.GetData();
for(loop=0;loop apEle[loop]->AsmbStf(adGK);
}


if(!smatA.Reorder())
return false;

if(!smatB.Reorder())
return false;

if(!smatA.Factor()) //Ok!!!
return false;

if(!smatB.Factor()) //Crashed here !!!
return false;

return true;
}

[Sergey_P_Intel2]

Hi!

It is impossible toget LU factorization of the input matrix. It is only possible to work with it through the handle.
Your code at first sight looks fine: each matrix is processed withits ownhandle, so LU factors are stored separately.The possible reason of the crash could be inallocation of input matrices, in DSS interface or PARDISO.Could you provide us withthe test-caseincluding linking line, input matrix generator, number of threads and so on?We will profileit in order to find the root of the problem.

Best regards,
Sergey

[sapcad]

Quoting - Sergey Pudov (Intel)

Hi!

It is impossible toget LU factorization of the input matrix. It is only possible to work with it through the handle.
Your code at first sight looks fine: each matrix is processed withits ownhandle, so LU factors are stored separately.The possible reason of the crash could be inallocation of input matrices, in DSS interface or PARDISO.Could you provide us withthe test-caseincluding linking line, input matrix generator, number of threads and so on?We will profileit in order to find the root of the problem.

Best regards,
Sergey

Dear Sir,

The Linker Command Line is listed as follows.
The input matrix is generated by another program. I thought the input matrix would be no problem, because several examples have been calculated and the results are all correct. If single sparse matrix is used in this unpassed example, the result is also correct.
I didnt set the number of threads, it is said the MKL will set the number of threads automatically.
In this unpassed example, the number of non-zero elements of the input matrix is 5,176,928. In computing the factorization of the first sparse matrix, Peakmem = 180,820k; Factormem = 68009k; Solvemem = 750,799k. Maybe two sparse matrixes of this example are too large to computing the factorization at same phase.
I would like to try em64t MKL.
Please give me more proposals.

Thanks a lot!


Linker Command Line

/OUT:"D:NosaPrj2007WndStaAna_TestReleaseWndStaAna_Test.exe" /INCREMENTAL:NO /NOLOGO /MANIFEST /MANIFESTFILE:"ReleaseWndStaAna_Test.exe.intermediate.manifest" /DEBUG /PDB:"d:NosaPrj2007WndStaAna_TestreleaseWndStaAna_Test.pdb" /SUBSYSTEM:WINDOWS /OPT:REF /OPT:ICF /LTCG /ENTRY:"wWinMainCRTStartup" /MACHINE:X86 /ERRORREPORT:PROMPT D:NosaPrj2007ComClassComClassReleaseComClass.lib c:SenseLockEliteIV_V2.3.2apidllsense4.lib mkl_intel_c_dll.lib mkl_intel_thread_dll.lib mkl_core_dll.lib mkl_solver_sequential.lib

[Sergey_P_Intel2]

I can reproduce the problem on Win32 for 2 coexisting matrices. The test case crushed on the factorization step for the second matrix, whentotal amount of allocated memoryis greater than 2G.On the other hand this test can solve two matrices at the same time, when each of them requires approx. 0.8G of memory. So bymy opinion the problem is in 2G memory limitfor the application on Win32. As I know em64t systems don't have such limitation.

With best regards,
Sergey