topic Hi Jens, in Intel® oneAPI Math Kernel Library

ILP64 and LP64 in the same application for sparse matrix operations

Jens_E_ — Thu, 03 Dec 2015 09:31:05 GMT

Hi,

I am working on a c++ application which does solves and operations with sparse matrices. Because the matrices are large, I am using pardiso_64 rather than pardiso. (I am using the LP64 interface.)

The problem is that I also do other things with the matrices, such as mkl_dcsrsymv for matrix-vector products. However there is no "mkl_dcsrsymv_64" and hence I must copy the 64-bit integer matrix index arrays (used in pardiso_64) to 32-bit integer arrays.

In my application matrix-vector products are used extensively, and I do not wish to duplicate the index data permanently. It is also not an option to switch to the ILP64 interface, because my application links in legacy fortran libraries which also links in mkl using the LP64 interface.

I am using the single-layer linking (mkl_rt.dll), mkl 11.2 (parallel studio xe2015) and the visual studio 2010 c++ compiler.

Is there an obvious solution to this, for example by switching between the ILP64 and LP64 interfaces runtime?

Thanks in advance!

Jens

Quote:It is also not an

mecej4 — Thu, 03 Dec 2015 13:53:00 GMT

It is also not an option to switch to the ILP64 interface, because my application links in legacy fortran libraries which also links in mkl using the LP64 interface.

If that is the case, and you are passing the same CSR indices to those routines and to Pardiso, you do not really require the integer arguments to Pardiso to be 8-byte integers, so you could use pardiso() instead of pardiso64(), and you could thus make your entire application adhere to the LP64 model.

Because the matrices are large, I am using pardiso_64 rather than pardiso.

That is not necessary unless the index arrays in the CSR matrix representation are expected to contain index values larger than MAXINT_32. Note that it is possible to have a matrix that uses up more than 4 GB without needing row-index and column values to be 64-bit integers. For example, a dense square matrix of size N X N, with N = MAXINT_32, would fill up 36.9 X 10¹⁸ bytes with the val array, and we could still make do with the LP64 model, in which addresses are 64-bit and integers are 32-bit.

Please justify the need for row-index and column values that must be 64-bit integers.

Thanks for the reply!

Jens_E_ — Thu, 03 Dec 2015 14:14:00 GMT

Thanks for the reply!

I will not require matrix indices that are larger than 32 bit integer. However, the pardiso documentation states that for problems with > 500 million dofs, the pardiso_64 call should be used. My application may in some cases (or in the future) be used for such problems.

So, are you saying that I may use the LP64 model, use the pardiso call (without _64), and have matrices with more than 500million dofs?

I do not know where that

mecej4 — Thu, 03 Dec 2015 14:48:23 GMT

I do not know where that advice regarding > 500 million non-zeros (not unknowns, or 'dofs', as you wrote) came from. The current manual for the Basel/Lugano Pardiso-5 makes no mention of it, and does not provide for passing 8-byte integer arrays to Pardiso.

How dense are your matrices expected to be?

They are matrices for

Jens_E_ — Thu, 03 Dec 2015 14:54:06 GMT

They are matrices for structural analysis FEA. So maybe 50-100 non-zeros per row (regardless of the total number of dofs).

Just read your comment

Jens_E_ — Fri, 04 Dec 2015 07:37:10 GMT

Just read your comment regarding unknowns vs number of non-zeros. 500 million non-zeros is actually not that much, with the sparsity I indicated (for example 10 million degrees of freedom with 50 nnz per row). So it is now very relevant to know for sure whether the "500 million nnz"-comment means anything.

One would think that with 32-bit indices arrays and 64-bit internal addressing, there is no such limit for pardiso, but then there must be a reason why you offer the pardiso_64 routine. Is it only intended for problems with more than MAXINT_32 dofs (2 billion)?

Quote:Jens E. wrote:...but

mecej4 — Fri, 04 Dec 2015 13:32:00 GMT

Jens E. wrote:
...but then there must be a reason why you offer the pardiso_64 routine...

Please note that I have the same status as you -- forum member -- and I do not speak for Intel. Some of the Intel personnel would be better positioned to answer questions regarding the design of the Pardiso interfaces, and I hope that one of them will answer your question in a few days.

By the way, please note that "dof" can be confusing to readers who do not have a background in mechanics or statistics; since "dof" is not an English word but an abbreviation of "degrees of freedom", it is not easy to look up without the dots, and D.O.F. can also stand for "depth of field".

Hi Jens,

Alexander_K_Intel2 — Sun, 06 Dec 2015 03:59:42 GMT

Hi Jens,

Currently we doesn't support calls of SparseBlas functionality wilh long int from lp64 library in same manner with pardiso_64 interface. We think about such kind of interface but doesn't support it now

Thanks,

Alex

Hi Alex,

Jens_E_ — Sun, 06 Dec 2015 08:22:29 GMT

Hi Alex, Thanks for your reply. So to be clear, is it correct that the pardiso routine in LP64 does not use 64 bit internal addressing? How large problems can I solve with LP64 pardiso? How large problems can I solve with LP64 pardiso_64?