topic Pardiso - error with user ordering in Intel® oneAPI Math Kernel Library

Pardiso - error with user ordering

damienhocking — Thu, 04 Mar 2010 19:37:39 GMT

I've been using Pardiso for a while now and I've started testing other ordering techniques. One that I really like is QAMD for quasi-dense rows, because it's very fast. I'm getting errors from Pardiso with a user-generated ordering on an usymmetric matrix. The problem is scalable, Pardiso doesn't fail on the smaller matrices but it does on the larger ones. It fails on the numeric factorisation, and says error -20. This is the 10.2 update 4 Windows release of Intel Pardiso that comes with the 11.1.060 compiler.

I have a few questions.

1) According to the docs, to use your own ordering, you set iparm(5) = 1 and set the perm array before a call to Pardiso with phase = 11. Is that all you need to do? I couldn't find anything about what to do with iparm(2) in this case, so I assumed Pardiso would ignore it.

2) Pardiso wants the perm array as a row permutation, correct? The reordering code I'm using returns a column permutation, so I send it the transpose of A to get an equivalent row ordering.

3) If I'm not doing anything wrong in my calling parameters, I might need to submit the matrix and the permutation vector. What format should they be in?

Damien

Pardiso - error with user ordering

Gennady_F_Intel — Fri, 05 Mar 2010 07:06:44 GMT

Damien,

It would be more useful ( this is the fastest way to find out the cause of the problem) if you submit the whole testcase ( source code + input data).

error == -20? I couldn't find such error numbers in documentation for 10.2 update 4

--Gennady

Pardiso - error with user ordering

Konstantin_A_Intel — Fri, 05 Mar 2010 07:21:13 GMT

Hello Damien,

Let me answer your questions:

1) Yes, you'd done all correctly. In case of user permutation iparm(2) parameter is just ignored.

2) PARDISO uses only symmetrical permutations (even for unsymmetrical matrices), so it means it will be applied to both colimns and rows.

3) As Gennady said, please sent the reference data.

Thank you,

Konstantin

Pardiso - error with user ordering

damienhocking — Fri, 05 Mar 2010 17:10:44 GMT

I'll put together a stand-alone example and attach it. Thanks.

-20 was a typo. It's -2.

Damien

Pardiso - error with user ordering

damienhocking — Fri, 05 Mar 2010 21:31:54 GMT

Here's the data and a small test driver. The symbolic phase dies on memory allocation.

Pardiso - error with user ordering

Gennady_F_Intel — Sun, 07 Mar 2010 07:14:32 GMT

Damien,error == -2 means not enough memory. If you use in-core mode of PARDISO, please try to solve this task by running PARDISO in out-of-core mode ( iparm[59] = 2, and set the MKL_PARDISO_OOC_MAX_CORE_SIZE <= RAM available at your system).

--Gennady

Pardiso - error with user ordering

Gennady_F_Intel — Sun, 07 Mar 2010 14:57:46 GMT

Damien,

I checked your problem on my local system :Core 2Duo system, RAM 2 GB ( but available memory was 1.1 Gb only. Because memory for solution your task (nequations= 800004 and nnz= 2096005) requires much larger memory then 1.1 Gb, PARDISO has been launched in out-of-core mode with MKL_PARDISO_OOC_MAX_CORE_SIZE=1000).

PARDISO solved the task successfully. See the output below:

--Gennady

+++++++++++++++++++++++++++++++++++++

The file .\pardiso_ooc.cfg was not opened

=== PARDISO is running in Out-Of-Core mode, because iparam(60)=2 ===

================ PARDISO: solving a real nonsymmetric system ================

Summary PARDISO: ( reorder to reorder )

================

Times:

======

Time fulladj: 0.232677 s

Time reorder: 0.008773 s

Time symbfct: 5.178061 s

Time malloc : 3.468740 s

Time total : 186.823628 s total - sum: 177.935377 s

Statistics:

===========

#equations: 800004

#non-zeros in A: 2096005

non-zeros in A (): 0.000327

#right-hand sides: 1

#columns for each panel: 128

#independent subgraphs: 0

#supernodes: 784130

size of largest supernode: 8000

number of nonzeros in L 153841611

number of nonzeros in U 151017799

number of nonzeros in L+U 304859410

Reordering completed ... Percentage of computed non-zeros for LL^T factorizatio

0 %

1 %

2 %

3 %

4 %

5 %

6 %

7 %

................................

98 %

99 %

100 %

================ PARDISO: solving a real nonsymmetric system ================

Summary PARDISO: ( factorize to factorize )

================

Times:

======

Time A to LU: 0.000000 s

Factorization: Time for writing to files : 141.430262

Factorization: Time for reading from files : 3901.309578

Time numfct : 6795.462132 s

Time malloc : 6.698081 s

Time total : 6802.782343 s total - sum: 0.622130 s

Statistics:

===========

#equations: 800004

#non-zeros in A: 2096005

non-zeros in A (): 0.000327

#right-hand sides: 1

#columns for each panel: 128

#independent subgraphs: 0

#supernodes: 784130

size of largest supernode: 8000

number of nonzeros in L 153841611

number of nonzeros in U 151017799

number of nonzeros in L+U 304859410

gflop for the numerical factorization: 3072.194687

gflop/s for the numerical factorization: 0.452095

Pardiso - error with user ordering

damienhocking — Sun, 07 Mar 2010 17:38:51 GMT

Wow. 150 *million* nonzeroes in the L and U factors? On a matrix with 2 million nonzeroes? I think that's the largest amount of fill-in I've ever seen on any problem.

I got that ordering from a call to the MUMPS solver. Using the same ordering and factorising with MUMPS it runs in about 0.4 seconds on one processor. There must be something about that ordering that Pardiso doesn't like, or I'm putting the ordering in the wrong way. Pardiso factorises this problem quite quickly with its own internal orderings. MUMPS has AMD and METIS available, I'll extract those orderings from Pardiso and MUMPS and compare them. Something's definitely not right here. If I discover anything useful I'll post back.

Pardiso - error with user ordering

damienhocking — Mon, 08 Mar 2010 16:23:46 GMT

I tested out the other orderings. They're similar but not the same, which makes sense because they're different codebases. Often they start out the same and then diverge. MUMPS returns a symmetric permutation, where perm is the position of variable i in the pivot order. Is that what Pardiso accepts?

Pardiso - error with user ordering - SOLVED

damienhocking — Tue, 09 Mar 2010 00:59:44 GMT

Allllllllrightythen. I found the problem. I'll skip the boring details of how I got to this. Depending on the implementation of the ordering code, the meaning of the symmetric permutation you get back can be different. For example,

perm[1] = 4

perm[2] = 87

perm[3] = 2

...

can mean two things. It can mean variable (or column) 1 permutes to column 4, v2 permutes to column 87, v3 permutes to column 2 etc. Or, it can mean the inverse. Permuted column 1 has the old column 4, permuted 2 has old 87, permuted 3 has old 2 etc. This was the difference. MUMPS is returning the equivalent of the second (inverse) method. Once I flipped that around, Pardiso can factorise that system in under 1 second.

Problem solved. Thank you Gennady and Konstantin for the help.