topic Hello Ivan, in Intel® oneAPI Math Kernel Library

MKL_pardiso weird behaviour with large matrices

Ivan_K_6 — Mon, 09 Nov 2015 22:44:08 GMT

Hello, everybody.

Still trying to solve very large systems of sparse equations with `mkl pardiso`, single node, multiple thread version. The solver is behaving really well for small systems, but not so much for larger equations. In our case, scalability is essential.

With a large enough system of linear equations (79999 x 79999, 100528321 non-zeros), the sovler returns a vectors of `-nan`s, without reporting any errors. The expected result is provided (`expected-result.txt`).

The example code is provided below. The data is linked to here: https://www.dropbox.com/s/jcvieffrkb7ivag/data.tar.gz?dl=0. It is a `tar.gz` archive with the binary matrix representation, read by the code provided below.

There is an additional strange behaviour of the solver, where the `allocation of internal data structures` in the factorization phase is dominating the runtime. It is taking one user thread (100% user time), and take the majority of the executable's time. This behaviour is similar to that described here: https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/596445. I believe the core of the issue is within `parMETIS`, not `pardiso`.

The code is linked against MKL `11.2.3`, release `composer_xe_2015.3.187`.

Thank you for your assistance. I would be happy to provide additional details.

Ivan, could to try to set

Gennady_F_Intel — Tue, 10 Nov 2015 03:31:24 GMT

Ivan, could to try to set iparm[1] = 0 and/or the latest MKL 11.3.1 and check how it will work?

Hello, Gennady.

Ivan_K_6 — Tue, 10 Nov 2015 20:09:00 GMT

Hello, Gennady.

You are right, this is not a `METIS` issue. The situation with 11.3 and minimum-degree algorithm is the same (aside from being significantly slower on the re-order step).

Please see the log attached.

Just as an aside, the executable, should you want to run it, takes two command line arguments - N (the matrix size) and NZ (the non-zero entries). The parameters for the provided input are: `./solver 79999 100528321`

Thanks!

Hello Ivan,

Gennady_F_Intel — Thu, 12 Nov 2015 09:14:00 GMT

Hello Ivan,

I see two problems here:

1. this matrix is inconsistency with CSR representation. I checked this case with pardiso ( not pardiso_64() ) API with Matrix Checker enabled. here is the log I see on my side:

Major version: 11
Minor version: 3
Update version: 1
Product status: Product
Build: 20151021
Platform: Intel(R) 64 architecture
Processor optimization: Intel(R) Advanced Vector Extensions (Intel(R) AVX) enabled processors
================================================================

*** Error in PARDISO (incorrect input matrix ) error_num= 21

*** Input check: i=79998, ia=50263452, ia[i+1]=0 are incompatible

ERROR during symbolic factorization: 4294967295

2. the second topic is the problem on our side, because when pardiso_64() is used, the matrix checker doesn't work properly when he called from Pardiso_64(). This problem is escalated.

thanks, Gennady

The problem with matrix

Gennady_F_Intel — Mon, 29 Feb 2016 07:15:06 GMT

The problem with matrix checker from Pardiso_64() API, has been fixed in MKL version 11.3 u2 available now.