topic Re:Can dgesv solve 100000x100000 system? in Intel® oneAPI Math Kernel Library

Can dgesv solve 100000x100000 system?

Konstantin4 — Sun, 23 Apr 2023 01:46:40 GMT

I am trying to solve really large system of linear equations 100000x100000 of doubles. I use Visual Studio 2017 with MKL (Package ID: w_comp_lib_2020.1.216), compile code for x64, and I have enough RAM (128 GB). I get error that matrix is singular no matter how I populate the matrix. With smaller matrices the code works fine. I am curious if MKL can solve such large system. n=100000 fits int variable, but the matrix of the system is stored as 1D array of size 100000*100000 which exceeds max number that can be stored in int variable. Integer formal parameters of dgesv have type "int" is there version which accepts "long long"?

Konstantin

Re:Can dgesv solve 100000x100000 system?

PraneethA_Intel — Tue, 25 Apr 2023 05:39:29 GMT

Hi Konstantin,

Thanks for posting in Intel communities.

Could you try using the MKL_INT data type for storing the variable and compiling it with ILP64? As shown in the below link, MKL_INT works as a long long for ILP64.

https://www.intel.com/content/www/us/en/docs/onemkl/developer-reference-c/2023-0/c-datatypes-specific-to-intel-mkl.html

Also, we could see that you are using an older version of MKL. It would be preferable if you upgraded it to the latest Intel oneMKL version.

Thanks and Regards,

Praneeth Achanta

Re:Can dgesv solve 100000x100000 system?

PraneethA_Intel — Tue, 02 May 2023 01:46:10 GMT

Hi Konstantin,

We have not heard back from you. Could you please provide an update on your issue?

Thanks & Regards,

Praneeth Achanta

Re:Can dgesv solve 100000x100000 system?

PraneethA_Intel — Tue, 09 May 2023 05:34:01 GMT

Hi Konstantin,

Hope the information provided helped. We have not heard back from you. This thread will no longer be monitored by Intel. If you need further assistance, please post a new question.

Thanks and Regards,

Praneeth Achanta

Re: Re:Can dgesv solve 100000x100000 system?

Konstantin4 — Tue, 09 May 2023 20:47:19 GMT

1. I used MKL_INT type (see the attached code) without success. File Output.txt contains the output.
If PRECISION_TEST is defined, the code runs a precision test (4x4 system) to verify that the code works correctly.
If PRECISION_TEST is not defined, the code runs a benchmark test with 100000x100000 matrix populated
with random variables. The only trick I used is the following. rand() function returns 7-bit positive random number
(maximum 32767 different numbers). My matrix contains 10^10 elements. If only 32767 will be different,
then there will be a big chance to have a singular matrix with linearly dependent rows or columns. So, I created a longer
"random" (which is not actually random, but it does not matter here) numbers such that most likely all my elements are different.

2. Yes, I use an older version of MKL because the latest version does not work with Visual Studio 2017 that I use.
I use one of the latest MKL versions that works with VS2017. Do you want to say that this version cannot handle
large matrices? If this is the case, I'll switch to VS2019 and the latest MKL.

Thank you.

Konstantin

Re: Re:Can dgesv solve 100000x100000 system?

Konstantin4 — Tue, 09 May 2023 20:52:26 GMT

I could not attach cpp-file