Intel® oneAPI Math Kernel Library
Ask questions and share information with other developers who use Intel® Math Kernel Library.
6951 Discussions

c# zgesv and Array Limitations with x64

Gianluca_G_
Beginner
1,252 Views

This post is a continuation of this: https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/597077

I'm using LAPACKE_zgesv (MKL 11.3.01) in C# with arrays of Complex.

I have problems when the arrays exceeded about 12000x12000 elements

.NET Have limitations but I overcomed it with gcAllowVeryLargeObjects enabled="true" directive.

By the way the problem seems to be with LAPACKE_zgesv() or at least something linked with it.
this is the message: "The array size exceed the adresses limits"

For example I have a Matrix Complex[12145,12145] and a vector of Complex[12145], if I apply the LU decomposition (implemented by me) to resolve that system it works. It takes a lot of time but it works.

If we pass these parameters to LAPACKE_zgesv function it doesn't work. Maybe it depends of C# declaration, but with smaller System it works.

I'm using a notebook with windows 10 and 10GB of RAM memory.

I attach the simple example I used to test it.

If you want I have also the files with data but are very large (http://we.tl/ypYNEAk2ZU).

Thank you for any help you can give

 

0 Kudos
10 Replies
Ying_H_Intel
Employee
1,252 Views

Hi Gian, 

I download your MKLTest.zip, but it seems can't build as the below error. 

:\windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.Common.targets(1605,5): warning MSB3270: There was a mismatch between the processor architecture of the project being built "MSIL" and the processor architecture of the reference "System.Data", "AMD64". This mismatch may cause runtime failures. Please consider changing the targeted processor architecture of your project through the Configuration Manager so as to align the processor architectures between your project and references, or take a dependency on references with a processor architecture that matches the targeted processor architecture of your project.
1>C:\windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.Common.targets(2318,5): error MSB3552: Resource file "Properties\Resources.resx" cannot be found.

But anyway, there are two piece of comments

1) i found your zgemm

  MKLWrapper.cblas_zgemm(LAPACK_ROW_MAJOR, CBLAS_NO_TRANS, CBLAS_NO_TRANS, m, n, k, Ref alpha, A, lda, B, ldb, Ref beta, c, ldc

is still use wrong interface. You may see the reply in another post. https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/595035

2) Regarding the memory limitation, 

2.1) as you know, if 32bit application, there is 2G limitation in stack   If X64, there is no the limitation, but not sure if there is limitation when C# call C dll.  I haven't such system with large memory. is it possible for you try the large martix directly in C/CPP code? Please let me know if any problem. 

2.2) Another possibility, As MKL also support ILP64,  int is 64bit, which may support bigger matrix size.  Could you please chang e your code (int to LAPACK_int)

Then go to command  line windows try
>set  MKL_INTERFACE_LAYER=MKL_INTERFACE_ILP64

>your exe. 

Best Regards,

Ying 

 

 

 

 

 

0 Kudos
Gianluca_G_
Beginner
1,252 Views

Hi Ying

I attach the complete code, some file were missing.

1) i found your zgemm ...

I know that, but it is not a problem for me because it works.

2.1) as you know, if 32bit application, there is 2G limitation in stack   If X64, there is no the limitation, but not sure if there is limitation when C# call C dll.  I haven't such system with large memory. is it possible for you try the large martix directly in C/CPP code? Please let me know if any problem. 

I'm not an expert of c++ enviroment and if possible I would like to find a solution directly in C#. I friend of mine tested it in C++ and it worked.

2.2) Another possibility, As MKL also support ILP64,  int is 64bit, which may support bigger matrix size.  Could you please change your code (int to LAPACK_int)

Could you explain me better this?

 

 

Thank you very much

 

Gianluca

 

0 Kudos
Ying_H_Intel
Employee
1,252 Views

Hi Gianluca, 

2.1) if possible I would like to find a solution directly in C#. I friend of mine tested it in C++ and it worked.

Do you mean your friend can solve  very large matrix (http://we.tl/ypYNEAk2ZU, i can't download the matrix from local network). in C++, right? 

If yes, that means MKL work as expected. The problem we need to solve in C# call C dll. 

2.2 MKL support both LP64 and ILP64 model. When ILP64, the mkl_int type is 64bit. So i ask you try this.

>make sure 64bit int type for example long long int or other , which is 64bit in C#. Then change all of MKL int parameter to long int. rebuild your exe. 

then go to command line windows to try 

mkl environment setting 

> mklvars.bat intel64

>set  MKL_INTERFACE_LAYER=MKL_INTERFACE_ILP64

>your.exe

Best Regards,

Ying

0 Kudos
Gianluca_G_
Beginner
1,252 Views

 

Hi Ying,

that is right: we need to solve in C# call C dll. 

 

By the way I'm not expert of C++ enviroment, I don't want to spend time in C++.

What I did was to use the dlls already compiled presents in Parallel Studei XE and installed with it.

The procedure that you suggest me is in order to recopile that dlls in C++?

If so, maybe I need a tutorial because I'm a beginner in this way. I'm sorry, can you drive me to a proper solution?

the sample that I upload was very simple it is not possible to analyze it?

 

 

Thank you very much 

Gianluca

 

 

0 Kudos
Ying_H_Intel
Employee
1,252 Views

Hi Gianluca, 

No, you don't need to work on C++ side. i mentioned is all about your C# code. 

if you can make sure "long int" is 64bit, then change all of int type to "long int" according to the function definition. 

for example, the original function definition as below

lapack_int LAPACKE_zgesv( int matrix_layout, lapack_int n, lapack_int nrhs,
                          lapack_complex_double* a, lapack_int lda,
                          lapack_int* ipiv, lapack_complex_double* b,
                          lapack_int ldb );

So please change your code as below

 [DllImport("mkl_rt.dll", CallingConvention = CallingConvention.Cdecl, ExactSpelling = true, SetLastError = false)]
        internal static extern long int cblas_zgesv(
             int matrix_layout,
           
            long  int n, 
             long int nrsh,

            long int lda

            long int  [] ipiv,

           long int ldb

Rebuild it and run with the environment set i mentioned. 

Best Regards,

Ying 

 

0 Kudos
Gianluca_G_
Beginner
1,252 Views

Dear Ying,

Yes I did. Infact in the example I've attached was declared long. 

It is also possible to pay for an advice in order to resolve? I mean, I just bought "Intel Parallel Studio XE" and I need to work with this kind of large matrix in C#, so I image this kind of problem should be normal for you.

thank you

 

Gianluca

0 Kudos
Ying_H_Intel
Employee
1,252 Views

Hi Gianluca, 

Yes, you can submit an issue by premier.intel.com, while is MKL official support channel. 

General speaking, if it works in C++, then the problem is not MKL internal issue, but it should be fine in C# too. So you may investigate the processing code in details. 

Regarding the error, do you see as below?   I try the matrix on one machine without MSVC C# installed.    From the below message, the problem seems happened in the function LoadMathLabMatrix (mkl dll is even not loaded) . Could you please debug this piece of code and see any issues.     (and could you attach the version with long type as current all are int type.  )

See the end of this message for details on invoking 
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.OutOfMemoryException: Array dimensions exceeded supported range.
   at MKLTest.MainForm.LoadMathLabMatrix(String sFileName, Int32& iMatrixOrder)

   at MKLTest.MainForm.btnMKL_Click(Object sender, EventArgs e)
   at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
   at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.ButtonBase.WndProc(Message& m)
   at System.Windows.Forms.Button.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

Best Regards,

Ying 

0 Kudos
Gianluca_G_
Beginner
1,252 Views

Hi Ying,

Before to compile the program, check if on the Program->Properties, there is a Platform x64 configured. 

You can see it on attached image.

The files loading takes about 15 minutes.

Another thing: you could find problem importing data, because you use a format number that use dot that separate decimal part.

in this case replace lines code like: double.Parse(row[2].Replace('.', ','), System.Globalization.NumberStyles.Float); 

with double.Parse(row[2], System.Globalization.NumberStyles.Float);

 

 

Thks

 

Gianluca

0 Kudos
Gianluca_G_1
Beginner
1,252 Views

 

Hello,

at the end we had a solution: you can find more informations in here

In short I have faced two kind of errors:

1) System.OutOfMemoryException

2) System.ArgumentException: Array size exceedsadressing limitation

Solutions:

1)  The first one is easier to solve because in 64bit platform you have to set the gcAllowVeryLargeObjects = true into the config file App.config

<runtime>
    <gcallowverylargeobjects enabled="true">
  </gcallowverylargeobjects>
 </runtime>

 

2) The second one was the most difficult to understand and depended on how you declare the matrix pointer in C# Marsalling.

This is the right way to declare a huge array

[In, Out] Complex* A,

So this is the code

 [DllImport("mkl_rt.dll", ExactSpelling = true, SetLastError = false, CallingConvention = CallingConvention.Cdecl)]
        internal static extern int LAPACKE_zgesv(
            long matrix_layout,
            long n,
            long nrhs,
            [In, Out] Complex* A,
            long lda,
            [In, Out] long[] ipiv,
            [In, Out] Complex* B,
            long ldb
        );

in order to call the function you have to use the fixed keyword:

fixed (Complex* p1 = &MSL[0,0])
{
  fixed (Complex* p2 = &VTN[0,0])
  {
    info = MKLWrapper.LAPACKE_zgesv(LAPACK_ROW_MAJOR, n, nrhs, p1, lda, ipiv, p2, ldb);
  }
}

 

Best Regards

 

Gianluca

0 Kudos
Ying_H_Intel
Employee
1,252 Views

Hello Gianluca, 

Thank you very much for the summary.  

I also add the thread into the article  https://software.intel.com/en-us/articles/using-intel-mkl-in-your-c-program

Best Regards,

Ying

 

 

0 Kudos
Reply