Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Beginner
52 Views

SCIF error no 6?

Hi,

I'm trying to use the MAGMA library to offload a dsyevd call onto the mic.

the example code MAGMA provided works fine, so I don't believe it to be a problem with MAGMA, however when I try to run my own code with a call to the MAGMA offload function I get the following error from scif

scif_writeto failed a for mic-1 with err 6

I cant find on-line what this error from scif actually means though, so I am struggling to debug as I have no idea what the actual problem is, any insight?

Greg

 

 

0 Kudos
10 Replies
Highlighted
52 Views

There used to be a SCIF API manual that was distributed with the MPSS. It is no longer there but the information is still available, if you know where to look and are a bit persistent. I looked in /usr/include/asm-generic/errno-base.h to find that errno 6 is also known as ENXIO. Then I went to /usr/include/scif.h and looked for scif_writeto and found:

[cpp]

 * scif_writeto - Copy to a remote address space

 *      \param epd              endpoint descriptor

 *      \param loffset          offset in local registered address space

 *                              from which to copy

 *      \param len              length of range to copy

 *      \param roffset          offset in remote registered address space to

 *                              which to copy

 *      \param rma_flags        transfer mode flags 

 *- ENXIO

 * - The range [loffset,loffset+len-1] is invalid for the registered address

 *   space of epd, or,

 * - The range [roffset , roffset + len -1] is invalid for the registered

 *   address space of the peer of epd, or

 * - loffset or roffset is negative

[/cpp]

Now, what this means in terms of dsyevd in the MAGMA  library, I'm not sure. This is the MAGMA library from http://icl.cs.utk.edu/magma/? Perhaps someone more familiar with that library has an idea what is going on here.

As far as the SCIF API manual no longer being available: The MPSS developers would like to discourage end users from calling the SCIF routines directly and instead guide them to the higher level libraries like COI which can give more meaningful error messages. So the MPSS developers would prefer not to distribute the manual but much of the information is available as comments in the header file /usr/include/scif.h for those who are persistent.

 

0 Kudos
Highlighted
Beginner
52 Views

Thanks, yes that is the library.

if anyone on here has used it before and gotten similar errors, please speak up.

0 Kudos
Highlighted
52 Views

Would you be willing to share your code where you set up the memory and partition the work? I have been looking through the MAGMA code and believe that the error occurs in a setmatrix or getmatrix routine.

0 Kudos
Highlighted
Beginner
52 Views

Thanks Francis :)

My code is basically a port of the testing_dsyevd.cpp code provided by MAGMA (which I have also posted here for clarity)

program greg

	use iso_c_binding
	implicit none
	
	interface
		subroutine turn_magma_on() bind(c,name='magma_init')
		end subroutine turn_magma_on

		subroutine host_allocate(a,b) bind(c,name='magma_malloc_pinned')
			import :: c_ptr
			import :: c_int
			import :: c_double

			type(c_ptr) :: a(*)
			integer(c_int), value :: b


		end subroutine

		subroutine dsyevd(jobz, uplo, ny, a, lda, evals, work, lwork, iwork, liwork, info) bind(c,name='magma_dsyevd')

			import :: c_char
			import :: c_int
			import :: c_ptr
			import :: c_double

			integer(c_int), value 	:: jobz
			integer(c_int), value 	:: uplo
			integer(c_int), value   :: ny
			real(c_double)       	:: a(*)
			integer(c_int), value   :: lda
			real(c_double)         	:: evals(*)
			real(c_double)        	:: work(*)
			integer(c_int), value   :: lwork
			real(c_double)     	:: iwork(*)
			integer(c_int), value   :: liwork
			integer(c_int)	 	:: info     	

		end subroutine dsyevd	
	end interface

	integer :: jobz
	integer :: uplo
	integer :: ny
	real(8), allocatable, target :: a(:)
	integer :: lda
	real(8), allocatable, target :: evals(:)
	
	!real :: aux_work
	!integer :: aux_iwork
	
	real(8), allocatable, target :: work(:)
	integer :: lwork
	real(8), allocatable, target :: iwork(:)
	integer :: liwork
	integer :: info

	integer :: queue 
	
	!init stuff
	integer :: ione
        integer :: ISEED(4)
        integer :: n2
        real(8), allocatable, target :: h_A(:)
        
        integer :: i,j
        real(8) :: z
        real(8) :: inc
            
        jobz = 302
        uplo = 121
        ny = 4724
        lda = ny
        info = 0
        queue = -66
        ione = 1
        ISEED = (/0,0,0,1/)
        n2 = ny*ny
        
	call turn_magma_on()
	!Query for workspace sizes
        !call dsyevd(jobz, uplo, ny, c_loc(a), lda, c_loc(evals), c_loc(aux_work),  -1, c_loc(aux_iwork), -1, c_loc(info))
	liwork = 3 + 5*ny
   	lwork = 1 + 6*ny + 2*ny*ny
        
        !regular allocation
        allocate (a(0:(ny*lda)-1), h_A(0:(ny*lda)-1))
        allocate (work(0:lwork-1), iwork(0:liwork-1), evals(0:ny-1))
        
        !crazy allocation
        call host_allocate(c_loc(c_loc(h_A)), ny*ny*8)
	
	call host_allocate(c_loc(c_loc(evals)), ny*8)
	
	call host_allocate(c_loc(c_loc(a)), ny*ny*8)

	call host_allocate(c_loc(c_loc(work)), lwork*8)

	call host_allocate(c_loc(c_loc(iwork)), liwork*4)

	call dlarnv(ione, ISEED, n2, h_A);
	call dlacpy('Full', ny, ny, h_A, ny, a, ny );
               
	call dsyevd (jobz, uplo, ny, a, lda, evals, work, lwork, iwork, liwork, info)
	write (*,*) 'BOOYAH!!!!!!!!!!!!!!!!!!!', info

end program greg

testing_dsyevd.cpp

/*
    -- micMAGMA (version 1.1) --
       Univ. of Tennessee, Knoxville
       Univ. of California, Berkeley
       Univ. of Colorado, Denver
       November 2011

    @precisions normal d -> s

*/

// includes, system
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <math.h>

// includes, project
#include "magma.h"
#include "magma_lapack.h"
#include "testings.h"

#define absv(v1) ((v1)>0? (v1): -(v1))

/* ////////////////////////////////////////////////////////////////////////////
   -- Testing dsyevd
*/
int main( int argc, char** argv)
{
    /* Initialize */
    magma_queue_t  queue;
    magma_device_t device;
    magma_int_t num = 0;
    magma_err_t err;

    err = MAGMA_SUCCESS;
    num = 1;
    
    magma_init();
    //err = magma_get_devices( &device, 1, &num );
    if ( err != MAGMA_SUCCESS or num < 1 ) {
       fprintf( stderr, "magma_get_devices failed: %d\n", err );
       exit(-1);
    }
    //err = magma_queue_create( device, &queue );
    if ( err != MAGMA_SUCCESS ) {
       fprintf( stderr, "magma_queue_create failed: %d\n", err );
       exit(-1);
    }

    real_Double_t   gpu_time, cpu_time;
    double *h_A, *h_R, *h_work;
    double *w1, *w2;
    magma_int_t *iwork;
    magma_int_t N, n2, info, lwork, liwork, lda, aux_iwork[1];
    magma_int_t izero    = 0;
    magma_int_t ione     = 1;
    magma_int_t ISEED[4] = {0,0,0,1};
    double result[3], eps, aux_work[1];
    eps = lapackf77_dlamch( "E" );

    magma_opts opts;
    parse_opts( argc, argv, &opts );
    
    double tol    = opts.tolerance * lapackf77_dlamch("E");
    double tolulp = opts.tolerance * lapackf77_dlamch("P");
    
    if ( opts.check && opts.jobz == MagmaNoVec ) {
        fprintf( stderr, "checking results requires vectors; setting jobz=V (option -JV)\n" );
        opts.jobz = MagmaVec;	
    }
    
    printf("    N   CPU Time (sec)   GPU Time (sec)\n");
    printf("=======================================\n");
    for( int i = 0; i < opts.ntest; ++i ) {
        for( int iter = 0; iter < opts.niter; ++iter ) {
            printf("%i\n",opts.jobz);           
 
            N = opts.nsize;
            n2  = N*N;
            lda = N;
	    fprintf(stderr, "N = %d \n", N);
            /* Query for workspace sizes */
            magma_dsyevd( opts.jobz, opts.uplo,
                          N, h_R, lda, w1,
                          aux_work,  -1,
                          aux_iwork, -1,
                          &info, queue );
	    fprintf(stderr, "N = %d \n", N);
	
            lwork  = (magma_int_t) aux_work[0];
            liwork = aux_iwork[0];
            
            /* Allocate host memory for the matrix */
            TESTING_MALLOC_HOST(    h_A, double, N*lda  );
            
            TESTING_MALLOC_HOST(    w1,  double, N      );
                        
            TESTING_MALLOC_HOST(    w2,  double, N      );
            
            TESTING_MALLOC_HOST(   h_R,  double, N*lda  );
            
            TESTING_MALLOC_HOST(h_work,  double, lwork  );
            
            TESTING_MALLOC_HOST( iwork, magma_int_t, liwork );
            
            exit(0);
            /* Initialize the matrix */
            lapackf77_dlarnv( &ione, ISEED, &n2, h_A );
            lapackf77_dlacpy( MagmaUpperLowerStr, &N, &N, h_A, &lda, h_R, &lda );
	    fprintf(stderr, "3 \n");
            /* warm up run */
            if ( opts.warmup ) {
                //magma_dsyevd( opts.jobz, opts.uplo,
                //              N, h_R, lda, w1,
                //              h_work, lwork,
                //              iwork, liwork,
                //              &info, queue );
                if (info != 0)
                    printf("magma_dsyevd returned error %d: %s.\n",
                           (int) info, magma_strerror( info ));
                lapackf77_dlacpy( MagmaUpperLowerStr, &N, &N, h_A, &lda, h_R, &lda );
            }
            fprintf(stderr, "4 \n");
            /* ====================================================================
               Performs operation using MAGMA
               =================================================================== */
            
	    gpu_time = magma_wtime();
            magma_dsyevd( opts.jobz, opts.uplo,
                          N, h_R, lda, w1,
                          h_work, lwork,
                          iwork, liwork,
                          &info, 55 );
            gpu_time = magma_wtime() - gpu_time;
            if (info != 0)
                printf("magma_dsyevd returned error %d: %s.\n",
                       (int) info, magma_strerror( info ));
            fprintf(stderr, "5 \n");
            if ( opts.check ) {
                /* =====================================================================
                   Check the results following the LAPACK's [zcds]drvst routine.
                   A is factored as A = U S U' and the following 3 tests computed:
                   (1)    | A - U S U' | / ( |A| N )
                   (2)    | I - U'U | / ( N )
                   (3)    | S(with U) - S(w/o U) | / | S |
                   =================================================================== */
                double temp1, temp2;
                
                // tau=NULL is unused since itype=1
                lapackf77_dsyt21( &ione, lapack_const(opts.uplo), &N, &izero,
                                  h_A, &lda,
                                  w1, h_work,
                                  h_R, &lda,
                                  h_R, &lda,
                                  NULL, h_work, &result[0] );
                
                lapackf77_dlacpy( MagmaUpperLowerStr, &N, &N, h_A, &lda, h_R, &lda );
                magma_dsyevd( MagmaNoVec, opts.uplo,
                              N, h_R, lda, w2,
                              h_work, lwork,
                              iwork, liwork,
                              &info, queue );
                if (info != 0)
                    printf("magma_dsyevd returned error %d: %s.\n",
                           (int) info, magma_strerror( info ));
                
                temp1 = temp2 = 0;
                for( int j=0; j<N; j++ ) {
                    temp1 = max(temp1, absv(w1));
                    temp1 = max(temp1, absv(w2));
                    temp2 = max(temp2, absv(w1-w2));
                }
                result[2] = temp2 / (((double)N)*temp1);
            }
            
            /* =====================================================================
               Performs operation using LAPACK
               =================================================================== */
            if ( opts.lapack ) {
                cpu_time = magma_wtime();
                lapackf77_dsyevd( lapack_const(opts.jobz), lapack_const(opts.uplo),
                                  &N, h_A, &lda, w2,
                                  h_work, &lwork,
                                  iwork, &liwork,
                                  &info );
                cpu_time = magma_wtime() - cpu_time;
                if (info != 0)
                    printf("lapackf77_dsyevd returned error %d: %s.\n",
                           (int) info, magma_strerror( info ));
                
                printf("%5d   %7.2f          %7.2f\n",
                       (int) N, cpu_time, gpu_time);
            }
            else {
                printf("%5d     ---            %7.2f\n",
                       (int) N, gpu_time);
            }
            
            /* =====================================================================
               Print execution time
               =================================================================== */
            if ( opts.check ) {
                printf("Testing the factorization A = U S U' for correctness:\n");
                printf("(1)    | A - U S U' | / (|A| N)     = %8.2e%s\n",   result[0]*eps, (result[0]*eps < tol ? "" : "  failed") );
                printf("(2)    | I -   U'U  | /  N          = %8.2e%s\n",   result[1]*eps, (result[1]*eps < tol ? "" : "  failed") );
                printf("(3)    | S(w/ U) - S(w/o U) | / |S| = %8.2e%s\n\n", result[2]    , (result[2]  < tolulp ? "" : "  failed") );
            }
            

            TESTING_FREE_HOST( h_A    );
            TESTING_FREE_HOST( w1     );
            TESTING_FREE_HOST( w2     );
            TESTING_FREE_HOST( iwork  );
            TESTING_FREE_HOST( h_work );
            TESTING_FREE_HOST( h_R    );
        }
        if ( opts.niter > 1 ) {
            printf( "\n" );
        }
    }
    
    /* Shutdown */
    magma_queue_destroy( queue );
    magma_finalize();

    return 0;
}

everything else is out of the box MAGMA.

I must admit I am a little confused as to what you mean when you say "set up the memory and partition the work" in this context, i thought MAGMA dealt with alot of that.

 

0 Kudos
Highlighted
Beginner
52 Views

Thanks Francis :)

My code (attached) is basically a port of the testing_dsyevd.cpp code provided by MAGMA (which I have also attached for clarity)

everything else is out of the box MAGMA.

I must admit I am a little confused as to what you mean when you say "set up the memory and partition the work" in this context, i thought MAGMA dealt with alot of that.

 

0 Kudos
Highlighted
Beginner
52 Views

Hello Greg C. and Frances I am trying to solve the 1D-Burgers equation using Intel MIC and magma-1.3.1. I am using LU decomposition method magma_zgesv_mic() from magma library but got the following same error: scif_readfrom failed for mic-1 with err 6. I am not getting what does this error actually mean. Thanks in advance.

#include <math.h>
#include <stdio.h>
#include<stdlib.h>

#include "flops.h"
#include <magma.h>
#include "magma_lapack.h"
#include "magma_types.h"
#include "testings.h"

real_Double_t t0(real_Double_t space,real_Double_t v)
{
return space/(1+exp((1/(4*v))*(pow(space,2))-0.25));
}

struct node
{
     float function;
   float time;
    float space;
};



int main()

{


magma_init();
magma_device_t device;

    magma_int_t num = 0;


    magma_int_t err;
  magma_queue_t queue;

    err = magma_get_devices( &device, 1, &num );
    if ( err != MAGMA_SUCCESS or num < 1 ) {
        fprintf( stderr, "magma_get_devices failed: %d\n", (int) err );
        exit(-1);
    }
    err = magma_queue_create( device, &queue );
    if ( err != MAGMA_SUCCESS ) {
        fprintf( stderr, "magma_queue_create failed: %d\n", (int) err );
        exit(-1);
    }
 real_Double_t dx=0.1;
    real_Double_t dt=0.1;
    real_Double_t v=1;
    real_Double_t a=0.0;
    real_Double_t b=8.0;
    real_Double_t time=1;
    magma_int_t tti,tsi;
    tti=(time/dt)+1;
    tsi=((b-a)/dx)+1;


    struct node xxx[tti][tsi];
magma_int_t      i,j,k;
    for (i = 0; i < tti; i++)
    {

        for (j = 0; j < tsi; j++)
        {
                if(i==0)
                {
                        xxx.function=t0(a+j*dx,v);
                        xxx.space=a+j*dx;
                        xxx.time=i*dt;
 }
                else
                {
                        if(j==0)
                        {
                                xxx.function=0;
                                xxx.space=a;
                                xxx.time=i*dt;

                        }
                        if(j==tsi-1)
                        {
                                xxx.function=0;
                                xxx.space=a+(j)*dx;
                                xxx.time=i*dt;

                        }
                        else
                        {
                                xxx.function=0;
                                xxx.space=a+j*dx;
                                xxx.time=i*dt;
 }

                }

        }
    }

printf("\n\n1\n\n\n");

    for (i = 0; i < tti; i++)
    {
        for (k = 0; k < tsi; k++)
        {
                printf("\t%f\t",xxx.function);

                }
        printf("\n");

        }
magma_int_t *piv,rows,columns;
rows=columns=tsi;
    magma_int_t  matrix_size,vector_size;
    matrix_size=tsi*tsi;
    vector_size=tsi;
    double *h_B,h_X[tti][tsi];

        double *h_A;

    double *d_A,*d_B;


    TESTING_MALLOC_HOST(h_B,double,vector_size);
    TESTING_MALLOC_HOST(h_A,double,matrix_size);
    TESTING_MALLOC_HOST(piv,magma_int_t,rows);

    TESTING_MALLOC_DEV(d_A,double,matrix_size);
    TESTING_MALLOC_DEV(d_B,double,vector_size);
for (k = 1; k < tti; k++)
    {
//      magma_int_t *piv;
        magma_int_t info;
printf("\n\n3\n\n\n");

        for (i = 0; i <tsi; i++)
                 {
                                float der2;

                                if(i==0 || i==(columns -1))
                                {
                                        der2 = xxx.function;
                                }
                                else
                                {
                                        float temp=xxx[k-1].function;
                                        der2=pow(temp,2)/dx;
                                        der2+=temp/dt;
                                }

                                h_B=der2;
 h_X=0;
                                //this was for h_B and h_X vectors

                                for (j = 0; j < tsi ; ++j)
                     {
                         float p =(-v)/pow(dx,2);
                         float temp2=xxx[k-1].function;
                         float der1=(1/dt) -(2* p) + (temp2/dx);

                         if(i==0 && j==0)
                         {
                                 h_A[i*tsi+j]=1;
                         }
                         else if(i==rows-1 && j==columns-1)
                         {

                                 h_A[i*tsi+j]=1;
                         }

                         else
                         {
                             if(i!=0 && i!=rows-1)
                             {
         h_A[i*tsi+j]=1;
                         }

                         else
                         {
                             if(i!=0 && i!=rows-1)
                             {

                                 h_A[i*tsi+i-1]=p;

                                 h_A[i*tsi+i+1]=p;

                                 h_A[i*tsi+i]=der1;
                             }
                         }
                     }
                 }

        printf("\n\n4\n\n\n");

                magma_dsetmatrix(rows,columns,h_A,0,rows,d_A,0,rows,queue);

                magma_dsetvector(rows,h_B,rows,d_B,rows,queue);

                magma_dgesv_mic(rows,1,d_A,0,rows,piv,d_B,0,rows,&info,queue);

                magma_dgetvector(rows,d_B,rows,h_X,rows,queue);



}

printf("\n\n5\n\n\n");

    for (i = 0; i < tti; i++)
   {
                for (k = 0; k < tsi; k++)
                {
                        printf("\t%f\t",h_X);

                }
                printf("\n");

        }

printf("\n\n6\n\n\n");

          TESTING_FREE_HOST( h_A );
            TESTING_FREE_HOST( h_B );
            TESTING_FREE_HOST( h_X );
           TESTING_FREE_HOST(piv );





    TESTING_FREE_DEV(d_A);
    TESTING_FREE_DEV(d_B);





  magma_queue_destroy( queue );
    magma_finalize();

printf("\n\n7\n\n\n");



return 0;
}




 

0 Kudos
Highlighted
52 Views

Hitesh,

Can you tell me in which call to the magma library the error occurred?

 

0 Kudos
Highlighted
Beginner
52 Views

Thanks Frances..!!  My Magma code on Intel MIC using magma_zgesv_mic() function(i.e., LU decomposition) is working fine. But it's giving correct output for small range problems and if I increase the time or spatial size it producing NAN as an output. Everything seems to be correct I dont know what happening wrong.

0 Kudos
Highlighted
52 Views

Hitesh,

Are you still getting the SCIF error or has that problem been replaced by the NANs.? Does your code produce good results when run just on the host? Is the problem data or just size/time dependant?

0 Kudos
Highlighted
Beginner
52 Views

Hi Frances

SCIF error has been resolved. Now I am facing problem with magma_zgesv_mic() function. On the host part I am running the same code for LU factorization using gsl library and its producing correct output. But using MAGMA library on Intel MIC its producing NaN output. 

Thank for your time and consideration.

0 Kudos