- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I'm trying to use the MAGMA library to offload a dsyevd call onto the mic.
the example code MAGMA provided works fine, so I don't believe it to be a problem with MAGMA, however when I try to run my own code with a call to the MAGMA offload function I get the following error from scif
scif_writeto failed a for mic-1 with err 6
I cant find on-line what this error from scif actually means though, so I am struggling to debug as I have no idea what the actual problem is, any insight?
Greg
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
There used to be a SCIF API manual that was distributed with the MPSS. It is no longer there but the information is still available, if you know where to look and are a bit persistent. I looked in /usr/include/asm-generic/errno-base.h to find that errno 6 is also known as ENXIO. Then I went to /usr/include/scif.h and looked for scif_writeto and found:
[cpp]
* scif_writeto - Copy to a remote address space
* \param epd endpoint descriptor
* \param loffset offset in local registered address space
* from which to copy
* \param len length of range to copy
* \param roffset offset in remote registered address space to
* which to copy
* \param rma_flags transfer mode flags
*- ENXIO
* - The range [loffset,loffset+len-1] is invalid for the registered address
* space of epd, or,
* - The range [roffset , roffset + len -1] is invalid for the registered
* address space of the peer of epd, or
* - loffset or roffset is negative
[/cpp]
Now, what this means in terms of dsyevd in the MAGMA library, I'm not sure. This is the MAGMA library from http://icl.cs.utk.edu/magma/? Perhaps someone more familiar with that library has an idea what is going on here.
As far as the SCIF API manual no longer being available: The MPSS developers would like to discourage end users from calling the SCIF routines directly and instead guide them to the higher level libraries like COI which can give more meaningful error messages. So the MPSS developers would prefer not to distribute the manual but much of the information is available as comments in the header file /usr/include/scif.h for those who are persistent.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks, yes that is the library.
if anyone on here has used it before and gotten similar errors, please speak up.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Would you be willing to share your code where you set up the memory and partition the work? I have been looking through the MAGMA code and believe that the error occurs in a setmatrix or getmatrix routine.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks Francis :)
My code is basically a port of the testing_dsyevd.cpp code provided by MAGMA (which I have also posted here for clarity)
program greg
use iso_c_binding
implicit none
interface
subroutine turn_magma_on() bind(c,name='magma_init')
end subroutine turn_magma_on
subroutine host_allocate(a,b) bind(c,name='magma_malloc_pinned')
import :: c_ptr
import :: c_int
import :: c_double
type(c_ptr) :: a(*)
integer(c_int), value :: b
end subroutine
subroutine dsyevd(jobz, uplo, ny, a, lda, evals, work, lwork, iwork, liwork, info) bind(c,name='magma_dsyevd')
import :: c_char
import :: c_int
import :: c_ptr
import :: c_double
integer(c_int), value :: jobz
integer(c_int), value :: uplo
integer(c_int), value :: ny
real(c_double) :: a(*)
integer(c_int), value :: lda
real(c_double) :: evals(*)
real(c_double) :: work(*)
integer(c_int), value :: lwork
real(c_double) :: iwork(*)
integer(c_int), value :: liwork
integer(c_int) :: info
end subroutine dsyevd
end interface
integer :: jobz
integer :: uplo
integer :: ny
real(8), allocatable, target :: a(:)
integer :: lda
real(8), allocatable, target :: evals(:)
!real :: aux_work
!integer :: aux_iwork
real(8), allocatable, target :: work(:)
integer :: lwork
real(8), allocatable, target :: iwork(:)
integer :: liwork
integer :: info
integer :: queue
!init stuff
integer :: ione
integer :: ISEED(4)
integer :: n2
real(8), allocatable, target :: h_A(:)
integer :: i,j
real(8) :: z
real(8) :: inc
jobz = 302
uplo = 121
ny = 4724
lda = ny
info = 0
queue = -66
ione = 1
ISEED = (/0,0,0,1/)
n2 = ny*ny
call turn_magma_on()
!Query for workspace sizes
!call dsyevd(jobz, uplo, ny, c_loc(a), lda, c_loc(evals), c_loc(aux_work), -1, c_loc(aux_iwork), -1, c_loc(info))
liwork = 3 + 5*ny
lwork = 1 + 6*ny + 2*ny*ny
!regular allocation
allocate (a(0:(ny*lda)-1), h_A(0:(ny*lda)-1))
allocate (work(0:lwork-1), iwork(0:liwork-1), evals(0:ny-1))
!crazy allocation
call host_allocate(c_loc(c_loc(h_A)), ny*ny*8)
call host_allocate(c_loc(c_loc(evals)), ny*8)
call host_allocate(c_loc(c_loc(a)), ny*ny*8)
call host_allocate(c_loc(c_loc(work)), lwork*8)
call host_allocate(c_loc(c_loc(iwork)), liwork*4)
call dlarnv(ione, ISEED, n2, h_A);
call dlacpy('Full', ny, ny, h_A, ny, a, ny );
call dsyevd (jobz, uplo, ny, a, lda, evals, work, lwork, iwork, liwork, info)
write (*,*) 'BOOYAH!!!!!!!!!!!!!!!!!!!', info
end program greg
testing_dsyevd.cpp
/*
-- micMAGMA (version 1.1) --
Univ. of Tennessee, Knoxville
Univ. of California, Berkeley
Univ. of Colorado, Denver
November 2011
@precisions normal d -> s
*/
// includes, system
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <math.h>
// includes, project
#include "magma.h"
#include "magma_lapack.h"
#include "testings.h"
#define absv(v1) ((v1)>0? (v1): -(v1))
/* ////////////////////////////////////////////////////////////////////////////
-- Testing dsyevd
*/
int main( int argc, char** argv)
{
/* Initialize */
magma_queue_t queue;
magma_device_t device;
magma_int_t num = 0;
magma_err_t err;
err = MAGMA_SUCCESS;
num = 1;
magma_init();
//err = magma_get_devices( &device, 1, &num );
if ( err != MAGMA_SUCCESS or num < 1 ) {
fprintf( stderr, "magma_get_devices failed: %d\n", err );
exit(-1);
}
//err = magma_queue_create( device, &queue );
if ( err != MAGMA_SUCCESS ) {
fprintf( stderr, "magma_queue_create failed: %d\n", err );
exit(-1);
}
real_Double_t gpu_time, cpu_time;
double *h_A, *h_R, *h_work;
double *w1, *w2;
magma_int_t *iwork;
magma_int_t N, n2, info, lwork, liwork, lda, aux_iwork[1];
magma_int_t izero = 0;
magma_int_t ione = 1;
magma_int_t ISEED[4] = {0,0,0,1};
double result[3], eps, aux_work[1];
eps = lapackf77_dlamch( "E" );
magma_opts opts;
parse_opts( argc, argv, &opts );
double tol = opts.tolerance * lapackf77_dlamch("E");
double tolulp = opts.tolerance * lapackf77_dlamch("P");
if ( opts.check && opts.jobz == MagmaNoVec ) {
fprintf( stderr, "checking results requires vectors; setting jobz=V (option -JV)\n" );
opts.jobz = MagmaVec;
}
printf(" N CPU Time (sec) GPU Time (sec)\n");
printf("=======================================\n");
for( int i = 0; i < opts.ntest; ++i ) {
for( int iter = 0; iter < opts.niter; ++iter ) {
printf("%i\n",opts.jobz);
N = opts.nsize;
n2 = N*N;
lda = N;
fprintf(stderr, "N = %d \n", N);
/* Query for workspace sizes */
magma_dsyevd( opts.jobz, opts.uplo,
N, h_R, lda, w1,
aux_work, -1,
aux_iwork, -1,
&info, queue );
fprintf(stderr, "N = %d \n", N);
lwork = (magma_int_t) aux_work[0];
liwork = aux_iwork[0];
/* Allocate host memory for the matrix */
TESTING_MALLOC_HOST( h_A, double, N*lda );
TESTING_MALLOC_HOST( w1, double, N );
TESTING_MALLOC_HOST( w2, double, N );
TESTING_MALLOC_HOST( h_R, double, N*lda );
TESTING_MALLOC_HOST(h_work, double, lwork );
TESTING_MALLOC_HOST( iwork, magma_int_t, liwork );
exit(0);
/* Initialize the matrix */
lapackf77_dlarnv( &ione, ISEED, &n2, h_A );
lapackf77_dlacpy( MagmaUpperLowerStr, &N, &N, h_A, &lda, h_R, &lda );
fprintf(stderr, "3 \n");
/* warm up run */
if ( opts.warmup ) {
//magma_dsyevd( opts.jobz, opts.uplo,
// N, h_R, lda, w1,
// h_work, lwork,
// iwork, liwork,
// &info, queue );
if (info != 0)
printf("magma_dsyevd returned error %d: %s.\n",
(int) info, magma_strerror( info ));
lapackf77_dlacpy( MagmaUpperLowerStr, &N, &N, h_A, &lda, h_R, &lda );
}
fprintf(stderr, "4 \n");
/* ====================================================================
Performs operation using MAGMA
=================================================================== */
gpu_time = magma_wtime();
magma_dsyevd( opts.jobz, opts.uplo,
N, h_R, lda, w1,
h_work, lwork,
iwork, liwork,
&info, 55 );
gpu_time = magma_wtime() - gpu_time;
if (info != 0)
printf("magma_dsyevd returned error %d: %s.\n",
(int) info, magma_strerror( info ));
fprintf(stderr, "5 \n");
if ( opts.check ) {
/* =====================================================================
Check the results following the LAPACK's [zcds]drvst routine.
A is factored as A = U S U' and the following 3 tests computed:
(1) | A - U S U' | / ( |A| N )
(2) | I - U'U | / ( N )
(3) | S(with U) - S(w/o U) | / | S |
=================================================================== */
double temp1, temp2;
// tau=NULL is unused since itype=1
lapackf77_dsyt21( &ione, lapack_const(opts.uplo), &N, &izero,
h_A, &lda,
w1, h_work,
h_R, &lda,
h_R, &lda,
NULL, h_work, &result[0] );
lapackf77_dlacpy( MagmaUpperLowerStr, &N, &N, h_A, &lda, h_R, &lda );
magma_dsyevd( MagmaNoVec, opts.uplo,
N, h_R, lda, w2,
h_work, lwork,
iwork, liwork,
&info, queue );
if (info != 0)
printf("magma_dsyevd returned error %d: %s.\n",
(int) info, magma_strerror( info ));
temp1 = temp2 = 0;
for( int j=0; j<N; j++ ) {
temp1 = max(temp1, absv(w1));
temp1 = max(temp1, absv(w2));
temp2 = max(temp2, absv(w1-w2));
}
result[2] = temp2 / (((double)N)*temp1);
}
/* =====================================================================
Performs operation using LAPACK
=================================================================== */
if ( opts.lapack ) {
cpu_time = magma_wtime();
lapackf77_dsyevd( lapack_const(opts.jobz), lapack_const(opts.uplo),
&N, h_A, &lda, w2,
h_work, &lwork,
iwork, &liwork,
&info );
cpu_time = magma_wtime() - cpu_time;
if (info != 0)
printf("lapackf77_dsyevd returned error %d: %s.\n",
(int) info, magma_strerror( info ));
printf("%5d %7.2f %7.2f\n",
(int) N, cpu_time, gpu_time);
}
else {
printf("%5d --- %7.2f\n",
(int) N, gpu_time);
}
/* =====================================================================
Print execution time
=================================================================== */
if ( opts.check ) {
printf("Testing the factorization A = U S U' for correctness:\n");
printf("(1) | A - U S U' | / (|A| N) = %8.2e%s\n", result[0]*eps, (result[0]*eps < tol ? "" : " failed") );
printf("(2) | I - U'U | / N = %8.2e%s\n", result[1]*eps, (result[1]*eps < tol ? "" : " failed") );
printf("(3) | S(w/ U) - S(w/o U) | / |S| = %8.2e%s\n\n", result[2] , (result[2] < tolulp ? "" : " failed") );
}
TESTING_FREE_HOST( h_A );
TESTING_FREE_HOST( w1 );
TESTING_FREE_HOST( w2 );
TESTING_FREE_HOST( iwork );
TESTING_FREE_HOST( h_work );
TESTING_FREE_HOST( h_R );
}
if ( opts.niter > 1 ) {
printf( "\n" );
}
}
/* Shutdown */
magma_queue_destroy( queue );
magma_finalize();
return 0;
}
everything else is out of the box MAGMA.
I must admit I am a little confused as to what you mean when you say "set up the memory and partition the work" in this context, i thought MAGMA dealt with alot of that.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks Francis :)
My code (attached) is basically a port of the testing_dsyevd.cpp code provided by MAGMA (which I have also attached for clarity)
everything else is out of the box MAGMA.
I must admit I am a little confused as to what you mean when you say "set up the memory and partition the work" in this context, i thought MAGMA dealt with alot of that.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Greg C. and Frances I am trying to solve the 1D-Burgers equation using Intel MIC and magma-1.3.1. I am using LU decomposition method magma_zgesv_mic() from magma library but got the following same error: scif_readfrom failed for mic-1 with err 6. I am not getting what does this error actually mean. Thanks in advance.
#include <math.h>
#include <stdio.h>
#include<stdlib.h>
#include "flops.h"
#include <magma.h>
#include "magma_lapack.h"
#include "magma_types.h"
#include "testings.h"
real_Double_t t0(real_Double_t space,real_Double_t v)
{
return space/(1+exp((1/(4*v))*(pow(space,2))-0.25));
}
struct node
{
float function;
float time;
float space;
};
int main()
{
magma_init();
magma_device_t device;
magma_int_t num = 0;
magma_int_t err;
magma_queue_t queue;
err = magma_get_devices( &device, 1, &num );
if ( err != MAGMA_SUCCESS or num < 1 ) {
fprintf( stderr, "magma_get_devices failed: %d\n", (int) err );
exit(-1);
}
err = magma_queue_create( device, &queue );
if ( err != MAGMA_SUCCESS ) {
fprintf( stderr, "magma_queue_create failed: %d\n", (int) err );
exit(-1);
}
real_Double_t dx=0.1;
real_Double_t dt=0.1;
real_Double_t v=1;
real_Double_t a=0.0;
real_Double_t b=8.0;
real_Double_t time=1;
magma_int_t tti,tsi;
tti=(time/dt)+1;
tsi=((b-a)/dx)+1;
struct node xxx[tti][tsi];
magma_int_t i,j,k;
for (i = 0; i < tti; i++)
{
for (j = 0; j < tsi; j++)
{
if(i==0)
{
xxx.function=t0(a+j*dx,v);
xxx.space=a+j*dx;
xxx.time=i*dt;
}
else
{
if(j==0)
{
xxx.function=0;
xxx.space=a;
xxx.time=i*dt;
}
if(j==tsi-1)
{
xxx.function=0;
xxx.space=a+(j)*dx;
xxx.time=i*dt;
}
else
{
xxx.function=0;
xxx.space=a+j*dx;
xxx.time=i*dt;
}
}
}
}
printf("\n\n1\n\n\n");
for (i = 0; i < tti; i++)
{
for (k = 0; k < tsi; k++)
{
printf("\t%f\t",xxx.function);
}
printf("\n");
}
magma_int_t *piv,rows,columns;
rows=columns=tsi;
magma_int_t matrix_size,vector_size;
matrix_size=tsi*tsi;
vector_size=tsi;
double *h_B,h_X[tti][tsi];
double *h_A;
double *d_A,*d_B;
TESTING_MALLOC_HOST(h_B,double,vector_size);
TESTING_MALLOC_HOST(h_A,double,matrix_size);
TESTING_MALLOC_HOST(piv,magma_int_t,rows);
TESTING_MALLOC_DEV(d_A,double,matrix_size);
TESTING_MALLOC_DEV(d_B,double,vector_size);
for (k = 1; k < tti; k++)
{
// magma_int_t *piv;
magma_int_t info;
printf("\n\n3\n\n\n");
for (i = 0; i <tsi; i++)
{
float der2;
if(i==0 || i==(columns -1))
{
der2 = xxx.function;
}
else
{
float temp=xxx[k-1].function;
der2=pow(temp,2)/dx;
der2+=temp/dt;
}
h_B=der2;
h_X=0;
//this was for h_B and h_X vectors
for (j = 0; j < tsi ; ++j)
{
float p =(-v)/pow(dx,2);
float temp2=xxx[k-1].function;
float der1=(1/dt) -(2* p) + (temp2/dx);
if(i==0 && j==0)
{
h_A[i*tsi+j]=1;
}
else if(i==rows-1 && j==columns-1)
{
h_A[i*tsi+j]=1;
}
else
{
if(i!=0 && i!=rows-1)
{
h_A[i*tsi+j]=1;
}
else
{
if(i!=0 && i!=rows-1)
{
h_A[i*tsi+i-1]=p;
h_A[i*tsi+i+1]=p;
h_A[i*tsi+i]=der1;
}
}
}
}
printf("\n\n4\n\n\n");
magma_dsetmatrix(rows,columns,h_A,0,rows,d_A,0,rows,queue);
magma_dsetvector(rows,h_B,rows,d_B,rows,queue);
magma_dgesv_mic(rows,1,d_A,0,rows,piv,d_B,0,rows,&info,queue);
magma_dgetvector(rows,d_B,rows,h_X,rows,queue);
}
printf("\n\n5\n\n\n");
for (i = 0; i < tti; i++)
{
for (k = 0; k < tsi; k++)
{
printf("\t%f\t",h_X);
}
printf("\n");
}
printf("\n\n6\n\n\n");
TESTING_FREE_HOST( h_A );
TESTING_FREE_HOST( h_B );
TESTING_FREE_HOST( h_X );
TESTING_FREE_HOST(piv );
TESTING_FREE_DEV(d_A);
TESTING_FREE_DEV(d_B);
magma_queue_destroy( queue );
magma_finalize();
printf("\n\n7\n\n\n");
return 0;
}
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hitesh,
Can you tell me in which call to the magma library the error occurred?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks Frances..!! My Magma code on Intel MIC using magma_zgesv_mic() function(i.e., LU decomposition) is working fine. But it's giving correct output for small range problems and if I increase the time or spatial size it producing NAN as an output. Everything seems to be correct I dont know what happening wrong.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hitesh,
Are you still getting the SCIF error or has that problem been replaced by the NANs.? Does your code produce good results when run just on the host? Is the problem data or just size/time dependant?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Frances
SCIF error has been resolved. Now I am facing problem with magma_zgesv_mic() function. On the host part I am running the same code for LU factorization using gsl library and its producing correct output. But using MAGMA library on Intel MIC its producing NaN output.
Thank for your time and consideration.
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page