- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I am currently porting my host userspace program into a kernel space module.
Most things seem to be the same as in userspace, but I currently have problems with the scif_register function. The API says the following about the return value:
*\return * Upon successful completion, scif_register() returns the offset at which the * mapping was placed (po); otherwise: in user mode SCIF_REGISTER_FAILED (that * is (off_t *)-1) is returned and errno is set to indicate the error; in * kernel mode the negative of one of the following errors is returned.
my code is currently like this
struct scifmodule {
size_t len;
scif_epd_t endpoint;
scif_epd_t remote_endpoint;
size_t port;
uint8_t *ptr;
uint8_t *return_ptr;
uint8_t *placeholder;
};
struct scifmodule a;
static int __init scif_start(void)
{
printk(KERN_INFO "Loading scif module...\n");
printk(KERN_INFO "\n");
a.len = 131072;
size_t ret;
a.port = 23968;
...
a.ptr = kmalloc(a.len,__GFP_NORETRY);
if (!a.ptr)
{
printk(KERN_INFO "PTR allocation failed\n");
kfree(a.ptr);
kfree(a.return_ptr);
kfree(a.placeholder);
scif_close(a.endpoint);
return -1;
}
printk(KERN_INFO "allocating ptr successfull\n");
printk(KERN_INFO "%ld\n", (off_t)a.ptr);
a.return_ptr = kmalloc(a.len,__GFP_NORETRY);
if (!a.return_ptr)
{
printk(KERN_INFO "return_PTR allocation failed\n");
kfree(a.return_ptr);
kfree(a.ptr);
kfree(a.placeholder);
scif_close(a.endpoint);
return -1;
}
ret = scif_register(a.remote_endpoint, a.ptr, a.len, (long)a.ptr, SCIF_PROT_READ | SCIF_PROT_WRITE, SCIF_MAP_FIXED);
if (ret != (off_t)a.ptr)
{
printk(KERN_INFO "scif_register ptr failed due to: %zu\n", ret);
kfree(a.return_ptr);
kfree(a.ptr);
kfree(a.placeholder);
scif_close(a.endpoint);
return -1;
}
printk(KERN_INFO "scif_register of ptr successfull\n");
ret = scif_register(a.remote_endpoint, a.return_ptr, a.len, (long)a.return_ptr, SCIF_PROT_READ | SCIF_PROT_WRITE, SCIF_MAP_FIXED);
if (ret != (long)a.return_ptr)
{
printk(KERN_INFO "scif_register of return_ptr failed due to: %zu\n", ret);
scif_unregister(a.remote_endpoint, (off_t)a.return_ptr, a.len);
kfree(a.return_ptr);
kfree(a.ptr);
kfree(a.placeholder);
scif_close(a.endpoint);
return 1;
}
printk(KERN_INFO "scif_register of return_ptr successfull\n");
...
}
The functions like scif_bind, open, accept, send, recieve, work fine (I already ported a simple scif_send, scif_recieve latency benchmark into kernel space) but I cant figure out why scif_register fails.
I already changed the return value to something like
if (ret <0)
{
...
}
but then the program executed on the MIC reported an error while performing a DMA read from the host.
any suggestions on what do change so that the code runs fine?
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
For scif_register, the memory allocation must be in whole pages and aligned on page boundaries. I wonder if the kmalloc isn't aligning on a page boundary - I don't think it is guaranteed that it will. What happens if you use something like alloc_pages instead?
'
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Francis
I was aware of that yesterday, therefore I changed the allocation from kmalloc to vmalloc, as the allocation is then done by whole pages
currently 16*PAGE_SIZE
The main problem I have currently is that I cannot understand how the return value is checked in Kernel Space. In Userspace I checked on a macro but in Kernel space it is checken against the pointer offset.
Therefore I currently changed my function to
ret = scif_register(a.remote_endpoint, a.ptr, a.len, 0, 0, SCIF_MAP_KERNEL);
So the offset is 0, and I check of the return value is !=0, if yes than abort.
Is this correct or should the offset be rather (off_t)a.ptr ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Perhaps the first thing I should have asked is what error scif_register was returning. Do you remember what it was?
And I am looking for someone with more kernel experience that I have to provide an example that you can use as a reference.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Error was -14 (EFAULT)
Return value was 4611686018427387904 which is a multiple of PAGE_SIZE (4096)
As the scif_register function itself has a error reporting part I devided to just check it against negative values and go on.
But now scif_readfrom is comlaining with error -6 (ENXIO)
Any suggestions?
here's the changed Host code
off_t regreturn;
regreturn = scif_register(a.remote_endpoint, a.ptr, a.len, 0, SCIF_PROT_READ, SCIF_MAP_KERNEL);
if (regreturn < 0)
{
printk(KERN_INFO "scif_register ptr failed due to: %lld\n", (long long)regreturn);
printk(KERN_INFO "%lld\n", (long long)regreturn/PAGE_SIZE);
goto exit;
}
printk(KERN_INFO "scif_register of ptr successfull\n");
regreturn = scif_register(a.remote_endpoint, a.return_ptr, a.len, 0, SCIF_PROT_WRITE, SCIF_MAP_KERNEL);
if (regreturn < 0)
{
printk(KERN_INFO "scif_register of return_ptr failed due to: %zu\n", regreturn);
goto exit;
}
printk(KERN_INFO "scif_register of return_ptr successfull\n");
ret = scif_send(a.remote_endpoint, &a.ptr, sizeof(uint8_t*), SCIF_SEND_BLOCK);
if (ret != sizeof(uint8_t*))
{
printk(KERN_INFO "scif_send failed due to %zu\n", ret);
goto exit;
}
printk(KERN_INFO "scif_send of ptr successfull\n");
ret = scif_send(a.remote_endpoint, &a.return_ptr, sizeof(uint8_t*), SCIF_SEND_BLOCK);
if (ret != sizeof(uint8_t*))
{
printk(KERN_INFO "scif_send of return_ptr failed due to %zu\n", ret);
goto exit;
}
printk(KERN_INFO "scif_send of return_ptr successfull\n");
printk(KERN_INFO "Waiting for MIC Code to be finished\n");
ret = scif_recv(a.remote_endpoint, a.placeholder, a.len, SCIF_RECV_BLOCK);
if (ret != a.len)
{
printk(KERN_INFO "scif_recv of placeholder failed due to %zu\n", ret);
goto exit;
}
printk(KERN_INFO "MIC Code finished\n");
return 0;
Here's the corresponding MIC code that runs in MICs User-Space
int main( )
{
size_t len = 131072;
int align = 4096;
scif_epd_t endpoint;
struct scif_portID portid;
int ret;
uint8_t *in_key = malloc(16 * sizeof(uint8_t));
struct crypto_tfm *tfm
= malloc(
sizeof(struct crypto_tfm) +
sizeof(struct crypto_aes_ctx)
);
struct crypto_aes_ctx *ctx
= crypto_tfm_ctx(tfm);
ctx->key_length = AES_KEYSIZE_128;
crypto_aes_set_key(tfm, in_key, AES_KEYSIZE_128);
// printf("output MIC\n");
endpoint = scif_open( );
if( endpoint == SCIF_OPEN_FAILED )
{
printf("scif open failed\n");
return 1;
}
ret = scif_bind(endpoint, 23955);
if(ret==-1)
{
printf("scif_bind failed");
return 1;
}
portid.node = 0;
portid.port = 23969;
ret = scif_connect(endpoint, &portid);
for( int attempt = 0; ret == -1 && attempt < 10; ++attempt )
{
sleep(1);
ret = scif_connect(endpoint, &portid);
printf("attempting connection %d \n", attempt);
}
if (ret==-1)
{
printf("scif_connect failed\n");
return 1;
}
void *ptr;
ret = posix_memalign((void**)&ptr, align, len);
if (ret)
{
printf("Allocating memory failed\n");
return 1;
}
memset(ptr, 0, len);
printf("memory allocated\n");
if( SCIF_REGISTER_FAILED == scif_register(endpoint, ptr, len, 0, SCIF_PROT_READ | SCIF_PROT_WRITE, 0 ) )
{
printf("scif_register of ptr failed due to: %s\n", strerror(errno));
return 1;
}
printf("scif_register success\n");
void *tempbuffer;
ret = posix_memalign((void**)&tempbuffer, align, len);
if (ret)
{
printf("Allocating tempbuffer failed\n");
return 1;
}
if( SCIF_REGISTER_FAILED == scif_register(endpoint, tempbuffer, len, 0, SCIF_PROT_READ | SCIF_PROT_WRITE, 0 ))
{
printf("scif_register of temp failed due to: %s\n", strerror(errno));
return 1;
}
printf("scif_register success\n");
void *outbuffer;
ret = posix_memalign((void**)&outbuffer, align, len);
if (ret)
{
printf("Allocating outbuffer failed %s\n", strerror(errno));
return 1;
}
if( SCIF_REGISTER_FAILED == scif_register(endpoint, outbuffer, len, 0, SCIF_PROT_READ | SCIF_PROT_WRITE, 0 ))
{
printf("scif_register of outbuffer failed due to: %s\n", strerror(errno));
return 1;
}
void *remote_ptr;
void *return_ptr;
ret = scif_recv(endpoint, &remote_ptr, sizeof(void*), SCIF_RECV_BLOCK);
if (ret==-1)
{
printf("scif_recv of remote_ptr failed due to: %s\n", strerror(errno));
return 1;
}
printf("scif_recv successfull\n");
ret = scif_recv(endpoint, &return_ptr, sizeof(void*), SCIF_RECV_BLOCK);
if (ret==-1)
{
printf("scif_recv f remote_ptr failed due to: %s\n", strerror(errno));
return 1;
}
printf("scif_recv successfull\n");
size_t sizearray[5] = {64, 1024, 4096, 16384, 131072 /*1048576, 134217728, 268435456, 536870912, 1073741824*/ };
for (int i=0; i<5; i++)
{
struct timespec start_for, stop_for;
clock_gettime(CLOCK_REALTIME, &start_for);
for (int j=1; j<16; j++)
{
if (scif_readfrom(endpoint, (long)ptr, sizearray, (long)remote_ptr, SCIF_RMA_SYNC))
{
printf("scif_readfrom failed due to: %d\n", errno);
return 1;
}
#pragma omp parallel for
for (int k = 0; k<sizearray; k+=16)
{ aes_encrypt(tfm, (uint8_t*)&tempbuffer, (uint8_t*)&ptr); }
if (scif_writeto(endpoint, (long)tempbuffer, sizearray, (long)return_ptr, SCIF_RMA_SYNC))
{
printf("scif_writeto failed due to: %d\n", errno);
return 1;
}
}
clock_gettime(CLOCK_REALTIME, &stop_for);
double time_for = (stop_for.tv_sec - start_for.tv_sec) + ( stop_for.tv_nsec - start_for.tv_nsec) / NANOSECONDS;
double result = sizearray/time_for*15/1048576;
printf("%1f \n", result);
}
for (int i=0; i<5; i++)
{
struct timespec start_for, stop_for;
clock_gettime(CLOCK_REALTIME, &start_for);
for (int j=1; j<16; j++)
{
if (scif_readfrom(endpoint, (long)ptr, sizearray, (long)remote_ptr, SCIF_RMA_SYNC))
{
printf("scif_readfrom failed due to: %d\n", errno);
return 1;
}
#pragma omp parallel for
for (int k=0; k<sizearray; k+=16)
{ aes_decrypt(tfm, (uint8_t*)&outbuffer, (uint8_t*)&tempbuffer); }
if (scif_writeto(endpoint, (long)outbuffer, sizearray, (long)return_ptr, SCIF_RMA_SYNC))
{
printf("scif_writeto failed due to: %d\n", errno);
return 1;
}
}
clock_gettime(CLOCK_REALTIME, &stop_for);
double time_for = (stop_for.tv_sec - start_for.tv_sec) + ( stop_for.tv_nsec - start_for.tv_nsec) / NANOSECONDS;
double result = sizearray/time_for*15/1048576;
printf("%1f \n", result);
}
ret = scif_send(endpoint, &ptr, sizeof(void*), SCIF_SEND_BLOCK);
if (ret==-1)
{
printf("scif_send failed due to: %s\n", strerror(errno));
return 1;
}
ret = scif_unregister(endpoint, (off_t)ptr, len );
if(ret==-1 && errno!=ENOTCONN )
{
printf("scif_unregister failed %s\n", strerror(errno));
return 1;
}
scif_close(endpoint);
}
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm not sure that you can register a window in user space on the coprocessor to an address in kernel space.
Also, I'm pretty sure that vmalloc won't work; the memory needs to actually be physically contiguous. I have found other code where people did use kmalloc. As long as they were allocating a multiple of the page size, they seemed to be ok.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm glad you have it working. I was truly out of my depth, here. According to the SCIF developers I talked to, both kmalloc and vmalloc were acceptable in this case and, as you found, you can indeed go between kernel space on the host and user space on the coprocessor. If it is not too much hassle, it might help future developers if you were to post the relevant code from your final scif_start.
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page