Software Archive
Read-only legacy content
17061 Discussions

Simple Offload Example Failing

Will_F_
Beginner
651 Views

Hello,

I'm attempting to run a simple offload example: 

#include <stdio.h>
#include <omp.h>

int main(){
double sum; int i,n, nt;

   n=2000000000;
   sum=0.0e0;

   #pragma offload target(mic:0)
   {
    #pragma omp parallel for reduction(+:sum)
    for(i=1;i<=n;i++){
       sum = sum + i;
    }
    //nt = omp_get_max_threads();
    #pragma omp parallel
    {
       #pragma omp single
       nt = omp_get_num_threads();
    }

    #ifdef __MIC__
       printf("Hello MIC reduction %f threads: %d\n",sum,nt);
    #else
       printf("Hello CPU reduction %f threads: %d\n",sum,nt);
    #endif
   }
}

This program ran fine previously but we recently rebooted our Phi nodes in our cluster and since then this offloading example will not run. The native compiled MIC binaries still run without a problem since the reboot.

Before running I type:

. /usr/local/intel/ClusterStudioXE_2013/composer_xe_2013_sp1/bin/compilervars.sh intel64
make
export MIC_OMP_NUM_THREADS=120
export MIC_ENV_PREFIX=MIC
export OFFLOAD_REPORT=3

Here is my Makefile:

CC=icc
CFLAGS=-std=c99 -O3 -vec-report3 -openmp -offload
EXE=reduce_offload_mic

$(EXE) : reduce_omp_mic.c
	$(CC) -o $@ $< $(CFLAGS) 

.PHONY: clean

clean:
	rm $(EXE)

However, when I run the program here is the output:

[frenchwr@vmp903 Offload]$ ./reduce_offload_mic 
offload error: cannot offload to MIC - device is not available
[Offload] [HOST]  [State]   Unregister data tables

I have ensured that mpss is running and even restarted the service with:

sudo service mpss restart

but still the same error (even after re-building the executable).

All of my mic tests pass:

[frenchwr@vmp903 Offload]$ miccheck
MicCheck 3.4-r1
Copyright 2013 Intel Corporation All Rights Reserved

Executing default tests for host
  Test 0: Check number of devices the OS sees in the system ... pass
  Test 1: Check mic driver is loaded ... pass
  Test 2: Check number of devices driver sees in the system ... pass
  Test 3: Check mpssd daemon is running ... pass
Executing default tests for device: 0
  Test 4 (mic0): Check device is in online state and its postcode is FF ... pass
  Test 5 (mic0): Check ras daemon is available in device ... pass
  Test 6 (mic0): Check running flash version is correct ... pass
  Test 7 (mic0): Check running SMC firmware version is correct ... pass
Executing default tests for device: 1
  Test 8 (mic1): Check device is in online state and its postcode is FF ... pass
  Test 9 (mic1): Check ras daemon is available in device ... pass
  Test 10 (mic1): Check running flash version is correct ... pass
  Test 11 (mic1): Check running SMC firmware version is correct ... pass

Status: OK

Here's the output from micinfo:

[frenchwr@vmp903 Offload]$ micinfo
MicInfo Utility Log
Created Fri Aug 28 18:14:23 2015


	System Info
		HOST OS			: Linux
		OS Version		: 2.6.32-431.29.2.el6.x86_64
		Driver Version		: 3.4-1
		MPSS Version		: 3.4
		Host Physical Memory	: 132110 MB

Device No: 0, Device Name: mic0

	Version
		Flash Version 		 : 2.1.02.0390
		SMC Firmware Version	 : 1.16.5078
		SMC Boot Loader Version	 : 1.8.4326
		uOS Version 		 : 2.6.38.8+mpss3.4
		Device Serial Number 	 : ADKC42900304

	Board
		Vendor ID 		 : 0x8086
		Device ID 		 : 0x225c
		Subsystem ID 		 : 0x7d95
		Coprocessor Stepping ID	 : 2
		PCIe Width 		 : Insufficient Privileges
		PCIe Speed 		 : Insufficient Privileges
		PCIe Max payload size	 : Insufficient Privileges
		PCIe Max read req size	 : Insufficient Privileges
		Coprocessor Model	 : 0x01
		Coprocessor Model Ext	 : 0x00
		Coprocessor Type	 : 0x00
		Coprocessor Family	 : 0x0b
		Coprocessor Family Ext	 : 0x00
		Coprocessor Stepping 	 : C0
		Board SKU 		 : C0PRQ-7120 P/A/X/D
		ECC Mode 		 : Enabled
		SMC HW Revision 	 : Product 300W Passive CS

	Cores
		Total No of Active Cores : 61
		Voltage 		 : 1037000 uV
		Frequency		 : 1238095 kHz

	Thermal
		Fan Speed Control 	 : N/A
		Fan RPM 		 : N/A
		Fan PWM 		 : N/A
		Die Temp		 : 46 C

	GDDR
		GDDR Vendor		 : Samsung
		GDDR Version		 : 0x6
		GDDR Density		 : 4096 Mb
		GDDR Size		 : 15872 MB
		GDDR Technology		 : GDDR5 
		GDDR Speed		 : 5.500000 GT/s 
		GDDR Frequency		 : 2750000 kHz
		GDDR Voltage		 : 1501000 uV

Device No: 1, Device Name: mic1

	Version
		Flash Version 		 : 2.1.02.0390
		SMC Firmware Version	 : 1.16.5078
		SMC Boot Loader Version	 : 1.8.4326
		uOS Version 		 : 2.6.38.8+mpss3.4
		Device Serial Number 	 : ADKC42900319

	Board
		Vendor ID 		 : 0x8086
		Device ID 		 : 0x225c
		Subsystem ID 		 : 0x7d95
		Coprocessor Stepping ID	 : 2
		PCIe Width 		 : Insufficient Privileges
		PCIe Speed 		 : Insufficient Privileges
		PCIe Max payload size	 : Insufficient Privileges
		PCIe Max read req size	 : Insufficient Privileges
		Coprocessor Model	 : 0x01
		Coprocessor Model Ext	 : 0x00
		Coprocessor Type	 : 0x00
		Coprocessor Family	 : 0x0b
		Coprocessor Family Ext	 : 0x00
		Coprocessor Stepping 	 : C0
		Board SKU 		 : C0PRQ-7120 P/A/X/D
		ECC Mode 		 : Enabled
		SMC HW Revision 	 : Product 300W Passive CS

	Cores
		Total No of Active Cores : 61
		Voltage 		 : 1040000 uV
		Frequency		 : 1238095 kHz

	Thermal
		Fan Speed Control 	 : N/A
		Fan RPM 		 : N/A
		Fan PWM 		 : N/A
		Die Temp		 : 47 C

	GDDR
		GDDR Vendor		 : Samsung
		GDDR Version		 : 0x6
		GDDR Density		 : 4096 Mb
		GDDR Size		 : 15872 MB
		GDDR Technology		 : GDDR5 
		GDDR Speed		 : 5.500000 GT/s 
		GDDR Frequency		 : 2750000 kHz
		GDDR Voltage		 : 1501000 uV

 

From searching online I see a few other users who have run into the:

offload error: cannot offload to MIC - device is not available
[Offload] [HOST]  [State]   Unregister data tables

issue, but I don't see any good resolution (other than by restarting mpss, which does not resolve the issue for me).

0 Kudos
1 Solution
Frances_R_Intel
Employee
651 Views

The coi_daemon should start up automatically when the coprocessor is booted and if it doesn't, you cannot offload any programs. You asked yesterday if user micuser was required. The answer to that is - it depends on how coi_daemon is started.

If you look in /etc/init.d/coi on the coprocessor, you will see one of the default parameters is: "coiparams='--coiuser=micuser'". Unless you provide an alternate value, coi_daemon will try to start up with the user identity of micuser. If micuser does not exist, coi_daemon will fail to start and you will not be able to run offload programs.

The fix is either create a user named micuser on the coprocessor or start coi_demon with a different user identity. Given that you were able to offload in the past but can not do it now, implies that one of these conditions existed in the past but that something changed on the host to prevent it from happening now.

For the first solution, you can use a passwd file created specifically for the coprocessor instead of copying the file from the host (the MPSS User's Guide that came with your MPSS release will tell you how to use 'micctrl --users' to do that)  or you can add a user named micuser to the host and then use that host passwd file on the coprocessor. The advantage of the second option is that, since you are NFS mounting the home directories, any files you want to access on the coprocessor can be copied to the micuser home directory on the host and they will be there when you want them.

For the second solution, you can create a /var/mpss/micN/etc/sysconfig/coi.conf or /var/mpss/micN/etc/coi.conf file (where N is the number of the coprocessor you will be using) where you set coiuser to a valid user name. Some sites change the value of coiuser and reboot the coprocessor between jobs. However these site also restrict access to the host so that only one user has access at a time. If they didn't do this then everyone on the host would end up running as that particular user when they offload to the coprocessor.

My recommendation is the first solution.

View solution in original post

0 Kudos
11 Replies
Frances_R_Intel
Employee
651 Views

With no change to the software? Only a reboot?

Can you check to be sure that coi_daemon is running on the coprocessors, that the micuser exists as a user on the coprocessors, that there is a .ssh directory in micuser's home directory, with valid ssh keys and that, when you ran the compilervars.sh, it set your MIC_LD_LIBRARY_PATH, as expected? Also, on both the host and coprocessors, check /var/log/messages and /var/log/mpssd. You are looking for something that shows a coi error of some kind.

0 Kudos
JJK
New Contributor III
651 Views

check the permissions on the /dev/mic/scif device ; alternatively, try running the offload program as root user - does it also report 'device not available' in that case?

Perhaps selinux is interfering with running an offloaded program?

Finally, I'd recommend upgrading to mpss 3.4.5 (you're running 3.4 now and it contains a minor security vulnerability related to ntp).

 

0 Kudos
Will_F_
Beginner
651 Views

Hello Frances and JJK, thanks for your replies.

Frances, yes, the example stopped working following the reboot. I did not personally configure these nodes but what I suspect has happened is that some configuration change has reverted back to an initial value as a result of the reboot. I do not see coi_daemon running on the mic card when this example is running, nor do I see any coi error messages in the logs on the host or device. Is having a micuser set up on the MIC cards a requirement? We don't recall doing this previously, and we do not have a micuser currently set up. Could you point me to some recent documentation that explains the recommended way to set this up?

JJK, I tried running the offload example as root -- same problem. The permissions on /dev/mic/scif (on both the host and device) are the following:

crw-rw-rw-    1 root     root      253,   1 Dec 31  1969 /dev/mic/scif

I'm not sure whether this a selinux issue or not. I'll look into it.

I will plan on upgrading to 3.4.5 in the near future.

Best,

Will  

 

0 Kudos
JJK
New Contributor III
651 Views

Hi,

can you run 'strace' on your offloaded executable? I've compiled and run your code on one of my boxen and it had no problems whatsoever (admittedly, using composer XE 2015). Here's a portion of the strace log on my box:

    340 open("/dev/mic/scif", O_RDWR)           = 3
    341 ioctl(3, STL_BRESET, 0x7ffe3e7ab7a8)    = 1
    342 close(3)                                = 0
    343 munmap(0x7f9a3b7a6000, 78250)           = 0
    344 open("/sys/class/mic/mic0/family", O_RDONLY) = 3
    345 fstat(3, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
    346 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9a3b7b9000
    347 read(3, "x100\n", 4096)                 = 5
    348 close(3)                                = 0
    349 munmap(0x7f9a3b7b9000, 4096)            = 0
    350 open("/dev/mic/scif", O_RDWR)           = 3
    351 ioctl(3, 0xc008730e, 0x7ffe3e7ab340)    = 0
    352 close(3)                                = 0
    353 open("/dev/mic/scif", O_RDWR)           = 3
    354 ioctl(3, 0xc008730e, 0x7ffe3e7ab420)    = 0
    355 close(3)                                = 0
    356 open("/dev/mic/scif", O_RDWR)           = 3
    357 fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
    358 fcntl(3, F_GETFD)                       = 0x1 (flags FD_CLOEXEC)
    359 fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
    360 ioctl(3, 0xc0087301, 0x7ffe3e7ab1bc)    = 0
    361 open("/dev/mic/scif", O_RDWR)           = 4
    362 fcntl(4, F_SETFD, FD_CLOEXEC)           = 0
    363 ioctl(4, 0xc0087301, 0x7ffe3e7ab19c)    = 0
    364 open("/dev/mic/scif", O_RDWR)           = 5
    365 ioctl(5, 0xc008730e, 0x7ffe3e7ab0e0)    = 0
    366 close(5)                                = 0
    367 mmap(NULL, 10489856, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f9a378a9000
    368 mprotect(0x7f9a378a9000, 4096, PROT_NONE) = 0
    369 clone(child_stack=0x7f9a382a8fd0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f9a382a99d0, tls=0x7        f9a382a9700, child_tidptr=0x7f9a382a99d0) = 29615
    370 open("/dev/mic/scif", O_RDWR)           = 5
    371 fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
    372 ioctl(5, 0xc0087303, 0x7ffe3e7ab0f0)    = 0
    373 futex(0x7f9a382a99d0, FUTEX_WAIT, 29615, NULL) = 0

 

I'm curious if the code on your box gets as far as this.

0 Kudos
Will_F_
Beginner
651 Views

Here's what I see after all the shared objects have been loaded:

 

open("/usr/lib64/libscif.so.0", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\n\0\3677\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=10480, ...}) = 0
mmap(0x37f7000000, 2103136, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x37f7000000
mprotect(0x37f7002000, 2093056, PROT_NONE) = 0
mmap(0x37f7201000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x37f7201000
close(3)                                = 0
mprotect(0x37f6e06000, 4096, PROT_READ) = 0
open("/dev/mic/scif", O_RDWR)           = 3
ioctl(3, 0x7317, 0x7fffe16f8d48)        = 1
close(3)                                = 0
munmap(0x7f5442609000, 52666)           = 0
open("/sys/class/mic/mic0/family", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5442615000
read(3, "x100\n", 4096)                 = 5
close(3)                                = 0
munmap(0x7f5442615000, 4096)            = 0
open("/dev/mic/scif", O_RDWR)           = 3
ioctl(3, 0xc008730e, 0x7fffe16f8790)    = 0
close(3)                                = 0
open("/dev/mic/scif", O_RDWR)           = 3
ioctl(3, 0xc008730e, 0x7fffe16f8870)    = 0
close(3)                                = 0
open("/dev/mic/scif", O_RDWR)           = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fcntl(3, F_GETFD)                       = 0x1 (flags FD_CLOEXEC)
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
ioctl(3, 0xc0087301, 0x7fffe16f860c)    = 0
open("/dev/mic/scif", O_RDWR)           = 4
fcntl(4, F_SETFD, FD_CLOEXEC)           = 0
ioctl(4, 0xc0087301, 0x7fffe16f85ec)    = 0
open("/dev/mic/scif", O_RDWR)           = 5
ioctl(5, 0xc008730e, 0x7fffe16f8530)    = 0
close(5)                                = 0
mmap(NULL, 10489856, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f543ff5c000
mprotect(0x7f543ff5c000, 4096, PROT_NONE) = 0
clone(child_stack=0x7f544095bfd0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f544095c9d0, tls=0x7f544095c700, child_tidptr=0x7f544095c9d0) = 19866
open("/dev/mic/scif", O_RDWR)           = 5
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
ioctl(5, 0xc0087303, 0x7fffe16f8540)    = 0
futex(0x7f544095c9d0, FUTEX_WAIT, 19866, NULL) = 0
close(4)                                = 0
ioctl(3, 0xc0087303, 0x7fffe16f85e0)    = -1 ECONNREFUSED (Connection refused)
nanosleep({0, 10000000}, NULL)          = 0
ioctl(3, 0xc0087303, 0x7fffe16f85e0)    = -1 ECONNREFUSED (Connection refused)
nanosleep({0, 20000000}, NULL)          = 0
ioctl(3, 0xc0087303, 0x7fffe16f85e0)    = -1 ECONNREFUSED (Connection refused)
nanosleep({0, 40000000}, NULL)          = 0
ioctl(3, 0xc0087303, 0x7fffe16f85e0)    = -1 ECONNREFUSED (Connection refused)
nanosleep({0, 80000000}, NULL)          = 0
ioctl(3, 0xc0087303, 0x7fffe16f85e0)    = -1 ECONNREFUSED (Connection refused)
nanosleep({0, 160000000}, NULL)         = 0
ioctl(3, 0xc0087303, 0x7fffe16f85e0)    = -1 ECONNREFUSED (Connection refused)
nanosleep({0, 320000000}, NULL)         = 0
ioctl(3, 0xc0087303, 0x7fffe16f85e0)    = -1 ECONNREFUSED (Connection refused)
nanosleep({0, 640000000}, NULL)         = 0
ioctl(3, 0xc0087303, 0x7fffe16f85e0)    = -1 ECONNREFUSED (Connection refused)
nanosleep({1, 280000000}, NULL)         = 0
ioctl(3, 0xc0087303, 0x7fffe16f85e0)    = -1 ECONNREFUSED (Connection refused)
nanosleep({2, 560000000}, NULL)         = 0
ioctl(3, 0xc0087303, 0x7fffe16f85e0)    = -1 ECONNREFUSED (Connection refused)
nanosleep({5, 120000000}, NULL)         = 0
ioctl(3, 0xc0087303, 0x7fffe16f85e0)    = -1 ECONNREFUSED (Connection refused)

 

So clearly something is not configured to allow offloading to the mic correctly. We are NFS mounting user /home and /scratch directories to the MIC cards, and we simply copied over the /etc/passwd, /etc/shadow, etc/group files from the MIC node. I'm able to ssh on to the MIC cards as root or as a normal user without any trouble.

Thanks for your help.

Will

0 Kudos
JJK
New Contributor III
651 Views

hmmm , it's definitely an issue with the SCIF interaction between host and the Phi. This explains why a regular login works, but offloading does not. Can you try attaching strace to the coi_daemon process on the Phi (strace can be installed as an RPM for the Xeon Phi)?

On the Phi, run

rpm -ivh strace-4.6-r1.k1om.rpm

strace -p `ps -u micuser | grep coi_daemon | awk '{ print $1 }'`

then on the host run the offloading example.

 

0 Kudos
Will_F_
Beginner
651 Views

When I run the offload example, I see no coi_daemon processes running at all on the MIC card. 

0 Kudos
JJK
New Contributor III
651 Views

that explains a few things. What does

[Phi]#   /etc/init.d/coi status

return ? if it is not running, try

[Phi]#   /etc/init.d/coi start

then rerun the offload example.

The coi daemon should have been started at Phi bootup via these script-links:

lrwxrwxrwx 1 root root 13 Jan  1  1970 /etc/rc3.d/S95coi -> ../init.d/coi
lrwxrwxrwx 1 root root 13 Jan  1  1970 /etc/rc5.d/S95coi -> ../init.d/coi

 

0 Kudos
Frances_R_Intel
Employee
652 Views

The coi_daemon should start up automatically when the coprocessor is booted and if it doesn't, you cannot offload any programs. You asked yesterday if user micuser was required. The answer to that is - it depends on how coi_daemon is started.

If you look in /etc/init.d/coi on the coprocessor, you will see one of the default parameters is: "coiparams='--coiuser=micuser'". Unless you provide an alternate value, coi_daemon will try to start up with the user identity of micuser. If micuser does not exist, coi_daemon will fail to start and you will not be able to run offload programs.

The fix is either create a user named micuser on the coprocessor or start coi_demon with a different user identity. Given that you were able to offload in the past but can not do it now, implies that one of these conditions existed in the past but that something changed on the host to prevent it from happening now.

For the first solution, you can use a passwd file created specifically for the coprocessor instead of copying the file from the host (the MPSS User's Guide that came with your MPSS release will tell you how to use 'micctrl --users' to do that)  or you can add a user named micuser to the host and then use that host passwd file on the coprocessor. The advantage of the second option is that, since you are NFS mounting the home directories, any files you want to access on the coprocessor can be copied to the micuser home directory on the host and they will be there when you want them.

For the second solution, you can create a /var/mpss/micN/etc/sysconfig/coi.conf or /var/mpss/micN/etc/coi.conf file (where N is the number of the coprocessor you will be using) where you set coiuser to a valid user name. Some sites change the value of coiuser and reboot the coprocessor between jobs. However these site also restrict access to the host so that only one user has access at a time. If they didn't do this then everyone on the host would end up running as that particular user when they offload to the coprocessor.

My recommendation is the first solution.

0 Kudos
Will_F_
Beginner
651 Views

Thank you both for your input. That was indeed the issue. After creating the micuser on the phi the offload example runs. Here's the output I see:

[Offload] [HOST]  [State]   Initialize logical card 0 = physical card 0
[Offload] [HOST]  [State]   Initialize logical card 1 = physical card 1
[Offload] [MIC 0] [File]            reduce_omp_mic.c
[Offload] [MIC 0] [Line]            17
[Offload] [MIC 0] [Tag]             Tag 0
[Offload] [HOST]  [Tag 0] [State]   Start Offload
[Offload] [HOST]  [Tag 0] [State]   Initialize function __offload_entry_reduce_omp_mic_c_17mainicc572303114teczsI
[Offload] [HOST]  [Tag 0] [State]   Send pointer data
[Offload] [HOST]  [Tag 0] [State]   CPU->MIC pointer data 0
[Offload] [HOST]  [Tag 0] [State]   Gather copyin data
[Offload] [HOST]  [Tag 0] [State]   CPU->MIC copyin data 20 
[Offload] [HOST]  [Tag 0] [State]   Compute task on MIC
[Offload] [HOST]  [Tag 0] [State]   Receive pointer data
[Offload] [HOST]  [Tag 0] [State]   MIC->CPU pointer data 0
[Offload] [MIC 0] [Tag 0] [State]   Start target function __offload_entry_reduce_omp_mic_c_17mainicc572303114teczsI
[Offload] [MIC 0] [Tag 0] [Var]     i  INOUT
[Offload] [MIC 0] [Tag 0] [Var]     n  INOUT
[Offload] [MIC 0] [Tag 0] [Var]     sum  INOUT
[Offload] [MIC 0] [Tag 0] [Var]     nt  INOUT
[Offload] [MIC 0] [Tag 0] [State]   Scatter copyin data
Hello MIC reduction 200000010000000.000000 threads: 120
[Offload] [MIC 0] [Tag 0] [State]   Gather copyout data
[Offload] [HOST]  [Tag 0] [State]   Scatter copyout data
[Offload] [HOST]  [Tag 0] [CPU Time]        0.677483(seconds)
[Offload] [MIC 0] [Tag 0] [CPU->MIC Data]   20 (bytes)
[Offload] [MIC 0] [Tag 0] [MIC Time]        0.135097(seconds)
[Offload] [MIC 0] [Tag 0] [MIC->CPU Data]   20 (bytes)

[Offload] [MIC 0] [Tag 0] [State]   MIC->CPU copyout data   20
[Offload] [MIC 1] [State]   Unregister data tables
[Offload] [MIC 0] [State]   Unregister data tables
[Offload] [HOST]  [State]   Unregister data tables

Is this consistent with what you see in your environments? I'm mainly wondering about the last three "Unregister data tables" messages. 

0 Kudos
Frances_R_Intel
Employee
651 Views

"Unregister data tables" is just coi cleaning up after itself. Looks like you are good to go.

0 Kudos
Reply