- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
I'm attempting to run a simple offload example:
#include <stdio.h> #include <omp.h> int main(){ double sum; int i,n, nt; n=2000000000; sum=0.0e0; #pragma offload target(mic:0) { #pragma omp parallel for reduction(+:sum) for(i=1;i<=n;i++){ sum = sum + i; } //nt = omp_get_max_threads(); #pragma omp parallel { #pragma omp single nt = omp_get_num_threads(); } #ifdef __MIC__ printf("Hello MIC reduction %f threads: %d\n",sum,nt); #else printf("Hello CPU reduction %f threads: %d\n",sum,nt); #endif } }
This program ran fine previously but we recently rebooted our Phi nodes in our cluster and since then this offloading example will not run. The native compiled MIC binaries still run without a problem since the reboot.
Before running I type:
. /usr/local/intel/ClusterStudioXE_2013/composer_xe_2013_sp1/bin/compilervars.sh intel64 make export MIC_OMP_NUM_THREADS=120 export MIC_ENV_PREFIX=MIC export OFFLOAD_REPORT=3
Here is my Makefile:
CC=icc CFLAGS=-std=c99 -O3 -vec-report3 -openmp -offload EXE=reduce_offload_mic $(EXE) : reduce_omp_mic.c $(CC) -o $@ $< $(CFLAGS) .PHONY: clean clean: rm $(EXE)
However, when I run the program here is the output:
[frenchwr@vmp903 Offload]$ ./reduce_offload_mic offload error: cannot offload to MIC - device is not available [Offload] [HOST] [State] Unregister data tables
I have ensured that mpss is running and even restarted the service with:
sudo service mpss restart
but still the same error (even after re-building the executable).
All of my mic tests pass:
[frenchwr@vmp903 Offload]$ miccheck MicCheck 3.4-r1 Copyright 2013 Intel Corporation All Rights Reserved Executing default tests for host Test 0: Check number of devices the OS sees in the system ... pass Test 1: Check mic driver is loaded ... pass Test 2: Check number of devices driver sees in the system ... pass Test 3: Check mpssd daemon is running ... pass Executing default tests for device: 0 Test 4 (mic0): Check device is in online state and its postcode is FF ... pass Test 5 (mic0): Check ras daemon is available in device ... pass Test 6 (mic0): Check running flash version is correct ... pass Test 7 (mic0): Check running SMC firmware version is correct ... pass Executing default tests for device: 1 Test 8 (mic1): Check device is in online state and its postcode is FF ... pass Test 9 (mic1): Check ras daemon is available in device ... pass Test 10 (mic1): Check running flash version is correct ... pass Test 11 (mic1): Check running SMC firmware version is correct ... pass Status: OK
Here's the output from micinfo:
[frenchwr@vmp903 Offload]$ micinfo MicInfo Utility Log Created Fri Aug 28 18:14:23 2015 System Info HOST OS : Linux OS Version : 2.6.32-431.29.2.el6.x86_64 Driver Version : 3.4-1 MPSS Version : 3.4 Host Physical Memory : 132110 MB Device No: 0, Device Name: mic0 Version Flash Version : 2.1.02.0390 SMC Firmware Version : 1.16.5078 SMC Boot Loader Version : 1.8.4326 uOS Version : 2.6.38.8+mpss3.4 Device Serial Number : ADKC42900304 Board Vendor ID : 0x8086 Device ID : 0x225c Subsystem ID : 0x7d95 Coprocessor Stepping ID : 2 PCIe Width : Insufficient Privileges PCIe Speed : Insufficient Privileges PCIe Max payload size : Insufficient Privileges PCIe Max read req size : Insufficient Privileges Coprocessor Model : 0x01 Coprocessor Model Ext : 0x00 Coprocessor Type : 0x00 Coprocessor Family : 0x0b Coprocessor Family Ext : 0x00 Coprocessor Stepping : C0 Board SKU : C0PRQ-7120 P/A/X/D ECC Mode : Enabled SMC HW Revision : Product 300W Passive CS Cores Total No of Active Cores : 61 Voltage : 1037000 uV Frequency : 1238095 kHz Thermal Fan Speed Control : N/A Fan RPM : N/A Fan PWM : N/A Die Temp : 46 C GDDR GDDR Vendor : Samsung GDDR Version : 0x6 GDDR Density : 4096 Mb GDDR Size : 15872 MB GDDR Technology : GDDR5 GDDR Speed : 5.500000 GT/s GDDR Frequency : 2750000 kHz GDDR Voltage : 1501000 uV Device No: 1, Device Name: mic1 Version Flash Version : 2.1.02.0390 SMC Firmware Version : 1.16.5078 SMC Boot Loader Version : 1.8.4326 uOS Version : 2.6.38.8+mpss3.4 Device Serial Number : ADKC42900319 Board Vendor ID : 0x8086 Device ID : 0x225c Subsystem ID : 0x7d95 Coprocessor Stepping ID : 2 PCIe Width : Insufficient Privileges PCIe Speed : Insufficient Privileges PCIe Max payload size : Insufficient Privileges PCIe Max read req size : Insufficient Privileges Coprocessor Model : 0x01 Coprocessor Model Ext : 0x00 Coprocessor Type : 0x00 Coprocessor Family : 0x0b Coprocessor Family Ext : 0x00 Coprocessor Stepping : C0 Board SKU : C0PRQ-7120 P/A/X/D ECC Mode : Enabled SMC HW Revision : Product 300W Passive CS Cores Total No of Active Cores : 61 Voltage : 1040000 uV Frequency : 1238095 kHz Thermal Fan Speed Control : N/A Fan RPM : N/A Fan PWM : N/A Die Temp : 47 C GDDR GDDR Vendor : Samsung GDDR Version : 0x6 GDDR Density : 4096 Mb GDDR Size : 15872 MB GDDR Technology : GDDR5 GDDR Speed : 5.500000 GT/s GDDR Frequency : 2750000 kHz GDDR Voltage : 1501000 uV
From searching online I see a few other users who have run into the:
offload error: cannot offload to MIC - device is not available [Offload] [HOST] [State] Unregister data tables
issue, but I don't see any good resolution (other than by restarting mpss, which does not resolve the issue for me).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The coi_daemon should start up automatically when the coprocessor is booted and if it doesn't, you cannot offload any programs. You asked yesterday if user micuser was required. The answer to that is - it depends on how coi_daemon is started.
If you look in /etc/init.d/coi on the coprocessor, you will see one of the default parameters is: "coiparams='--coiuser=micuser'". Unless you provide an alternate value, coi_daemon will try to start up with the user identity of micuser. If micuser does not exist, coi_daemon will fail to start and you will not be able to run offload programs.
The fix is either create a user named micuser on the coprocessor or start coi_demon with a different user identity. Given that you were able to offload in the past but can not do it now, implies that one of these conditions existed in the past but that something changed on the host to prevent it from happening now.
For the first solution, you can use a passwd file created specifically for the coprocessor instead of copying the file from the host (the MPSS User's Guide that came with your MPSS release will tell you how to use 'micctrl --users' to do that) or you can add a user named micuser to the host and then use that host passwd file on the coprocessor. The advantage of the second option is that, since you are NFS mounting the home directories, any files you want to access on the coprocessor can be copied to the micuser home directory on the host and they will be there when you want them.
For the second solution, you can create a /var/mpss/micN/etc/sysconfig/coi.conf or /var/mpss/micN/etc/coi.conf file (where N is the number of the coprocessor you will be using) where you set coiuser to a valid user name. Some sites change the value of coiuser and reboot the coprocessor between jobs. However these site also restrict access to the host so that only one user has access at a time. If they didn't do this then everyone on the host would end up running as that particular user when they offload to the coprocessor.
My recommendation is the first solution.
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
With no change to the software? Only a reboot?
Can you check to be sure that coi_daemon is running on the coprocessors, that the micuser exists as a user on the coprocessors, that there is a .ssh directory in micuser's home directory, with valid ssh keys and that, when you ran the compilervars.sh, it set your MIC_LD_LIBRARY_PATH, as expected? Also, on both the host and coprocessors, check /var/log/messages and /var/log/mpssd. You are looking for something that shows a coi error of some kind.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
check the permissions on the /dev/mic/scif device ; alternatively, try running the offload program as root user - does it also report 'device not available' in that case?
Perhaps selinux is interfering with running an offloaded program?
Finally, I'd recommend upgrading to mpss 3.4.5 (you're running 3.4 now and it contains a minor security vulnerability related to ntp).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Frances and JJK, thanks for your replies.
Frances, yes, the example stopped working following the reboot. I did not personally configure these nodes but what I suspect has happened is that some configuration change has reverted back to an initial value as a result of the reboot. I do not see coi_daemon running on the mic card when this example is running, nor do I see any coi error messages in the logs on the host or device. Is having a micuser set up on the MIC cards a requirement? We don't recall doing this previously, and we do not have a micuser currently set up. Could you point me to some recent documentation that explains the recommended way to set this up?
JJK, I tried running the offload example as root -- same problem. The permissions on /dev/mic/scif (on both the host and device) are the following:
crw-rw-rw- 1 root root 253, 1 Dec 31 1969 /dev/mic/scif
I'm not sure whether this a selinux issue or not. I'll look into it.
I will plan on upgrading to 3.4.5 in the near future.
Best,
Will
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
can you run 'strace' on your offloaded executable? I've compiled and run your code on one of my boxen and it had no problems whatsoever (admittedly, using composer XE 2015). Here's a portion of the strace log on my box:
340 open("/dev/mic/scif", O_RDWR) = 3 341 ioctl(3, STL_BRESET, 0x7ffe3e7ab7a8) = 1 342 close(3) = 0 343 munmap(0x7f9a3b7a6000, 78250) = 0 344 open("/sys/class/mic/mic0/family", O_RDONLY) = 3 345 fstat(3, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0 346 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9a3b7b9000 347 read(3, "x100\n", 4096) = 5 348 close(3) = 0 349 munmap(0x7f9a3b7b9000, 4096) = 0 350 open("/dev/mic/scif", O_RDWR) = 3 351 ioctl(3, 0xc008730e, 0x7ffe3e7ab340) = 0 352 close(3) = 0 353 open("/dev/mic/scif", O_RDWR) = 3 354 ioctl(3, 0xc008730e, 0x7ffe3e7ab420) = 0 355 close(3) = 0 356 open("/dev/mic/scif", O_RDWR) = 3 357 fcntl(3, F_SETFD, FD_CLOEXEC) = 0 358 fcntl(3, F_GETFD) = 0x1 (flags FD_CLOEXEC) 359 fcntl(3, F_SETFD, FD_CLOEXEC) = 0 360 ioctl(3, 0xc0087301, 0x7ffe3e7ab1bc) = 0 361 open("/dev/mic/scif", O_RDWR) = 4 362 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 363 ioctl(4, 0xc0087301, 0x7ffe3e7ab19c) = 0 364 open("/dev/mic/scif", O_RDWR) = 5 365 ioctl(5, 0xc008730e, 0x7ffe3e7ab0e0) = 0 366 close(5) = 0 367 mmap(NULL, 10489856, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f9a378a9000 368 mprotect(0x7f9a378a9000, 4096, PROT_NONE) = 0 369 clone(child_stack=0x7f9a382a8fd0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f9a382a99d0, tls=0x7 f9a382a9700, child_tidptr=0x7f9a382a99d0) = 29615 370 open("/dev/mic/scif", O_RDWR) = 5 371 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 372 ioctl(5, 0xc0087303, 0x7ffe3e7ab0f0) = 0 373 futex(0x7f9a382a99d0, FUTEX_WAIT, 29615, NULL) = 0
I'm curious if the code on your box gets as far as this.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Here's what I see after all the shared objects have been loaded:
open("/usr/lib64/libscif.so.0", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\n\0\3677\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=10480, ...}) = 0 mmap(0x37f7000000, 2103136, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x37f7000000 mprotect(0x37f7002000, 2093056, PROT_NONE) = 0 mmap(0x37f7201000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x37f7201000 close(3) = 0 mprotect(0x37f6e06000, 4096, PROT_READ) = 0 open("/dev/mic/scif", O_RDWR) = 3 ioctl(3, 0x7317, 0x7fffe16f8d48) = 1 close(3) = 0 munmap(0x7f5442609000, 52666) = 0 open("/sys/class/mic/mic0/family", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5442615000 read(3, "x100\n", 4096) = 5 close(3) = 0 munmap(0x7f5442615000, 4096) = 0 open("/dev/mic/scif", O_RDWR) = 3 ioctl(3, 0xc008730e, 0x7fffe16f8790) = 0 close(3) = 0 open("/dev/mic/scif", O_RDWR) = 3 ioctl(3, 0xc008730e, 0x7fffe16f8870) = 0 close(3) = 0 open("/dev/mic/scif", O_RDWR) = 3 fcntl(3, F_SETFD, FD_CLOEXEC) = 0 fcntl(3, F_GETFD) = 0x1 (flags FD_CLOEXEC) fcntl(3, F_SETFD, FD_CLOEXEC) = 0 ioctl(3, 0xc0087301, 0x7fffe16f860c) = 0 open("/dev/mic/scif", O_RDWR) = 4 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 ioctl(4, 0xc0087301, 0x7fffe16f85ec) = 0 open("/dev/mic/scif", O_RDWR) = 5 ioctl(5, 0xc008730e, 0x7fffe16f8530) = 0 close(5) = 0 mmap(NULL, 10489856, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f543ff5c000 mprotect(0x7f543ff5c000, 4096, PROT_NONE) = 0 clone(child_stack=0x7f544095bfd0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f544095c9d0, tls=0x7f544095c700, child_tidptr=0x7f544095c9d0) = 19866 open("/dev/mic/scif", O_RDWR) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 ioctl(5, 0xc0087303, 0x7fffe16f8540) = 0 futex(0x7f544095c9d0, FUTEX_WAIT, 19866, NULL) = 0 close(4) = 0 ioctl(3, 0xc0087303, 0x7fffe16f85e0) = -1 ECONNREFUSED (Connection refused) nanosleep({0, 10000000}, NULL) = 0 ioctl(3, 0xc0087303, 0x7fffe16f85e0) = -1 ECONNREFUSED (Connection refused) nanosleep({0, 20000000}, NULL) = 0 ioctl(3, 0xc0087303, 0x7fffe16f85e0) = -1 ECONNREFUSED (Connection refused) nanosleep({0, 40000000}, NULL) = 0 ioctl(3, 0xc0087303, 0x7fffe16f85e0) = -1 ECONNREFUSED (Connection refused) nanosleep({0, 80000000}, NULL) = 0 ioctl(3, 0xc0087303, 0x7fffe16f85e0) = -1 ECONNREFUSED (Connection refused) nanosleep({0, 160000000}, NULL) = 0 ioctl(3, 0xc0087303, 0x7fffe16f85e0) = -1 ECONNREFUSED (Connection refused) nanosleep({0, 320000000}, NULL) = 0 ioctl(3, 0xc0087303, 0x7fffe16f85e0) = -1 ECONNREFUSED (Connection refused) nanosleep({0, 640000000}, NULL) = 0 ioctl(3, 0xc0087303, 0x7fffe16f85e0) = -1 ECONNREFUSED (Connection refused) nanosleep({1, 280000000}, NULL) = 0 ioctl(3, 0xc0087303, 0x7fffe16f85e0) = -1 ECONNREFUSED (Connection refused) nanosleep({2, 560000000}, NULL) = 0 ioctl(3, 0xc0087303, 0x7fffe16f85e0) = -1 ECONNREFUSED (Connection refused) nanosleep({5, 120000000}, NULL) = 0 ioctl(3, 0xc0087303, 0x7fffe16f85e0) = -1 ECONNREFUSED (Connection refused)
So clearly something is not configured to allow offloading to the mic correctly. We are NFS mounting user /home and /scratch directories to the MIC cards, and we simply copied over the /etc/passwd, /etc/shadow, etc/group files from the MIC node. I'm able to ssh on to the MIC cards as root or as a normal user without any trouble.
Thanks for your help.
Will
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
hmmm , it's definitely an issue with the SCIF interaction between host and the Phi. This explains why a regular login works, but offloading does not. Can you try attaching strace to the coi_daemon process on the Phi (strace can be installed as an RPM for the Xeon Phi)?
On the Phi, run
rpm -ivh strace-4.6-r1.k1om.rpm strace -p `ps -u micuser | grep coi_daemon | awk '{ print $1 }'`
then on the host run the offloading example.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
When I run the offload example, I see no coi_daemon processes running at all on the MIC card.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
that explains a few things. What does
[Phi]# /etc/init.d/coi status
return ? if it is not running, try
[Phi]# /etc/init.d/coi start
then rerun the offload example.
The coi daemon should have been started at Phi bootup via these script-links:
lrwxrwxrwx 1 root root 13 Jan 1 1970 /etc/rc3.d/S95coi -> ../init.d/coi lrwxrwxrwx 1 root root 13 Jan 1 1970 /etc/rc5.d/S95coi -> ../init.d/coi
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The coi_daemon should start up automatically when the coprocessor is booted and if it doesn't, you cannot offload any programs. You asked yesterday if user micuser was required. The answer to that is - it depends on how coi_daemon is started.
If you look in /etc/init.d/coi on the coprocessor, you will see one of the default parameters is: "coiparams='--coiuser=micuser'". Unless you provide an alternate value, coi_daemon will try to start up with the user identity of micuser. If micuser does not exist, coi_daemon will fail to start and you will not be able to run offload programs.
The fix is either create a user named micuser on the coprocessor or start coi_demon with a different user identity. Given that you were able to offload in the past but can not do it now, implies that one of these conditions existed in the past but that something changed on the host to prevent it from happening now.
For the first solution, you can use a passwd file created specifically for the coprocessor instead of copying the file from the host (the MPSS User's Guide that came with your MPSS release will tell you how to use 'micctrl --users' to do that) or you can add a user named micuser to the host and then use that host passwd file on the coprocessor. The advantage of the second option is that, since you are NFS mounting the home directories, any files you want to access on the coprocessor can be copied to the micuser home directory on the host and they will be there when you want them.
For the second solution, you can create a /var/mpss/micN/etc/sysconfig/coi.conf or /var/mpss/micN/etc/coi.conf file (where N is the number of the coprocessor you will be using) where you set coiuser to a valid user name. Some sites change the value of coiuser and reboot the coprocessor between jobs. However these site also restrict access to the host so that only one user has access at a time. If they didn't do this then everyone on the host would end up running as that particular user when they offload to the coprocessor.
My recommendation is the first solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thank you both for your input. That was indeed the issue. After creating the micuser on the phi the offload example runs. Here's the output I see:
[Offload] [HOST] [State] Initialize logical card 0 = physical card 0 [Offload] [HOST] [State] Initialize logical card 1 = physical card 1 [Offload] [MIC 0] [File] reduce_omp_mic.c [Offload] [MIC 0] [Line] 17 [Offload] [MIC 0] [Tag] Tag 0 [Offload] [HOST] [Tag 0] [State] Start Offload [Offload] [HOST] [Tag 0] [State] Initialize function __offload_entry_reduce_omp_mic_c_17mainicc572303114teczsI [Offload] [HOST] [Tag 0] [State] Send pointer data [Offload] [HOST] [Tag 0] [State] CPU->MIC pointer data 0 [Offload] [HOST] [Tag 0] [State] Gather copyin data [Offload] [HOST] [Tag 0] [State] CPU->MIC copyin data 20 [Offload] [HOST] [Tag 0] [State] Compute task on MIC [Offload] [HOST] [Tag 0] [State] Receive pointer data [Offload] [HOST] [Tag 0] [State] MIC->CPU pointer data 0 [Offload] [MIC 0] [Tag 0] [State] Start target function __offload_entry_reduce_omp_mic_c_17mainicc572303114teczsI [Offload] [MIC 0] [Tag 0] [Var] i INOUT [Offload] [MIC 0] [Tag 0] [Var] n INOUT [Offload] [MIC 0] [Tag 0] [Var] sum INOUT [Offload] [MIC 0] [Tag 0] [Var] nt INOUT [Offload] [MIC 0] [Tag 0] [State] Scatter copyin data Hello MIC reduction 200000010000000.000000 threads: 120 [Offload] [MIC 0] [Tag 0] [State] Gather copyout data [Offload] [HOST] [Tag 0] [State] Scatter copyout data [Offload] [HOST] [Tag 0] [CPU Time] 0.677483(seconds) [Offload] [MIC 0] [Tag 0] [CPU->MIC Data] 20 (bytes) [Offload] [MIC 0] [Tag 0] [MIC Time] 0.135097(seconds) [Offload] [MIC 0] [Tag 0] [MIC->CPU Data] 20 (bytes) [Offload] [MIC 0] [Tag 0] [State] MIC->CPU copyout data 20 [Offload] [MIC 1] [State] Unregister data tables [Offload] [MIC 0] [State] Unregister data tables [Offload] [HOST] [State] Unregister data tables
Is this consistent with what you see in your environments? I'm mainly wondering about the last three "Unregister data tables" messages.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
"Unregister data tables" is just coi cleaning up after itself. Looks like you are good to go.
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page