topic Problem accessing the sampling driver in MIC. in Analyzers

Problem accessing the sampling driver in MIC.

Surya_Narayanan_N_ — Tue, 14 Jan 2014 10:36:51 GMT

Am using mic in network mode. So after the admin restarts the PC i am able to run my test program

amplxe-cl -collect knc-general-exploration ssh mic0 "/home/snataraj/test/nsleep_mic" (Here it works fine)

after that, when i try to run my benchmark something goes wrong and my vtune fails to connect to mic0. (At times I kill both amplxe-cl and amplxe-runss process)

If I try my previous test program again I get the error and henceforth till the PC is rebooted my account has this issue.

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card <1:1046>, errno 111amplxe: Error: Problem accessing the sampling driver. The driver may need to be (re)started. See Installing the Sampling Driver help topic to learn how to configure the sampling driver.
amplxe: Internal Error

And I don't have admin rights to restart mpss service or to perform any of the 3 ways to rectify this error. What kind of modification should my admin make in my account to get rid of this issue?

Running "service mpss restart

Peter_W_Intel — Wed, 15 Jan 2014 03:34:23 GMT

Running "service mpss restart" to restart mic device, vtune driver will be ready, but admin privilege is required. Restarting PC will be helpful since mic device will be restart. Running such as "ssh mic0" to check "lsmod | grep sep3" to know if vtune driver is loaded, if not you need to install vtune driver (sep_micboot_install.sh) from /vtune_amplifier_xe_2013/bin64/k1om, I think.

I have the same problem with

Sebastian_Kalcher — Tue, 28 Jan 2014 15:51:44 GMT

I have the same problem with Vtune:

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...

[...]

I tried your suggestion (/opt/intel/vtune_amplifier_xe/bin64/k1om)

sudo ./sep_micboot_install.sh

but it returns with

cannot find sep driver

I noticed the kernel modules in the same directory:

-rw-r--r-- 1 root root 1419009 Aug 7 14:36 sep3_10-k1om-2.6.38.8-g5f2543dsmp.ko

-rw-r--r-- 1 root root 1424513 Aug 7 14:36 sep3_10-k1om-2.6.38.8+mpss3.0smp.ko

-rw-r--r-- 1 root root 1423385 Aug 7 14:36 sep3_10-k1om-2.6.38.8smp.ko

However the kernel version running on the MIC is

uname -r

2.6.38.8+mpss3.1.2

Dissecting the install script shows, that it is looking for

sep3_10-k1om-2.6.38.8+mpss3.1.2smp.ko

So it looks like Vtune doesn't come with the latest kernel modules.

VTune Amplifier doesn't

robert-reed — Tue, 28 Jan 2014 16:51:00 GMT

VTune Amplifier doesn't support it? Perhaps you need a newer VTune Amplifier? I have VTune Amplifier XE 2013 Update 15 installed, and when I repeat your experiment I get this

$ ls *.ko
sep3_10-k1om-2.6.38.8-g5f2543dsmp.ko
sep3_10-k1om-2.6.38.8+mpss3.1.1smp.ko
sep3_10-k1om-2.6.38.8+mpss3.0smp.ko

I have MPSS 3.1.2 installed on this machine and these drivers are close enough for me to be able to collect coprocessor data. Perhaps all you need is a VTune Amplifier upgrade to match your MPSS version?

I found a solution to build

Sebastian_Kalcher — Tue, 28 Jan 2014 17:17:27 GMT

I found a solution to build the sep kernel modules for the current MIC kernel.

The sep SDK for MIC can be found in

/opt/intel/vtune_amplifier_xe/mic_sepdk/src

there you find the script

build_mic_driver.sh

Instructions can be found in the file (have a look at the environment vars)

README_MIC.txt

The MIC kernel sources are needed for the compilation. You have to download the MPSS sources and find the file

linux-2.6.38+mpss3.1.2.tar.bz2

Unpack the MIC kernel into

/tmp/MPSS/mic_linux

cd into this directory and prepare the kernel

cp config-mic-knc-2.6.38 .config

make defconfig-miclinux

make modules_prepare

Now go back to

/opt/intel/vtune_amplifier_xe/mic_sepdk/src

and

sudo ./build_mic_driver.sh

This builds a new kernel module in the same directory

sep3_10-k1om-2.6.38.8+mpss3.1.2smp.ko

Copy this file to

/opt/intel/vtune_amplifier_xe/bin64/k1om

and install it to the MIC

sudo ./sep_micboot_install.sh

and restart MPSS

sudo service mpss restart

That should do it.

Well, yeah, that would work,

robert-reed — Tue, 28 Jan 2014 17:17:57 GMT

Well, yeah, that would work, too. And congratulations in making it through the extra work to make this work. My suggestion should have also worked, trading your source download and kernel rebuilt time for a fresh product download time, which should have produced an installation with a prebuilt coprocessor driver that should also have worked and has the added advantage that it's using the latest release of VTune Amplifier, with whatever additional bug fixes and feature releases that come with the new code. But I am happy to hear that you found a solution on your own. Congratulations!

I'm not sure any of this helps Surya, though. Surya, the failure mode you describe is one I have not seen reported by anyone else. Is it possible that either the MPSS or VTune Amplifier installations suffered installation errors that were not detected? Perhaps the trouble you are seeing results from some trouble with the installed code? If that is not the case, is there something special your test code is doing that may be leading VTune Amplifier to fail? I'm not sure what that would be, but as a diagnostic, it would be useful to know whether your system suffers the same fate running a simple workload.

Robert,

Sebastian_Kalcher — Tue, 28 Jan 2014 20:27:50 GMT

Robert,

thanks. I finished working on this before I could read your reply. Yes, updating VTune is definitely easier. I also observed this with MPSS 3.1.0 and a fresh install of Cluster Studio. I didn't occur to me to check for more recent versions of Vtune. Anyway, thanks again. At least I learned something about the architecture of the MIC ecosystem.

@Robert

Surya_Narayanan_N_ — Wed, 29 Jan 2014 10:36:26 GMT

@Robert

This problem occurs very occasionally. I try to run benchmark with VTUNE and with various inputset sizes. For some input size this problem occurs. But the normal execution of the program has no issues.

When I run the benchmark for like 64 threads at times while reporting VTUNE crashes at some tb6 file corrupt.

I am using

~ $ uname -r
2.6.38.8-g2593b11

Intel(R) VTune(TM) Amplifier XE 2013 Update 7

I hate "occasional problems"-

robert-reed — Tue, 18 Feb 2014 17:45:44 GMT

I hate "occasional problems"--they tax our patience to reproduce them. It does sound like perhaps your benchmark imposes stresses to the system that result in hanging the collection driver, but you also note above that you're using Update 7, which is getting pretty old in VTune Amplifier years--we're already up to Update 15 of the 2013 product. With a gap of that many releases, there's a chance that the intervening changes may have already closed the bug that is hanging your collections, though of course there's no guarantee of that.

Since it sounds like you're at the mercy of a computer center staff, I would suggest that you arrange with them to install a VTune Amplifier closer to the current released version, because even if you could discover a reliable way to cause the collection driver to hang in the fashion you describe, our developers would want to have confirmation that the problem still exists in the newest releases.

Hello,

Martin_K_2 — Fri, 18 Jul 2014 21:24:25 GMT

Hello,

I need a little help. I have a problem similar to Surya's. When I execute amplxe-cl:

amplxe-cl -collect knc-hotspots --search-dir all:rp=./ -- ssh mic0 /root/mymic

I obtain these errors:

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...

ERROR connecting to MIC card <1:1046>, errno 111amplxe: Error: Problem accessing the sampling driver. The driver may need to be (re)started. See Installing the Sampling Driver help topic to learn how to configure the sampling driver.

amplxe: Internal Error

However, I have these files in /opt/intel/composerxe2013/vtune_amplifier_xe_2013/bin64/k1om

sep3_10-k1om-2.6.38.8-g5f2543dsmp.ko

sep3_10-k1om-2.6.38.8+mpss3.0smp.ko

sep3_10-k1om-2.6.38.8smp.ko

I run sep_micboot_install.sh and get this other error:

cannot find MPSS kernel information

I'm lost in here. Any help would be appreciated. Thanks

A possibility seems to be

TimP — Fri, 18 Jul 2014 21:35:43 GMT

A possibility seems to be that you may have updated your MPSS without upgrading to a consistent version of VTune. When such changes happened in the past, the VTune releases lagged a bit behind the MPSS.