Problem accessing the sampling driver in MIC.

[Surya_Narayanan_N_]

Am using mic in network mode. So after the admin restarts the PC i am able to run my test program 

amplxe-cl -collect knc-general-exploration  ssh mic0 "/home/snataraj/test/nsleep_mic"  (Here it works fine)

after that, when i try to run my benchmark something goes wrong and my vtune fails to connect to mic0. (At times I kill both amplxe-cl and amplxe-runss process)

If I try my previous test program again I get the error and henceforth till the PC is rebooted my account has this issue.

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card <1:1046>, errno 111amplxe: Error: Problem accessing the sampling driver. The driver may need to be (re)started. See Installing the Sampling Driver help topic to learn how to configure the sampling driver.
amplxe: Internal Error

And I don't have admin rights to restart mpss service or to perform any of the 3 ways to rectify this error. What kind of modification should my admin make in my account to get rid of this issue?

[Peter_W_Intel]

Running "service mpss restart" to restart mic device, vtune driver will be ready, but admin privilege is required. Restarting PC will be helpful since mic device will be restart. Running such as "ssh mic0" to check "lsmod | grep sep3" to know if vtune driver is loaded, if not you need to install vtune driver (sep_micboot_install.sh) from /vtune_amplifier_xe_2013/bin64/k1om, I think.

[Sebastian_Kalcher]

I have the same problem with Vtune:

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...

[...]

I tried your suggestion (/opt/intel/vtune_amplifier_xe/bin64/k1om)

sudo ./sep_micboot_install.sh 

but it returns with

cannot find sep driver​

I noticed the kernel modules in the same directory:

-rw-r--r-- 1 root root 1419009 Aug  7 14:36 sep3_10-k1om-2.6.38.8-g5f2543dsmp.ko

-rw-r--r-- 1 root root 1424513 Aug  7 14:36 sep3_10-k1om-2.6.38.8+mpss3.0smp.ko

-rw-r--r-- 1 root root 1423385 Aug  7 14:36 sep3_10-k1om-2.6.38.8smp.ko

However the kernel version running on the MIC is

uname -r

2.6.38.8+mpss3.1.2

Dissecting the install script shows, that it is looking for 

sep3_10-k1om-2.6.38.8+mpss3.1.2smp.ko

So it looks like Vtune doesn't come with the latest kernel modules.

[robert-reed]

VTune Amplifier doesn't support it?  Perhaps you need a newer VTune Amplifier?  I have VTune Amplifier XE 2013 Update 15 installed, and when I repeat your experiment I get this
 

$ ls *.ko
sep3_10-k1om-2.6.38.8-g5f2543dsmp.ko  
sep3_10-k1om-2.6.38.8+mpss3.1.1smp.ko
sep3_10-k1om-2.6.38.8+mpss3.0smp.ko

I have MPSS 3.1.2 installed on this machine and these drivers are close enough for me to be able to collect coprocessor data.  Perhaps all you need is a VTune Amplifier upgrade to match your MPSS version?

[Sebastian_Kalcher]

I found a solution to build the sep kernel modules for the current MIC kernel.

The sep SDK for MIC can be found in 

/opt/intel/vtune_amplifier_xe/mic_sepdk/src

there you find the script

build_mic_driver.sh

Instructions can be found in the file (have a look at the environment vars)

README_MIC.txt 

The MIC kernel sources are needed for the compilation. You have to download the MPSS sources and find the file

linux-2.6.38+mpss3.1.2.tar.bz2 

Unpack the MIC kernel into

/tmp/MPSS/mic_linux

cd into this directory and prepare the kernel 

cp config-mic-knc-2.6.38 .config

make defconfig-miclinux

make modules_prepare

Now go back to 

/opt/intel/vtune_amplifier_xe/mic_sepdk/src

and 

sudo ./build_mic_driver.sh

This builds a new kernel module in the same directory

sep3_10-k1om-2.6.38.8+mpss3.1.2smp.ko

Copy this file to

/opt/intel/vtune_amplifier_xe/bin64/k1om

and install it to the MIC

sudo ./sep_micboot_install.sh

and restart MPSS

sudo service mpss restart 

That should do it.

[robert-reed]

Well, yeah, that would work, too.  And congratulations in making it through the extra work to make this work.  My suggestion should have also worked, trading your source download and kernel rebuilt time for a fresh product download time, which should have produced an installation with a prebuilt coprocessor driver that should also have worked and has the added advantage that it's using the latest release of VTune Amplifier, with whatever additional bug fixes and feature releases that come with the new code.  But I am happy to hear that you found a solution on your own.  Congratulations!

I'm not sure any of this helps Surya, though.  Surya, the failure mode you describe is one I have not seen reported by anyone else.  Is it possible that either the MPSS or VTune Amplifier installations suffered installation errors that were not detected?  Perhaps the trouble you are seeing results from some trouble with the installed code?  If that is not the case, is there something special your test code is doing that may be leading VTune Amplifier to fail?  I'm not sure what that would be, but as a diagnostic, it would be useful to know whether your system suffers the same fate running a simple workload.

[Sebastian_Kalcher]

Robert,

thanks. I finished working on this before I could read your reply. Yes, updating VTune is definitely easier. I also observed this with MPSS 3.1.0 and a fresh install of Cluster Studio. I didn't occur to me to check for more recent versions of Vtune. Anyway, thanks again. At least I learned something about the architecture of the MIC ecosystem.

[Surya_Narayanan_N_]

@Robert

This problem occurs very occasionally. I try to run benchmark with VTUNE  and with various inputset sizes. For some input size this problem occurs. But the normal execution of the program has no issues.

When I run the benchmark for like 64 threads at times while reporting VTUNE crashes at some tb6 file corrupt. 

I am using

~ $ uname -r
2.6.38.8-g2593b11

Intel(R) VTune(TM) Amplifier XE 2013 Update 7

[robert-reed]

I hate "occasional problems"--they tax our patience to reproduce them.  It does sound like perhaps your benchmark imposes stresses to the system that result in hanging the collection driver, but you also note above that you're using Update 7, which is getting pretty old in VTune Amplifier years--we're already up to Update 15 of the 2013 product.  With a gap of that many releases, there's a chance that the intervening changes may have already closed the bug that is hanging your collections, though of course there's no guarantee of that.  

Since it sounds like you're at the mercy of a computer center staff, I would suggest that you arrange with them to install a VTune Amplifier closer to the current released version, because even if you could discover a reliable way to cause the collection driver to hang in the fashion you describe, our developers would want to have confirmation that the problem still exists in the newest releases.

[Martin_K_2]

Hello,

I need a little help. I have a problem similar to Surya's. When I execute amplxe-cl:

amplxe-cl -collect knc-hotspots --search-dir all:rp=./ -- ssh mic0 /root/mymic

I obtain these errors:

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...

ERROR connecting to MIC card <1:1046>, errno 111amplxe: Error: Problem accessing the sampling driver. The driver may need to be (re)started. See Installing the Sampling Driver help topic to learn how to configure the sampling driver.

amplxe: Internal Error

However, I have these files in /opt/intel/composerxe2013/vtune_amplifier_xe_2013/bin64/k1om

sep3_10-k1om-2.6.38.8-g5f2543dsmp.ko

sep3_10-k1om-2.6.38.8+mpss3.0smp.ko

sep3_10-k1om-2.6.38.8smp.ko

I run sep_micboot_install.sh and get this other error:

cannot find MPSS kernel information

I'm lost in here. Any help would be appreciated. Thanks

[TimP]

A possibility seems to be that you may have updated your MPSS without upgrading to a consistent version of VTune.   When such changes happened in the past, the VTune releases lagged a bit behind the MPSS.