- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Am using mic in network mode. So after the admin restarts the PC i am able to run my test program
amplxe-cl -collect knc-general-exploration ssh mic0 "/home/snataraj/test/nsleep_mic" (Here it works fine)
after that, when i try to run my benchmark something goes wrong and my vtune fails to connect to mic0. (At times I kill both amplxe-cl and amplxe-runss process)
If I try my previous test program again I get the error and henceforth till the PC is rebooted my account has this issue.
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card <1:1046>, errno 111amplxe: Error: Problem accessing the sampling driver. The driver may need to be (re)started. See Installing the Sampling Driver help topic to learn how to configure the sampling driver.
amplxe: Internal Error
And I don't have admin rights to restart mpss service or to perform any of the 3 ways to rectify this error. What kind of modification should my admin make in my account to get rid of this issue?
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Running "service mpss restart" to restart mic device, vtune driver will be ready, but admin privilege is required. Restarting PC will be helpful since mic device will be restart. Running such as "ssh mic0" to check "lsmod | grep sep3" to know if vtune driver is loaded, if not you need to install vtune driver (sep_micboot_install.sh) from /vtune_amplifier_xe_2013/bin64/k1om, I think.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have the same problem with Vtune:
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
[...]
I tried your suggestion (/opt/intel/vtune_amplifier_xe/bin64/k1om)
sudo ./sep_micboot_install.sh
but it returns with
cannot find sep driver
I noticed the kernel modules in the same directory:
-rw-r--r-- 1 root root 1419009 Aug 7 14:36 sep3_10-k1om-2.6.38.8-g5f2543dsmp.ko
-rw-r--r-- 1 root root 1424513 Aug 7 14:36 sep3_10-k1om-2.6.38.8+mpss3.0smp.ko
-rw-r--r-- 1 root root 1423385 Aug 7 14:36 sep3_10-k1om-2.6.38.8smp.ko
However the kernel version running on the MIC is
uname -r
2.6.38.8+mpss3.1.2
Dissecting the install script shows, that it is looking for
sep3_10-k1om-2.6.38.8+mpss3.1.2smp.ko
So it looks like Vtune doesn't come with the latest kernel modules.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
VTune Amplifier doesn't support it? Perhaps you need a newer VTune Amplifier? I have VTune Amplifier XE 2013 Update 15 installed, and when I repeat your experiment I get this
$ ls *.ko
sep3_10-k1om-2.6.38.8-g5f2543dsmp.ko
sep3_10-k1om-2.6.38.8+mpss3.1.1smp.ko
sep3_10-k1om-2.6.38.8+mpss3.0smp.ko
I have MPSS 3.1.2 installed on this machine and these drivers are close enough for me to be able to collect coprocessor data. Perhaps all you need is a VTune Amplifier upgrade to match your MPSS version?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I found a solution to build the sep kernel modules for the current MIC kernel.
The sep SDK for MIC can be found in
/opt/intel/vtune_amplifier_xe/mic_sepdk/src
there you find the script
build_mic_driver.sh
Instructions can be found in the file (have a look at the environment vars)
README_MIC.txt
The MIC kernel sources are needed for the compilation. You have to download the MPSS sources and find the file
linux-2.6.38+mpss3.1.2.tar.bz2
Unpack the MIC kernel into
/tmp/MPSS/mic_linux
cd into this directory and prepare the kernel
cp config-mic-knc-2.6.38 .config
make defconfig-miclinux
make modules_prepare
Now go back to
/opt/intel/vtune_amplifier_xe/mic_sepdk/src
and
sudo ./build_mic_driver.sh
This builds a new kernel module in the same directory
sep3_10-k1om-2.6.38.8+mpss3.1.2smp.ko
Copy this file to
/opt/intel/vtune_amplifier_xe/bin64/k1om
and install it to the MIC
sudo ./sep_micboot_install.sh
and restart MPSS
sudo service mpss restart
That should do it.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Well, yeah, that would work, too. And congratulations in making it through the extra work to make this work. My suggestion should have also worked, trading your source download and kernel rebuilt time for a fresh product download time, which should have produced an installation with a prebuilt coprocessor driver that should also have worked and has the added advantage that it's using the latest release of VTune Amplifier, with whatever additional bug fixes and feature releases that come with the new code. But I am happy to hear that you found a solution on your own. Congratulations!
I'm not sure any of this helps Surya, though. Surya, the failure mode you describe is one I have not seen reported by anyone else. Is it possible that either the MPSS or VTune Amplifier installations suffered installation errors that were not detected? Perhaps the trouble you are seeing results from some trouble with the installed code? If that is not the case, is there something special your test code is doing that may be leading VTune Amplifier to fail? I'm not sure what that would be, but as a diagnostic, it would be useful to know whether your system suffers the same fate running a simple workload.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Robert,
thanks. I finished working on this before I could read your reply. Yes, updating VTune is definitely easier. I also observed this with MPSS 3.1.0 and a fresh install of Cluster Studio. I didn't occur to me to check for more recent versions of Vtune. Anyway, thanks again. At least I learned something about the architecture of the MIC ecosystem.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Robert
This problem occurs very occasionally. I try to run benchmark with VTUNE and with various inputset sizes. For some input size this problem occurs. But the normal execution of the program has no issues.
When I run the benchmark for like 64 threads at times while reporting VTUNE crashes at some tb6 file corrupt.
I am using
~ $ uname -r
2.6.38.8-g2593b11
Intel(R) VTune(TM) Amplifier XE 2013 Update 7
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I hate "occasional problems"--they tax our patience to reproduce them. It does sound like perhaps your benchmark imposes stresses to the system that result in hanging the collection driver, but you also note above that you're using Update 7, which is getting pretty old in VTune Amplifier years--we're already up to Update 15 of the 2013 product. With a gap of that many releases, there's a chance that the intervening changes may have already closed the bug that is hanging your collections, though of course there's no guarantee of that.
Since it sounds like you're at the mercy of a computer center staff, I would suggest that you arrange with them to install a VTune Amplifier closer to the current released version, because even if you could discover a reliable way to cause the collection driver to hang in the fashion you describe, our developers would want to have confirmation that the problem still exists in the newest releases.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
I need a little help. I have a problem similar to Surya's. When I execute amplxe-cl:
amplxe-cl -collect knc-hotspots --search-dir all:rp=./ -- ssh mic0 /root/mymic
I obtain these errors:
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card <1:1046>, errno 111amplxe: Error: Problem accessing the sampling driver. The driver may need to be (re)started. See Installing the Sampling Driver help topic to learn how to configure the sampling driver.
amplxe: Internal Error
However, I have these files in /opt/intel/composerxe2013/vtune_amplifier_xe_2013/bin64/k1om
sep3_10-k1om-2.6.38.8-g5f2543dsmp.ko
sep3_10-k1om-2.6.38.8+mpss3.0smp.ko
sep3_10-k1om-2.6.38.8smp.ko
I run sep_micboot_install.sh and get this other error:
cannot find MPSS kernel information
I'm lost in here. Any help would be appreciated. Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
A possibility seems to be that you may have updated your MPSS without upgrading to a consistent version of VTune. When such changes happened in the past, the VTune releases lagged a bit behind the MPSS.
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page