Xeon Phi not using more than 1 or 2 cores

[Nick_W_]

Hi,

We have an engineering sample of the Xeon Phi (60 core version). We have installed the mpss_gold_update_3-2.1.6720-16-rhel-6.4

1) We followed the instructions to upgrade flash but micinfo is reporting this:

Host: Linux

OS version: 2.6.32-358.el6.x86_64

Driver version: 6720-16

MPSS version: 2.1.6720-16

BUT: flash version, SMC, UOS and device serial number are all reported as "NotAvailable"...

2) micsmc produces error messages every few seconds such as "Warning: mic0 device connection lost!" and "Information: mic0: Device connection restored"

3) We have a C++ native application running on the Phi. It can be configured to run multithreaded (pthreads), we have taken benchmarks. It runs fractionally more slowly in 240 threads (60 cores x 4 HW threads per core) compared to a single thread... (we have accounted for the overhead of starting new threads)

The application runs against a fixed size sample data as follows:

- when compiled for the host system: 0.2 seconds

- when run on the Phi in a single thread: 0.59 seconds

- when run on the Phi in 240 threads: 0.6 seconds

Repeated runs while micsmc is running shows that no more than 2 cores are being used at any one time...

Any help with this would be much appreciated

[Victor_L_2]

Nick,

I had no problems running on CentOS 6.4. In order to make CentOS 6.4 machine appear as RedHat 6.4 do the following (as root):

1. rm /etc/redhat-release
2. echo "Red Hat Enterprise Linux Server release 6.4 (Santiago)" > /etc/redhat-release

I've being told the release version doesn't really matter, as long as it says "Red Hat Enterprise Linux Server ..."

[robert-reed]

The combination of symptoms you report are confusing me a little bit.  It sounds like you're able to run your native code on the coprocessor, and even now see some scaling now that you've upped the thread count, but still have problems with the VTune Amplifier collector on the coprocessor, and on getting status from the coprocessor via micinfo.  Sounds like Intel MPSS is at least partially running.  If you do a service mpss start/restart (one or the other) and then immediately run micinfo under sudo or some other root enabler, are you still getting NotAvailable for the coprocessor specific fields?  If those fields are still coming up empty, my first suspicion would be a mismatch between the MPSS version installed and the flash downloaded into the coprocessor.  Getting a clean run of micinfo would be my first priority.  I was hoping that miccheck might show something; the lack of that confiirmation is part of what is confusing here.

[Nick_W_]

Yes, I get the NotAvailable message regardless...

I installed the software we have from the following files:

cpp_studio_xe_2013_update1_intel64

mpss_gold_update_3-2.1.6720-16-rhel-6.4

As far as I can tell, everything is working normally apart from the install won't install the sampling drivers onto the mic.

I tried making the install think it is Redhat but that hasn't fixed the sampling drivers problem. I think we probably have an old version of amplifier which still has the sep_micboot script. It compares what drivers the mic is expecting compared to what drivers it has - and the mic wants version _38 drivers which the install doesn't have...

[robert-reed]

And apart from the fact that you're getting NotAvailable for some of the micinfo fields.  And that could be a problem regardless of the version of VTune Amplifier that you have.  Let me run a little demonstration and see whether any of this has any bearing on what you're seeing.

If I shut down mpss as a service on a machine, then try to run micinfo, I see something like the following

[bash]

$ sudo /opt/intel/mic/bin/micinfo
MicInfo Utility Log

Created Tue Sep 24 14:14:00 2013

System Info
HOST OS : Linux
OS Version : 2.6.32-220.el6.x86_64
Driver Version : 6720-16
MPSS Version : 2.1.6720-16
Host Physical Memory : 65923 MB

Device No: 0, Device Name: mic0

Version
Flash Version : NotAvailable
SMC Firmware Version : NotAvailable
SMC Boot Loader Version : NotAvailable
uOS Version : NotAvailable
Device Serial Number : NotAvailable

Board
Vendor ID : 0x8086
Device ID : 0x225d
Subsystem ID : 0x3608
Coprocessor Stepping ID : 2
PCIe Width : x16

[/bash]

I get NotAvailable values, particularly regarding the details of the flash version and other version parameters associated with the coprocessor.  However, for this machine all I need to do is start mpss running, the picture changes:

[bash]

$ sudo service mpss start
[sudo] password for rreed:
Starting MPSS Stack: [ OK ]
mic0: online (mode: linux image: /lib/firmware/mic/uos.img)
mic1: online (mode: linux image: /lib/firmware/mic/uos.img)
[rreed@orspt-le64-82:~]
$ sudo /opt/intel/mic/bin/micinfo
MicInfo Utility Log

Created Tue Sep 24 14:21:53 2013

System Info
HOST OS : Linux
OS Version : 2.6.32-220.el6.x86_64
Driver Version : 6720-16
MPSS Version : 2.1.6720-16
Host Physical Memory : 65923 MB

Device No: 0, Device Name: mic0

Version
Flash Version : 2.1.03.0386
SMC Firmware Version : 1.15.4830
SMC Boot Loader Version : 1.8.4326
uOS Version : 2.6.38.8-g2593b11
Device Serial Number : ADKC30900885

Board
Vendor ID : 0x8086
Device ID : 0x225d
Subsystem ID : 0x3608
Coprocessor Stepping ID : 2
PCIe Width : x16

[/bash]

All the NotAvailable fields have been filled in.  Note that at no time did I do anything regarding VTune Amplifier.  If your machine does not behave in a similar fashion, then there may be something more fundamental than merely an old copy of VTune Amplifier.  In particular, if the NotAvailable fields you're seeing include the fields that indicate the flash and SMC firmware versions, micinfo will not reveal their values.  With what we know, we cannot determine whether flash and SMC firmware versions are in sync with your MPSS version.

[Nick_W_]

I am pretty sure that (at least one of) the problem(s) is that our composer software is old. We installed an evaluation copy of vtune yesterday and it works fine on our native Phi application. This hasn't fixed the micinfo problem (and I have certainly tried restarting mpss). We're going to purchase a license for the latest release which will take time to be authorised. In the meantime we seem to be getting good performance from the Phi - well within the parameters we were hoping for. Thanks for your help.