Error getting OFED to compile when mic is selected.

Holstad__Dan · ‎05-22-2015

Compiling OFED with with phi and --all fails when compiling compat-rdma.
Compile without the "--with-xeon-phi" option works. 


./install.pl --with-xeon-phi --all

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.1 (Maipo)

# uname -r
3.10.0-229.1.2.el7.x86_64

rpm -qi mpss-sdk-k1om-3.5-1.x86_64
Name        : mpss-sdk-k1om
Version     : 3.5
Release     : 1
Architecture: x86_64
Install Date: Thu 09 Apr 2015 10:00:28 PM CDT
Group       : base
Size        : 484359036
License     : various
Signature   : DSA/SHA1, Thu 02 Apr 2015 06:57:59 AM CDT, Key ID
718a1696ef328191
Source RPM  : mpss-sdk-k1om-3.5-1.src.rpm
Build Date  : Thu 02 Apr 2015 06:57:59 AM CDT
Build Host  : yocto-182-62
Relocations : (not relocatable)
Packager    : Intel Corporation
URL         : http://software.intel.com/xeonphi
Summary     : SDK package for MPSS on Intel MIC co-processors
Description :
A self-contained cross-toolchain for building programs for k1om co-
processors running MPSS.




compat-rdma.rpmbuild.log contains the following errors

CC  
/var/tmp/OFED_topdir/BUILD/compat-rdma-3.18/drivers/infiniband/hw/qib/qib_cq.o
  CC  
/var/tmp/OFED_topdir/BUILD/compat-rdma-3.18/drivers/infiniband/hw/qib/qib_diag.o
In file included from include/asm-generic/bug.h:13:0,
                 from
/usr/src/kernels/3.10.0-229.1.2.el7.x86_64/arch/x86/include/asm/bug.h:38,
                 from include/linux/bug.h:4,
                 from include/linux/thread_info.h:11,
                 from include/linux/preempt.h:9,
                 from include/linux/spinlock.h:50,
                 from include/linux/wait.h:7,
                 from include/linux/fs.h:6,
                 from
/var/tmp/OFED_topdir/BUILD/compat-rdma-3.18/include/linux/compat-3.5.h:5,
                 from
/var/tmp/OFED_topdir/BUILD/compat-rdma-3.18/include/linux/compat-2.6.h:64,
                 from <command-line>:0:
/var/tmp/OFED_topdir/BUILD/compat-rdma-3.18/drivers/infiniband/hw/mthca/mthca_memfree.c:
In function 'mthca_map_user_db':
include/linux/kernel.h:844:27: error: dereferencing pointer to incomplete type
  const typeof( ((type *)0)->member ) *__mptr = (ptr); \
                           ^
include/linux/list.h:351:2: note: in expansion of macro 'container_of'
  container_of(ptr, type, member)
  ^
/var/tmp/OFED_topdir/BUILD/compat-rdma-3.18/drivers/infiniband/hw/mthca/mthca_memfree.c:500:10:
note: in expansion of macro 'list_entry'
  chunk = list_entry(db_tab->page.umem->chunk_list.next,
          ^
/var/tmp/OFED_topdir/BUILD/compat-rdma-3.18/drivers/infiniband/hw/mthca/mthca_memfree.c:500:41:
error: 'struct ib_umem' has no member named 'chunk_list'
  chunk = list_entry(db_tab->page.umem->chunk_list.next,
                                         ^
include/linux/kernel.h:844:49: note: in definition of macro 'container_of'
  const typeof( ((type *)0)->member ) *__mptr = (ptr); \
                                                 ^
/var/tmp/OFED_topdir/BUILD/compat-rdma-3.18/drivers/infiniband/hw/mthca/mthca_memfree.c:500:10:
note: in expansion of macro 'list_entry'
  chunk = list_entry(db_tab->page.umem->chunk_list.next,
          ^
In file included from include/linux/compiler-gcc.h:103:0,
                 from include/linux/compiler.h:54,
                 from include/uapi/linux/stddef.h:1,
                 from include/linux/stddef.h:4,
                 from include/linux/posix_types.h:4,
                 from include/uapi/linux/types.h:13,
                 from include/linux/types.h:5,

Frances_R_Intel · ‎05-27-2015

I am unclear what your goal is here. Are you trying to compile a new version of OFED for the coprocessor or are you trying to compile a new versions of OFED for the host processor?

Normally, you would not compile a new version of OFED for the coprocessor but instead use the precompiled one which came with the MPSS. That version will install on the coprocessor by default. I don't know what changes would be needed to the source code for OFED to make it run properly on the coprocessor and I don't believe there is any available OFED source code that can be compiled to run on the coprocessor without change, other than the version that comes with the MPSS. If you decided you wanted to play around with this, you would need to set up the compile environment, as shown in https://software.intel.com/en-us/forums/topic/401046#comment-1786509, to be sure you pick up the right libraries, include files and loader, in addition to the compiler.

If you wish to compile OFED for the host system, using code from OFA, you would use either OFED-1.5.4.1, OFED-3.5-2-MIC or OFED-3.12-1. This last version you would compile using the --with-xeon-phi flag. I don't know what the effect of adding the --all flag would be. You would compile using a version of gcc for the host system, not the compiler from mpss-sdk-k1om. OFED-3.18 is not one of the officially supported versions for the host. This doesn't mean you are not welcome to try it - only that we might not be able to track down answers to questions if things go wrong.

Does that help any?

Holstad__Dan · ‎05-28-2015

I am trying to configure this on the host. Its rhel7, and I was having issues getting a clean compile from any of the branches of OFED. I was having issues getting any version of OFED to compile and work with the infiniband virtual drivers. I have included a list of my issues below. We decided to try 3.18 daily because it was the newest and was supposed to be patched against rhel 7.1 (which is what we are installing on). We were able to get a clean installation by using the just patched version of OFED after opening a ticket with OFED, but are still having issues with the device showing up but is getting "invalid transport".

Compile error: http://bugs.openfabrics.org/bugzilla/show_bug.cgi?id=2543

Invalid Transport: http://bugs.openfabrics.org/bugzilla/show_bug.cgi?id=2547

Issues with OFED versions:

1)   Install intel compiles (worked fine)
2)   2) Install OFED to drive the IB controllers
a.   Version 3.12 does not compile cleanly on redhat 7
include/linux/netdevice.h: In function 'napi_enable':
include/linux/netdevice.h:504:2: error: implicit declaration of function 'smp_mb__before_clear_bit' [-Werror=implicit-function-declaration]
smp_mb__before_clear_bit();
b.   Version 3.18 does not compile on redhat 7
from <command-line>:0:
/var/tmp/OFED_topdir/BUILD/compat-rdma-3.18/drivers/infiniband/hw/mthca/mthca_memfree.c: In function 'mthca_map_user_db':
include/linux/kernel.h:844:27: error: dereferencing pointer to incomplete type
const typeof( ((type *)0)->member ) *__mptr = (ptr); \
c.   Version 3.5-mic also does not compile on rhel7 because of the uapi kernel header split
include/linux/compat.h:14:22: fatal error: linux/if.h: No such file or directory
#include <linux/if.h>
d.   Mellonox drivers (these do install but require the intel ofed drvers to work…)
3)   Recompile intel OFED driver for kernel ( fails due to uapi kernel headers split)