ofed-mic cannot load ipb_server kernel module on SL6.5

Taras_Shapovalov · ‎11-18-2014

Hello,

Does MPSS 3.4.1 support kernel 2.6.32-431.29.2? From the documentation I see it should. We have installed OFED-3.5-2-MIC as recommended on Scientific Linux 6.5 and ofed-mic driver is built successfully (MPSS 3.4.1), but it cannot be started because ibp_server cannot be loaded. With other kernels it works fine although (but in that particular case we need to use that kernel for now).

[root@node001 ~]# modprobe ibp_server
FATAL: Error inserting ibp_server (/lib/modules/2.6.32-431.29.2.el6.x86_64/updates/drivers/infiniband/ibp/drv/ibp_server.ko): Unknown symbol in module, or unknown parameter (see dmesg)

[root@node001 ~]# dmesg | tail
ibp_server: disagrees about version of symbol ib_query_pkey
ibp_server: Unknown symbol ib_query_pkey
ibp_server: disagrees about version of symbol ib_destroy_qp
ibp_server: Unknown symbol ib_destroy_qp
ibp_server: disagrees about version of symbol ib_dealloc_pd
ibp_server: Unknown symbol ib_dealloc_pd
ibp_server: disagrees about version of symbol ib_query_gid
ibp_server: Unknown symbol ib_query_gid
ibp_server: disagrees about version of symbol ib_attach_mcast
ibp_server: Unknown symbol ib_attach_mcast
[root@node001 ~]#

[root@node001 ~]# modinfo ibp_server
filename:       /lib/modules/2.6.32-431.29.2.el6.x86_64/updates/drivers/infiniband/ibp/drv/ibp_server.ko
version:        1.0
description:    CCL Direct Server
license:        Dual BSD/GPL
author:         Jay Sternberg
author:         Phil Cayton
author:         Jerrie Coffman
srcversion:     5E8E5CC8B8279276EFA9C77
depends:        mic,ib_core
vermagic:       2.6.32-431.29.2.el6.x86_64 SMP mod_unload modversions
parm:           port:Connection port (int)
parm:           backlog:Connection backlog (int)
parm:           timeout:Listen/Poll time in milliseconds (int)

[root@node001 ~]# uname -a
Linux node001 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 13:45:55 CDT 2014 x86_64 x86_64 x86_64 GNU/Linux

[root@node001 ~]# modinfo mic
filename:       /lib/modules/2.6.32-431.29.2.el6.x86_64/extra/mic.ko
license:        GPL
build_scmver:   f40adc1e51c97bded88a7c557b08f3a24a449fed
build_ondate:   2014-11-18 03:33:17 -0800
build_bywhom:   root@node001
build_number:   0
license:        GPL
license:        GPL
srcversion:     7709226C0A6F98BD6E74494
depends:
vermagic:       2.6.32-431.29.2.el6.x86_64 SMP mod_unload modversions
parm:           vnet:Vnet operating mode, one of: poll intr dma (vnetmode)
parm:           vnet_num_buffers:Number of buffers used by the VNET driver (int)
parm:           vnet_addr:Vnet driver host ring address (ulong)
parm:           ulimit:SCIF ulimit check (bool)
parm:           reg_cache:SCIF registration caching (bool)
parm:           huge_page:SCIF Huge Page Support (bool)
parm:           p2p:SCIF peer-to-peer (bool)
parm:           p2p_proxy:SCIF peer-to-peer proxy DMA support (bool)
parm:           watchdog:SCIF Watchdog (bool)
parm:           watchdog_auto_reboot:SCIF Watchdog auto reboot (bool)
parm:           msi:bool
parm:           mic_msi_enable:To enable MSIx in the driver.
parm:           pm_qos_cpu_dma_lat:int
parm:           mic_pm_qos_cpu_dma_lat:PM QoS CPU DMA latency in usecs.
parm:           ramoops_count:Maximum frame count for the ramoops driver. (int)
parm:           crash_dump:bool
parm:           mic_crash_dump_enabled:MIC Crash Dump enabled.
parm:           psmi:Enable/disable mic psmi (bool)
[root@node001 ~]#

Has anybody idea why it fails on that particular kernel?

Loc_N_Intel · ‎11-21-2014

Hi Taras,

I gave it a try yesterday. I succesfully installed MPSS 3.4.1 and ofed-3.5-2-MIC with the kernel-2.6.32-431.29.2

Here is the steps I did:

1. Install a new CentOS 6.5 (kernel-2.6.32-431.el6.x86_64).

2. Get the source RPM kernel-2.6.32-431.29.2.el6.src.rpm

3. Rebuild the source RPM:
# rpmbuild --rebuild kernel-2.6.32-431.29.2.el6.src.rpm
It indicated that some dependency packages are needed.

4. Get these dependency packages:
   asciidoc-8.4.5-4.1.el6.noarch.rpm,
   newt-devel-0.52.11-3.el6.x86_64.rpm,
   perl-ExtUtils-Embed-1.28-136.el6.x86_64.rpm,
   slang-devel-2.2.1-1.el6.x86_64.rpm,
   xmlto-0.0.23-3.el6.x86_64.rpm

5. Install the above packages:
# rpm -i *.rpm
They are installed successfully.

6. Reboot the system.

7. Rebuild the source RPM again. New kernel RPMs are found under ~/rpmbuild/RPMS/x86_64/ :
   kernel-2.6.32-431.29.2.el6.x86_64.rpm,
   kernel-debug-2.6.32-431.29.2.el6.x86_64.rpm,
   kernel-debug-debuginfo-2.6.32-431.29.2.el6.x86_64.rpm,
   kernel-debug-devel-2.6.32-431.29.2.el6.x86_64.rpm,
   kernel-debuginfo-2.6.32-431.29.2.el6.x86_64.rpm,
   kernel-debuginfo-common-x86_64-2.6.32-431.29.2.el6.x86_64.rpm,
   kernel-devel-2.6.32-431.29.2.el6.x86_64.rpm,
   kernel-headers-2.6.32-431.29.2.el6.x86_64.rpm,
   perf-2.6.32-431.29.2.el6.x86_64.rpm,
   perf-debuginfo-2.6.32-431.29.2.el6.x86_64.rpm,
   python-perf-2.6.32-431.29.2.el6.x86_64.rpm,
   python-perf-debuginfo-2.6.32-431.29.2.el6.x86_64.rpm

8. Install these newly builts:
# cd ~/rpmbuild/RPMS/x86_64
# yum install *.rpm
It indicates that package kernel-firmware-2.6.32-504.el6.noarch.rpm is needed.

9. Get and install kernel-firmware-2.6.32-504.el6.noarch.rpm
# rpm -i kernel-firmware-2.6.32-504.el6.noarch.rpm

10. Install the newly builts again:
# cd ~/rpmbuild/RPMS/x86_64
# yum install *.rpm
The new kernel is installed succesfully.

11. Reboot the system.

12. Dowload MPSS 3.4.1 and untar the package:
# tar xvf mpss-3.4.1-linux.tar

13. Because the host system is running with an updated kernel, re-build the MPSS modules:
# cd <path to extracted MPSS*-linux.tar>/src/
# rpmbuild --rebuild mpss-modules-3.4.1-1.src.rpm
That generates two new RPMs located at ~/rpmbuild/RPMS/x86_64/:
mpss-modules-2.6.32-431.29.2.el6.x86_64-3.4.1-1.x86_64.rpm
mpss-modules-dev-2.6.32-431.29.2.el6.x86_64-3.4.1-1.x86_64.rpm

14. Copy these two rebuilts MPSS modules to <path to extracted MPSS*-linux.tar>:
# cp ~/rpmbuild/RPMS/x86_64/mpss-modules-2.6.32-431.29.2.el6.x86_64-3.4.1-1.x86_64.rpm <path to extracted MPSS*-linux.tar>
# cp ~/rpmbuild/RPMS/x86_64/mpss-modules-dev-2.6.32-431.29.2.el6.x86_64-3.4.1-1.x86_64.rpm <path to extracted MPSS*-linux.tar>

15. Go to MPSS directory and install all the RPMs:
# cd <path to extracted MPSS*-linux.tar>
# yum instasll *.rpm

16. Load mic.ko and initialize MPSS default settings:
# modprobe mic
# micctrl --initdefaults

17. Reboot the system.

18. Dowload OFED stack OFED-3.5-2-MIC.tgz

19. Untar the OFED stack:
# tar xvf OFED-3.5-2-MIC.tgz

20. Run the install script to install
# ./install.pl

21. At this point, you should start MPSS and OFED normally.

Hope this helps. Thank you.

Taras_Shapovalov · ‎11-24-2014

Hi loc-nguyen,

Thank you for your help. I have reinstalled all from scratch and it is working also fine now. Lets assume there is no issue with SL6.5.

Thanks,

Loc_N_Intel · ‎11-25-2014

You are welcome. It great to know that Scientific Linux 6.5 works with MPSS 3.4.1 . Thank you for sharing.