Swap on xeon phi - Page 2

Philip_S_ · ‎07-10-2014

I have a Xeon Phi card running MPSS Gold update 3.

I want to use swap but mic_virtblk will not load into the kernel

message is cannot find mic_virtblk.

Do I need to use MPSS 3.2.3 to use swap?

David_M_8 · ‎07-13-2015

Thank you JJK and Frances,

true the error message got me confused (Frances: <What modprobe is complaining about is not that it cannot find the kernel module but that it cannot find the device on the host that is providing the actual swap space. Without that space the module can't load.> )

the /var/log/messages seems to confirm this:

Jul 13 10:03:45 mic0 user.warn kernel: [ 7.021619] Module mic_virtblk loaded at 0xffffffffa007a000
Jul 13 10:03:45 mic0 user.err kernel: [ 7.029411] Have you set virtblk file?

But i still didn't get the module to load. I'm mostly trying to mount an ext2 partition now, not to swap. But i tried swap too and the point of failure is the same.

_________________________________________________

Here is my complete attempt to mount an ext2 partition sdb1 (on freshly formatted disk sdb) :

("phi" is my host, mic0 is one of the 8 coprocessors. RHEL7.1 , MPSS 3.5.1, Supermicro 4028 TRT server containing a PCIe splitter/expansion)

[root@phi ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 118.8G 0 part
├─rhel-swap 253:0 0 11.9G 0 lvm [SWAP]
├─rhel-root 253:1 0 50G 0 lvm /
└─rhel-home 253:2 0 56.8G 0 lvm /home
sdb 8:16 0 59.6G 0 disk
[root@phi ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): N
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-125045423, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-125045423, default 125045423):
Using default value 125045423
Partition 1 of type Linux and of size 59.6 GiB is set

Command (m for help): p

Disk /dev/sdb: 64.0 GB, 64023257088 bytes, 125045424 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xa054008f

Device Boot Start End Blocks Id System
/dev/sdb1 2048 125045423 62521688 83 Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@phi ~]# mkfs.ext2 /dev/sdb1
mke2fs 1.42.9 (28-Dec-2013)
/dev/sdb1 is mounted; will not make a filesystem here!
[root@phi ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 118.8G 0 part
├─rhel-swap 253:0 0 11.9G 0 lvm [SWAP]
├─rhel-root 253:1 0 50G 0 lvm /
└─rhel-home 253:2 0 56.8G 0 lvm /home
sdb 8:16 0 59.6G 0 disk
└─sdb1 8:17 0 59.6G 0 part /mnt/sdb1
[root@phi ~]# umount /dev/sdb1
[root@phi ~]# mkfs.ext2 /dev/sdb1
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: done
warning: 86 blocks unused.

Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
3915216 inodes, 15630336 blocks
781516 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
477 block groups
32768 blocks per group, 32768 fragments per group
8208 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done

[root@phi ~]# mount /dev/sdb1 /mnt/sdb1
[root@phi ~]# mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=32839680k,nr_inodes=8209920,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/rhel-root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=35,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
/dev/sda1 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/rhel-home on /home type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/sdb1 on /mnt/sdb1 type ext2 (rw,relatime,seclabel)
[root@phi ~]# service mpss unload
Shutting down Intel(R) MPSS: [ OK ]
Removing MIC Module: [ OK ]
[root@phi ~]# modprobe mic
[root@phi ~]# echo /dev/sdb1 > /sys/class/mic/mic0/virtblk_file
[root@phi ~]# cat /sys/class/mic/mic0/virtblk_file
/dev/sdb1
[root@phi ~]# service mpss start
Starting mpss (via systemctl): [ OK ]
[root@phi ~]# ssh mic0
Password:
[root@mic0 ~]# mkdir /mnt/vda
[root@mic0 ~]# mount /dev/vda /mnt/vda
mount: mounting /dev/vda on /mnt/vda failed: No such file or directory
[root@mic0 ~]#

from /var/log/messages:

Jul 13 10:03:45 mic0 user.warn kernel: [ 7.021619] Module mic_virtblk loaded at 0xffffffffa007a000
Jul 13 10:03:45 mic0 user.err kernel: [ 7.029411] Have you set virtblk file?

______________________________________________________________________________

I have also noticed that /sys/class/mic/mic0 redirects me to a directory "/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/0000:03:08.0/0000:04:00.0/mic/mic0"

It does however contain the virtblk_file with the line i echoed in, before mpss start and after start too.

JJK · ‎07-14-2015

I am not sure that you can use a block device /dev/sdb1 as a value for virtblk_file ; I've created a file on disk, formatted it as an EXT2 filesystem and mounted it. That works fine with mpss 3.5.1:

# service mpss stop
# dd if=/dev/zero of=/var/mpss/mic0.virtblk_file bs=1024k count=10000
# echo /var/mpss/mic0.virtblk_file > /sys/class/mic/mic0/virtblk_file
# service mpss start

Then, on the Phi, I can format /dev/vda, mount it, etc.

And, on my box, /sys/class/mic/mic0 is a symlink to /sys/devices/pci0000:80/0000:80:03.0/0000:87:00.0/0000:88:10.0/0000:8a:00.0/mic/mic0 , so the value you are seeing does not surprise me.