Chrooting into a icelake machine from alder/raptor lake using intel's SDE.

[dEtechno]

I'm trying to chroot into an x86-64 (the whole OS is built using gcc -march=icelake-client (gentoo based)) ROOTfs from another x86-64 system who's CPU has incompatible instructions to icelake-client (like alderlake). The incompatibility is avx512. I'm trying to use Intel's SDE for the purpose.

However I'm not able to chroot using it to the this ROOT. I copied over the SDE (which is a directory) to ROOT and --

chroot /usr/x86_64-mypl-linux-gnu/ /sde64 -- /bin/bash

However I'm getting --

traps: sde64[229146] trap invalid opcode ip:7f56db09475d sp:7ffd55266fa8 error:0 in ld-linux-x86-64.so.2[1e75d,7f56db077000+2c000]

So the ROOT's ld-linux-x86-64.so.2 is being used to execute sde64; also sde64 links against libc. So what I did is copied the host's libc.so.6 and ld-linux-x86-64.so.2 to /alder/ of ROOT and --

chroot /usr/x86_64-mypl-linux-gnu/ /alder/ld-linux-x86-64.so.2 --library-path /alder/ ./sde64 -icl -- /bin/bash

which AGAIN results in

Illegal instruction (core dumped)

traps: pinbin[229213] trap invalid opcode ip:7f92d3d3775d sp:7fffca461648 error:0 in ld-linux-x86-64.so.2[1e75d,7f92d3d1a000+2c000]

Interestingly the sde64 binary itself works. For e.g --help works.

I tried the other way round by executing chroot using sde64 --

./sde64 -icl -- /usr/bin/chroot /usr/x86_64-mypl-linux-gnu/

Here it sde64 executable exits and chroot just hangs. It has to be SIGKILLed.

When /usr/x86_64-mypl-linux-gnu/bin/bash is directly executed using ./sde64 is works.

[AdyT_Intel]

This is an interesting idea that I've never tried to enable.

Intel SDE is a pin tool where pin is our binary instrumentation system. Therefore, all the work of pin is done natively (i.e. without SDE emulation) which means that pin and SDE itself, are not expected to run under the chroot.

The common practice it this sense is to run bash under SDE like in:

% sde64 -icl -- /bin/bash

Inside this bash you might execute chroot 'new-root' command. I'm not certain that this will work, but if you are interested in chroot to new runtime, then this direction may work.

[dEtechno]

Yes, I did try that, but chroot is still hanging, until I kill it using a SIGKILL.

I tried strace but --

E: Attach to pid 233843 failed: Operation not permitted

This happens just as I chroot. The last strace is attached. Without sde64 (chrooted into another machine), it works.

the 'yama' feature is not even enabled (/proc/sys/kernel/yama/ doesn't even exist).

Another symptom is that when I su I get --

su: cannot set groups: Operation not permitted

So to get to the root prompt, I have to run sde as root.

[AdyT_Intel]

I consulted with the pin team regarding this issue.

They said that you need to mount /dev and /proc inside your new root and copy the kit also to a location inside the new root.

Recommended method for mounting /proc & /dev

mount -t proc /proc ttt/proc/

mount --rbind /dev ttt/dev

 They recommended running SDE as follows:

% <newroot>/<path-to-kit>/sde64 <args> -- chroot <newroot> app

[dEtechno]

Yes, those VFSs are mounted.

The chroot setup is fine since I can chroot in the machine when the relevant packages (bash, ncurses, glibc etc...) where compiled with generic instructions).

Finally I was able to chroot successfully by copying the SDE to / of the chroot dir (which is /usr/x86_64-mypl-linux-gnu/ in the host machine) AND also copying it to /usr/x86_64-mypl-linux-gnu/ relative to the chroot dir or (/usr/x86_64-mypl-linux-gnu/usr/x86_64-mypl-linux-gnu/ relative to the host machine).

However other issues follow... will create a new post for that.