vmlinux checksum mismatch, but correct kernel-debuginfo package is installed

rbeasley · ‎07-22-2020

Hi all,

Classic question but for which I can't find a good answer.

Problem:

VTune is unable to resolve kernel symbols, so my only visibility of the kernel is a single function named [vmlinux].
Diagnostic message from the Collection Log tab:
"Cannot locate file `vmlinux'. `/usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux' has unexpected checksum. The binary file may have been recompiled."

Environment:

CentOS Linux release 8.2.2004 (Core)
Intel VTune Profiler 2020.1.0.607630.

I installed the kernel-debuginfo package corresponding to my kernel:

$ rpm -q kernel{,-debuginfo}
kernel-4.18.0-193.6.3.el8_2.x86_64
kernel-debuginfo-4.18.0-193.6.3.el8_2.x86_64

And, using the "Binary/Symbol Search" dialog, I pointed VTune to the root of the directory owned by the kernel-debuginfo package:
```
$ rpm -ql kernel-debuginfo | grep vmlinux
/usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux
```

I can confirm that the two line up:

# I extracted /boot/vmlinuz-4.18.0-193.6.3.el8_2.x86_64 to vmlinux.
# See https://superuser.com/questions/298826/how-do-i-uncompress-vmlinuz-to-vmlinux
$ readelf -n vmlinux | grep 'Build ID'                                                    
    Build ID: 1f29b9f85e204c62d7d6f7ed0d72615a3bbed5ef
$ readelf -n /usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux  | grep 'Build ID'
    Build ID: 1f29b9f85e204c62d7d6f7ed0d72615a3bbed5ef

That vmlinux should correspond to my running kernel. The embedded build IDs are a match. However, VTune indicates a checksum mismatch. Could Intel engineers please shed some light on what checksum is being used and how it's calculated?

Thanks!

Kirill_U_Intel · ‎07-22-2020

Hi.

In general you don't need to extract vmlinuz yourself. VTune produces that automatically.

Could you provide output from our tool?

INSTALL_VTUNE_DIR/bin64/amplxe-runss --find-bin-file vmlinux --kernel-version `uname -r`

and

INSTALL_VTUNE_DIR/bin64/amplxe-runss --md5sum vmlinux --kernel-version `uname -r`

Kirill

rbeasley · ‎07-23-2020

Hi Kirill,

Thanks for following up!

$ /opt/intel/vtune_profiler/bin64/amplxe-runss --find-bin-file vmlinux --kernel-version $(uname -r)
path: /tmp/amplxe-tmp-beasleyr/vmlinux_cache/3cb5a4661a70c21eaafe2aab7e6b5e24/vmlinux
status: SUCCESS
$ /opt/intel/vtune_profiler/bin64/amplxe-runss --md5sum vmlinux --kernel-version $(uname -r)
path: /tmp/amplxe-tmp-beasleyr/vmlinux_cache/3cb5a4661a70c21eaafe2aab7e6b5e24/vmlinux
md5sum: 0b8d8906f4528be5f76abaa39a195167

I dug through /tmp/amplxe-log-beasleyr to find logs from the symbol resolver. Maybe the following would also help?

160513 [140435474503424] WARN dicerengine <> - `/usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux' has unexpected checksum. The binary file may have been recompiled., at file: vcs/dicerengine2/src/core/file_finder_locator.cpp:257                                                                                                                                                                                                                                    160583 [140435474503424] WARN dicerengine <> - `/boot/vmlinuz-4.18.0-193.6.3.el8_2.x86_64' has unexpected checksum. The binary file may have been recompiled., at file: vcs/dicerengine2/src/core/file_finder_locator.cpp:257              160583 [140435474503424] WARN dicerengine <> - cannot locate file vmlinux, at file: vcs/dicerengine2/src/core/file_finder_locator.cpp:597

If there are any additional details I can provide, please let me know.

Thanks again!

Kirill_U_Intel · ‎07-24-2020

Could you remove/hide vmlinux that was extracted by you from vmlinuz?

It has different checksum with VTune extracted.

Try restart the collection and check resolving.

Kirill

rbeasley · ‎07-24-2020

Well, there's a problem with that request: the vmlinux I extracted was in a separate temporary directory. I extracted it only to look up the build ID ELF tag and make sure it matched the one provided by the kernel-debuginfo package.

In any event, I deleted the temp directories and purged VTune's symbol search directories. Running a fresh collection, I'm back to square one, where VTune has absolutely no visibility into the kernel:

Cannot locate file `vmlinux'.

When I then use the Binary/Symbol Search dialog to point to /usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64, VTune reports the original error:

Cannot locate file `vmlinux'. `/usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux' has unexpected checksum. The binary file may have been recompiled.

To reiterate, this vmlinux is provided by the kernel-debuginfo package:

$ rpm -qf /usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux 
kernel-debuginfo-4.18.0-193.6.3.el8_2.x86_64

The MD5 of /tmp/amplxe-tmp-beasleyr/vmlinux_cache/3cb5a4661a70c21eaafe2aab7e6b5e24/vmlinux doesn't match /usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux, presumably because the running kernel (as provided by the distribution) is stripped. (/boot/vmlinux-4.18.0-193.6.3.el8_2.x86_64 is only 8.6M. The debuginfo vmlinux is 729M.)

Is VTune able to load kernel symbols from a separate symbol file?

Please lmk if there are any add'l details I can collect. Thanks!

Denis_M_Intel · ‎07-27-2020

Hi,

Yes, VTune is able to load symbols from a separate symbol file.

I can't reproduce your issue but we may try following:
- remove /tmp/amplxe-tmp-beasleyr/vmlinux_cache directory
- copy vmlinux as /boot/vmlinux-<kernel version> and make sure it is accessible;
- re-finalize a VTune result: vtune -finalize -r <vtune_result_dir>

The kernel-debuginfo is manly needed to have code - source line mapping. Normally VTune uses /boot/System.map-<kernel_version> or /proc/kallsyms tables to resolve symbols for the Linux kernel. Make sure System.map is readable or kallsyms has non-zero addresses (kptr_restrict value should be 0).

rbeasley · ‎07-27-2020

Hi Denis,

Thanks for following up!

I have two quick questions, if you wouldn't mind:

> I can't reproduce your issue

Are you attempting to reproduce using RHEL/CentOS 8 or some other distribution? I'd hope it wouldn't matter, but perhaps there are some quirks about how the kernel-core and kernel-debuginfo packages are provided by this distribution which are confusing VTune.

> - copy vmlinux as /boot/vmlinux-<kernel version> and make sure it is accessible;

Is vmlinux the copy from the kernel-debuginfo package, or the copy manually extracted from vmlinuz? I've tried both with different but still (distinct) failing outcomes.

If I copy vmlinux from the kernel-debuginfo package, I see a "Cannot load symbols from `'. Make sure the file format is valid." message:

# Copying vmlinux from the kernel-debuginfo package.
$ pwd
/boot
$ sudo cp /usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux vmlinux-4.18.0-193.6.3.el8_2.x86_64
$ ls -al *-$(uname -r)*         
-rw-r--r--. 1 root root    187643 Jun 10 07:17 config-4.18.0-193.6.3.el8_2.x86_64                                    
-rw-------. 1 root root  30398711 Jun 26 11:45 initramfs-4.18.0-193.6.3.el8_2.x86_64.img                             
-rw-------. 1 root root  20660440 Jul 20 14:48 initramfs-4.18.0-193.6.3.el8_2.x86_64kdump.img                        
-rw-r--r--. 1 root root   3910484 Jun 10 07:17 System.map-4.18.0-193.6.3.el8_2.x86_64                                
-rwxr-xr-x. 1 root root 764207696 Jul 27 10:29 vmlinux-4.18.0-193.6.3.el8_2.x86_64                                   
-rwxr-xr-x. 1 root root   8913656 Jun 10 07:17 vmlinuz-4.18.0-193.6.3.el8_2.x86_64
$ vtune -finalize -r r1595625324.3838192
...
vtune: Warning: Cannot locate file `vmlinux'.                                                                        
vtune: Executing actions 45 % Resolving information for `vmlinux'                                                                                                                                                                          vtune: Warning: Cannot load symbols from `'. Make sure the file format is valid.                                     
vtune: Executing actions 100 % done

If I extract vmlinux from vmlinuz-$(uname -r), then while I no longer see any errors about mismatches, VTune then reports that it's unable to locate debugging information.

# Placing vmlinux from the manually extracted file.
$ pwd
/boot
$ sudo cp /work/symbols-flat/vmlinux vmlinux-4.18.0-193.6.3.el8_2.x86_64
$ ls -al *-$(uname -r)*
-rw-r--r--. 1 root root   187643 Jun 10 07:17 config-4.18.0-193.6.3.el8_2.x86_64
-rw-------. 1 root root 30398711 Jun 26 11:45 initramfs-4.18.0-193.6.3.el8_2.x86_64.img
-rw-------. 1 root root 20660440 Jul 20 14:48 initramfs-4.18.0-193.6.3.el8_2.x86_64kdump.img
-rw-r--r--. 1 root root  3910484 Jun 10 07:17 System.map-4.18.0-193.6.3.el8_2.x86_64
-rw-r--r--. 1 root root 47077768 Jul 27 10:38 vmlinux-4.18.0-193.6.3.el8_2.x86_64
-rwxr-xr-x. 1 root root  8913656 Jun 10 07:17 vmlinuz-4.18.0-193.6.3.el8_2.x86_64
$ vtune -finalize -r r1595625324.3838192
...
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
vtune: Executing actions 100 % done

In both cases, System.map-$(uname -r) is present and readable (see `ls` output above), and /proc/kallsyms is also accessible.

$ head /proc/kallsyms
0000000000000000 A irq_stack_union
0000000000000000 A __per_cpu_start
0000000000004000 A cpu_debug_store
0000000000005000 A cpu_tss_rw
0000000000008000 A gdt_page
...

And, finally, in the case of the manually extracted vmlinux file, I've tried passing --search-dir=/usr/lib/debug/lib/modules/$(uname -r) to the finalize command line, but this brings me back to the original "unexpected checksum" error.

I've run `rpm -V kernel-core kernel-debuginfo` to ensure that I haven't somehow corrupted them. (The only reported difference is the mode change to /boot/System.map-$(uname -r), since its default permissions are 0600.)

Is there any way I can cheat and force VTune to trust me and not verify checksums?

Attached are some log file bundles recorded when running `vtune -finalize`in a variety of scenarios. (See README.md at the toplevel for a summary.) Hope this helps!

Denis_M_Intel · ‎07-28-2020

* I tried VTune uarch-exploration driverless collection (perf-based) on CentOS 7.5 with following system settings:
/proc/sys/kernel/kptr_restrict = 0
/proc/sys/kernel/perf_event_paranoid = 0
I had all vmlinux functions resolved.

* I installed the kernel debug package, collected a new result with ‘-search-dir /usr/lib/debug/lib/modules/3.10.0-862.el7.x86_64’ option (it is needed to force using the debug version of vmlinux from /usr/lib/debug/lib/modules/3.10.0-862.el7.x86_64/vmlinux). VTune resolved all vmlinux functions and their source files.

The “Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions” message is printed when VTune fails to find debug version of vmlinux. It uses System.map or /proc/kallsyms to resolve function names in this case.
Do you see vmlinux function names in VTune reports if this message is printed?

The message “`/usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux' has unexpected checksum. The binary file may have been recompiled." is not expected. Can you share following?
- any VTune result with this message
- System.map-<kernel_version>
- `/usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux' binary
- vmlinuz-<kernel_version> binary

As for the issue with fuse from README.md, VTune found debug info.

constant_test_bit function (constant_test_bit: fuse ! constant_test_bit - bitops.h) has source file but it looks like source line information for other functions is missed in debug info ([unknown source file]). Can you check that other functions of fuse have source line information (you can use objdump or addr2line utilities on fuse)?

rbeasley · ‎07-30-2020

Thank you again for following up!

* I tried VTune uarch-exploration driverless collection (perf-based) on CentOS 7.5 with following system settings:
/proc/sys/kernel/kptr_restrict = 0
/proc/sys/kernel/perf_event_paranoid = 0
I had all vmlinux functions resolved.

For whatever it's worth, I'm using hardware event-based driver based collection.

$ cat /proc/sys/kernel/kptr_restrict
0
# Might not matter since I'm using hw drivers, not perf.
$ cat /proc/sys/kernel/perf_event_paranoid
-1

* I installed the kernel debug package, collected a new result with ‘-search-dir /usr/lib/debug/lib/modules/3.10.0-862.el7.x86_64’ option (it is needed to force using the debug version of vmlinux from /usr/lib/debug/lib/modules/3.10.0-862.el7.x86_64/vmlinux). VTune resolved all vmlinux functions and their source files.

I must say, I'm jealous. In my best case, I can get basic symbols, but no line numbers. (More on this below.)

The “Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions” message is printed when VTune fails to find debug version of vmlinux. It uses System.map or /proc/kallsyms to resolve function names in this case.
Do you see vmlinux function names in VTune reports if this message is printed?

Yes, I see function names in this case. See case 04-manual_vmlinux_search_modules_xzcat_debuginfo from vtune-logs.zip for an example. In this case, I'll see entries like vmlinux!_raw_spin_lock - [unknown source file] for all kernel (vmlinux and module) frames.

The message “`/usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux' has unexpected checksum. The binary file may have been recompiled." is not expected. Can you share following?
- any VTune result with this message

By "result", are you asking for a copy of the directory containing log, config, data.0, sqlite-db, etc.? Or are you asking for something else? My current results are on the order of a few hundred megabytes, but I can make a new, smaller recording. Please let me know, and I'll follow up asap.

- System.map-<kernel_version>
- `/usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux' binary
- vmlinuz-<kernel_version> binary

These are the same as provided by the following RPMs:

I downloaded and extracted these packages locally, and the sha256sums of the named files match those from my system:

$ cd $(mktemp -d)

# kernel-core
$ wget http://mirror.centos.org/centos/8/BaseOS/x86_64/os/Packages/kernel-core-4.18.0-193.6.3.el8_2.x86_64.rpm
$ rpm2cpio kernel-core-$(uname -r).rpm | cpio -i --make-directories

$ sha256sum /boot/vmlinuz-$(uname -r) ./lib/modules/$(uname -r)/vmlinuz
5a1494eaec137a424a4af01984f6040601bceb22070e36e4822117782475c54b  /boot/vmlinuz-4.18.0-193.6.3.el8_2.x86_64
5a1494eaec137a424a4af01984f6040601bceb22070e36e4822117782475c54b  ./lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinuz

$ sha256sum /boot/System.map-$(uname -r) ./lib/modules/$(uname -r)/System.map
fe8b2665298eab5ad32c06ad1e05e3e528b8683baecadff32ee6b850410e79b9  /boot/System.map-4.18.0-193.6.3.el8_2.x86_64
fe8b2665298eab5ad32c06ad1e05e3e528b8683baecadff32ee6b850410e79b9  ./lib/modules/4.18.0-193.6.3.el8_2.x86_64/System.map

# kernel-debuginfo
$ wget http://debuginfo.centos.org/8/x86_64/Packages/kernel-debuginfo-4.18.0-193.6.3.el8_2.x86_64.rpm
$ rpm2cpio kernel-debuginfo-$(uname -r).rpm | cpio -i --make-directories

$ sha256sum /usr/lib/debug/lib/modules/$(uname -r)/vmlinux  ./usr/lib/debug/lib/modules/$(uname -r)/vmlinux
82494e5b4667822e9697d11af25cc1d5c39240e172818c3218c84f8da36690f2  /usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux
82494e5b4667822e9697d11af25cc1d5c39240e172818c3218c84f8da36690f2  ./usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux

As for the issue with fuse from README.md, VTune found debug info.

constant_test_bit function (constant_test_bit: fuse ! constant_test_bit - bitops.h) has source file but it looks like source line information for other functions is missed in debug info ([unknown source file]). Can you check that other functions of fuse have source line information (you can use objdump or addr2line utilities on fuse)?

Addr2line is able to display line number information. For example, when I open the result corresponding to my findings in vtune-logs.zip, VTune shows the following frames:

CPU Time
1 of 15: 80.3% (5.331s of 6.637s)

vmlinux ! __wake_up_common_lock - [unknown source file]
fuse ! constant_test_bit - bitops.h
fuse ! fuse_request_end + 0x9a - [unknown source file]
fuse ! fuse_dev_do_write + 0x24f - [unknown source file]
fuse ! fuse_dev_write + 0x53 - [unknown source file]
libc-2.28.so ! __GI___writev + 0x4f - writev.c:24
sandboxfs ! _$LT$fuse..channel..ChannelSender$u20$as$u20$fuse..reply..ReplySender$GT$::send::h63e4cdc52fd6f60a + 0xf6 - channel.rs
sandboxfs ! fuse::reply::ReplyRaw$LT$T$GT$::send::h3e1982db5045d8a6 + 0x1a4 - reply.rs:594
sandboxfs ! fuse::reply::ReplyEmpty::error::ha95418959362e094 + 0x26 - reply.rs
sandboxfs ! _$LT$sandboxfs..SandboxFS$u20$as$u20$fuse..Filesystem$GT$::lookup::h79410ca202f62d3f + 0x67 - lib.rs:591
sandboxfs ! fuse::request::Request::dispatch::h1ba7168e9d014ac9.llvm.10366026729934250672 + 0x20f4 - request.rs:150
sandboxfs ! fuse::session::Session$LT$FS$GT$::run::h58e6380df9a07e05 + 0xf5 - session.rs:34
sandboxfs ! sandboxfs::mount::hb0f9fd733fb6d3ec + 0x160b - lib.rs:1548

If I click fuse_request_end to open the disassembly in a new fuse.ko tab, I see the following:

0xeaa   Block 13:
0xeaa   > movq  0x30(%rbx), %rax
0xeae   test $0x8, %ah

So, then heading over to a terminal I'm able to generate the following:

$ addr2line -e /usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/kernel/fs/fuse/fuse.ko.debug -f 0xeaa 0xeae
constant_test_bit
/usr/src/debug/kernel-4.18.0-193.6.3.el8_2/linux-4.18.0-193.6.3.el8_2.x86_64/./arch/x86/include/asm/bitops.h:325
fuse_request_end
/usr/src/debug/kernel-4.18.0-193.6.3.el8_2/linux-4.18.0-193.6.3.el8_2.x86_64/fs/fuse/dev.c:325

$ ls -al /usr/src/debug/kernel-4.18.0-193.6.3.el8_2/linux-4.18.0-193.6.3.el8_2.x86_64/./arch/x86/include/asm/bitops.h /usr/src/debug/kernel-4.18.0-193.6.3.el8_2/linux-4.18.0-193.6.3.el8_2.x86_64/fs/fuse/dev.c
-rw-r--r--. 1 root root 14271 Jun  1 15:12 /usr/src/debug/kernel-4.18.0-193.6.3.el8_2/linux-4.18.0-193.6.3.el8_2.x86_64/./arch/x86/include/asm/bitops.h
-rw-r--r--. 1 root root 52860 Jun  1 15:12 /usr/src/debug/kernel-4.18.0-193.6.3.el8_2/linux-4.18.0-193.6.3.el8_2.x86_64/fs/fuse/dev.c

Denis_M_Intel · ‎08-03-2020

Thank you for the detailed information.

Yes, by "VTune result" I'm asking a copy of the directory with log, config, data.0, sqlite-db. A small result (collected for any application, e.g /bin/ls or matrix from VTune samples) which has vmlinux function with unresolved source lines is enough.

As for the 'fuse' case: do you have "/usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/kernel/fs/fuse" in the VTune binary search paths?
It would be great to have a reproducer for fuse as well: small result + fuse.ko + fuse.ko.debug.

rbeasley · ‎08-11-2020

I apologize for the delayed response. I'm attaching results from running the matrix:multiply1 sample program. The first tarball contains the raw collected results w/ -finalization-mode=deferred, and the second tarball contains the results after finalization in the VTune GUI. Like before, I'm able to get symbol information for the kernel, but no line information. Addr2line is able to resolve addresses to lines. (See below.) As for kernel modules, VTune reports it could not locate symbols for kvm.ko and vtspp.ko.

If I missed anything or can collect anything else, please let me know. Thanks!

$ addr2line -e /usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux 0xffffffff8113b651
/usr/src/debug/kernel-4.18.0-193.6.3.el8_2/linux-4.18.0-193.6.3.el8_2.x86_64/./include/linux/seqlock.h:114

Here's my command line, for reference:

$ . /opt/intel/vtune_profiler/vtune-vars.sh
$ vtune -collect hotspots -knob sampling-mode=hw \
    -knob enable-stack-collection=true \
    -knob stack-size=0 -quiet \
    -user-data-dir /work/vtune/profiles/sample_for_intel \
    -target-duration-type veryshort -return-app-exitcode \
    -no-follow-child -data-limit=4096 \ 
    -finalization-mode=deferred -- ./matrix

Denis_M_Intel · ‎08-14-2020

I've downloaded the results. Thank you.
The issue with vmlinux checksum mismatch is caused by "-finalization-mode=deferred" option. VTune saves checksums of all modules in this case and uses them to verify modules on the finalization step. That is why the debug version of vmlinux is not used (it has different checksum).
Why do you need the "-finalization-mode=deferred" option?
Can you try to collect VTune result without this option but add "-search-dir=/usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64" on finalization?

rbeasley · ‎08-14-2020

Aha! Good to know.

I'm using "-finalization-mode=deferred" because I can't readily simulate the workload I'm measuring. So instead I'm replacing the profiled program with a simple shell wrapper which invokes "vtune <args> -- ./myprogram <myprogramargs>". Since I didn't want the parent program to block for too long after my program terminated, I figured I could used deferred finalization and instead pay that cost when I actually opened the result in the VTune GUI.

I'll give this a shot when I have cycles in the coming days and will report back. Thanks!