topic CPU: Intel(R) Xeon(R) CPU E5 in Analyzers

KVM guest OS crash with DPDK workload

Dang_Tran__Frederic — Thu, 15 Nov 2018 16:24:02 GMT

Hi,

I'm trying to use VTune Amplifier to profile an OVS-DPDK virtual switch that runs on a target Linux host along with a KVM guest running a DPDK L2 forwarding application (testpmd). Each time I start a Memory Access or Micro-architecture profiling session, the KVM guest OS crashes with a kernel panic: BUG: unable to handle kernel paging request at xxxx.
My setup is as follows:
-VTune Amplifier 2019 (build 570779)
-sampling drivers on target Linux host: sepdk_v5_575421
-target Linux OS: Ubuntu 16.04 64bit, kernel version: 4.15.0-39-generic
-guest Linux OS: Ubuntu 18.04 64bit, kernel version 4.15.0-34-generic
-OVS 2.8.1
-QEMU emulator version 2.9.1
-24-core server, all cores except core #0 are isolated
-VM backed by 1G huge pages, two vNICs connected to vhost-user ports.

It is worth noting that
-the crash occurs whatever process is profiled by VTune
-VTune works fine when profiling the DPDK application running on bare-metal (no OVS switch or VM)

Any idea/suggestion ?

Hi Frederic,

Roman_K_Intel — Fri, 16 Nov 2018 15:38:50 GMT

Hi Frederic,

Are you running Memory Access / Microarchitecture exploration analysis inside KVM Guest or on the host OS? Are you using "Profile System" or "Attach to Process" mode?

Hi Ronan:

Dang_Tran__Frederic — Fri, 16 Nov 2018 16:50:11 GMT

Hi Ronan:

I'm running Memory Access / Microarchitecture exploration analysis on the host OS and I use the "Attach to process" mode (pid of ovs-vswitchd).

What is the CPU architecture?

Roman_K_Intel — Fri, 16 Nov 2018 17:20:55 GMT

What is the CPU architecture?

You could try to switch to perf (with root):

# cd /opt/intel/vtune_amplifier_2019/sepdk/src/
# ./rmmod-sep
# echo 0 > /proc/sys/kernel/perf_event_paranoid

Then try to run Microarchitecture Exploration analysis. Does this crash reproduce?

CPU: Intel(R) Xeon(R) CPU E5

Dang_Tran__Frederic — Fri, 16 Nov 2018 18:20:46 GMT

CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz

Driverless mode works OK without crash. Thanks !

Hi Frederic,

Roman_K_Intel — Wed, 21 Nov 2018 09:01:00 GMT

Hi Frederic,

I hope driverless mode unblocks your workflow. We'll try to reproduce the crash on our side.

May I ask your opinion / feedback for two features which were introduced in VTune 2019 and may be relevant for your workload:

Profile KVM Kernel and User Space from the Host

This feature allows to run the collection on the Linux host and dive in to one KVM Guest. This may be helpful if you need to analyze Host and Guest activities simultaneously. Another use case is when you need to run analysis like Microarchitecture Exploration inside VM, but VM doesn't virtualize all the required counters.

Core Utilization in DPDK Apps

This feature allows you to analyse how DPDK Rx utilizes the CPU.

Thanks for the pointers Roman

Dang_Tran__Frederic — Thu, 22 Nov 2018 12:40:58 GMT

Thanks for the pointers Roman.
Currently I'm precisely in the process of testing your DPDK support to distinguish busy vs non-busy poll loops in DPDK applications along with the recent power management enhancements in DPDK.