Virtualization with KVM on the PinePhone Pro

June 23, 2023

PinePhone, Mobian, OpenBSD, KVM, BSD, Linux

Basic Setup #

All the tools we need for running VM are already packaged on Mobian, to install them, run:

sudo apt install virt-manager

then add your user to the libvirt group:

sudo adduser mobian libvirt

Reboot and then run virt-host-validate, it should indicate /dev/kvm exists and is accessible.

Trouble with Heterogeneous Architecture #

Trying to start qemu-system-aarch64 with -enable-kvm flag can yield the following, rather unhelpfully worded error:

qemu-system-aarch64: kvm_init_vcpu: kvm_arch_init_vcpu failed (0): Invalid argument

Turns out the RK3399s SoC used on this device is built around Arm’s heterogeneous big.Little architecture, and contains 4 slower Cortex A53 cores and 2 faster Cortex A72 cores, this allows the kernel to dynamically schedule tasks on different types of cores to improve performance and save energy. However, this configuration is not yet supported by KVM, and when the expected CPU type differs from the scheduled type (e.g. expecting A72 but kernel scheduled a process on an A53 core), it will panic.

Before KVM is able to work with this setup, we can workaround it by manually set the CPU affinity of qemu by launching it with taskset. To only use A72 cores:

taskset -c 4,5 qemu-system-aarch64 <qemu options>

To only use the slower A53 cores:

taskset -c 0,1,2,3 qemu-system-aarch64 <qemu options>

To apply this workaround globally, we need a wrapper.

dpkg-divert #

Simply replacing the qemu-system-aarch64 binary with a wrapper is not a great idea because upstream Debian package can override our warpper when upgrading qemu. To ensure Debian will not override it, we can divert package’s version of the binary to another location with dpkg-divert:

sudo dpkg-divert --rename /usr/bin/qemu-system-aarch64

The --rename option ensures the existing binary will be moved to a new name, which by default is qemu-system-aarch64.distrib. Finally, create the wrapper under usr/bin/qemu-system-aarch64 (I decide to only use faster cores, A53 cores are too slow for most workload):

#!/usr/bin/env sh
taskset -c 4,5 /usr/bin/qemu-system-aarch64.distrib "$@"

Launching VM #

The following scripts will launch VM of different BSD OS, doing the same for Linux distros is similar. I’m using [user networking (SLIRP)][1] as network backend which does not require root privileges. This backend has the drawback of lower performance compare to TAP or VDE, but still fast enough for me.

OpenBSD #

Setup

mkdir openbsd.vm
cd openbsd.vm
# create disk image
qemu-img create -f qcow2 openbsd.vm.qcow2 32G
# use arm64 uefi firmware from package qemu-efi-aarch64
cp /usr/share/AAVMF/AAVMF_CODE.fd ./

Boot to installer

Assume using miniroot73.img as installer, -smp is needed for installer to enable MP kernel.

qemu-system-aarch64 \
        -enable-kvm \
        -m 1024 \
        -cpu host -M virt \
        -nographic \
        -drive if=pflash,file=aavmf_code.fd,format=raw \
        -drive if=virtio,file=miniroot73.img,format=raw \
        -drive if=virtio,file=openbsd.vm.qcow2,format=qcow2 \
        -netdev user,id=obsd \
        -device virtio-net,netdev=obsd \
        -smp 2

Launch VM

qemu-system-aarch64 \
        -enable-kvm \
        -m 1024 \
        -cpu host -M virt \
        -nographic \
        -drive if=pflash,file=aavmf_code.fd,format=raw \
        -drive if=virtio,file=openbsd.vm.qcow2,format=qcow2 \
        -netdev user,id=obsd \
        -device virtio-net,netdev=obsd \
        -smp 2

NetBSD #

Setup

mkdir netbsd.vm
cd netbsd.vm
# use arm64 uefi firmware from package qemu-efi-aarch64
cp /usr/share/AAVMF/AAVMF_CODE.fd ./

Launch VM

NetBSD provides ready to boot image for arm64, the daily snapshot is available at:

https://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/latest/evbarm-aarch64/binary/gzimg/arm64mbr.img.gz

qemu-system-aarch64 \
        -enable-kvm \
        -m 1024 \
        -cpu host -M virt \
        -nographic \
        -drive if=pflash,file=aavmf_code.fd,format=raw \
        -drive if=virtio,file=arm64mbr.img,format=raw \
        -netdev user,id=nbsd \
        -device virtio-net,netdev=nbsd \
        -smp 2

virt-manager and arm64 UEFI secure boot #

Virt-manager seems to use secure boot enabled firmware by default when creating new VM, this might not work for your prefered system (It certainly does not work with OpenBSD) and will yield a Script Error Status: Access Denied error for unsupported install media. To disable secure boot, select Customize configuration before install during the last step of creating new VM, go to Overview section, and change the firmware from AAVMF_CODE.ms.fd to UEFI aarch64: /usr/share/AAVMF/AAVMF_CODE.fd. This cannot be changed easily after VM is created.