June 23, 2023
Basic Setup #
All the tools we need for running VM are already packaged on Mobian, to install them, run:
sudo apt install virt-manager
then add your user to the libvirt group:
sudo adduser mobian libvirt
Reboot and then run
virt-host-validate, it should indicate /dev/kvm exists and is accessible.
Trouble with Heterogeneous Architecture #
Trying to start
-enable-kvm flag can yield the following, rather unhelpfully worded error:
qemu-system-aarch64: kvm_init_vcpu: kvm_arch_init_vcpu failed (0): Invalid argument
Turns out the RK3399s SoC used on this device is built around Arm’s heterogeneous big.Little architecture, and contains 4 slower Cortex A53 cores and 2 faster Cortex A72 cores, this allows the kernel to dynamically schedule tasks on different types of cores to improve performance and save energy. However, this configuration is not yet supported by KVM, and when the expected CPU type differs from the scheduled type (e.g. expecting A72 but kernel scheduled a process on an A53 core), it will panic.
Before KVM is able to work with this setup, we can workaround it by manually set the CPU affinity of qemu by launching it with taskset. To only use A72 cores:
taskset -c 4,5 qemu-system-aarch64 <qemu options>
To only use the slower A53 cores:
taskset -c 0,1,2,3 qemu-system-aarch64 <qemu options>
To apply this workaround globally, we need a wrapper.
Simply replacing the qemu-system-aarch64 binary with a wrapper is not a great idea because upstream Debian package can override our warpper when upgrading qemu. To ensure Debian will not override it, we can divert package’s version of the binary to another location with
sudo dpkg-divert --rename /usr/bin/qemu-system-aarch64
--rename option ensures the existing binary will be moved to a new name, which by default is
qemu-system-aarch64.distrib. Finally, create the wrapper under
usr/bin/qemu-system-aarch64 (I decide to only use faster cores, A53 cores are too slow for most workload):
#!/usr/bin/env sh taskset -c 4,5 /usr/bin/qemu-system-aarch64.distrib "$@"
Launching VM #
The following scripts will launch VM of different BSD OS, doing the same for Linux distros is similar. I’m using [user networking (SLIRP)] as network backend which does not require root privileges. This backend has the drawback of lower performance compare to TAP or VDE, but still fast enough for me.
mkdir openbsd.vm cd openbsd.vm # create disk image qemu-img create -f qcow2 openbsd.vm.qcow2 32G # use arm64 uefi firmware from package qemu-efi-aarch64 cp /usr/share/AAVMF/AAVMF_CODE.fd ./
Boot to installer
miniroot73.img as installer,
-smp is needed for installer to enable MP kernel.
qemu-system-aarch64 \ -enable-kvm \ -m 1024 \ -cpu host -M virt \ -nographic \ -drive if=pflash,file=aavmf_code.fd,format=raw \ -drive if=virtio,file=miniroot73.img,format=raw \ -drive if=virtio,file=openbsd.vm.qcow2,format=qcow2 \ -netdev user,id=obsd \ -device virtio-net,netdev=obsd \ -smp 2
qemu-system-aarch64 \ -enable-kvm \ -m 1024 \ -cpu host -M virt \ -nographic \ -drive if=pflash,file=aavmf_code.fd,format=raw \ -drive if=virtio,file=openbsd.vm.qcow2,format=qcow2 \ -netdev user,id=obsd \ -device virtio-net,netdev=obsd \ -smp 2
mkdir netbsd.vm cd netbsd.vm # use arm64 uefi firmware from package qemu-efi-aarch64 cp /usr/share/AAVMF/AAVMF_CODE.fd ./
NetBSD provides ready to boot image for arm64, the daily snapshot is available at:
qemu-system-aarch64 \ -enable-kvm \ -m 1024 \ -cpu host -M virt \ -nographic \ -drive if=pflash,file=aavmf_code.fd,format=raw \ -drive if=virtio,file=arm64mbr.img,format=raw \ -netdev user,id=nbsd \ -device virtio-net,netdev=nbsd \ -smp 2
virt-manager and arm64 UEFI secure boot #
Virt-manager seems to use secure boot enabled firmware by default when creating new VM, this might not work for your prefered system (It certainly does not work with OpenBSD) and will yield a
Script Error Status: Access Denied error for unsupported install media. To disable secure boot, select
Customize configuration before install during the last step of creating new VM, go to
Overview section, and change the firmware from
UEFI aarch64: /usr/share/AAVMF/AAVMF_CODE.fd. This cannot be changed easily after VM is created.