Virtualizing Raspbian (or any ARM/Linux distro) headless using QEMU

For testing or development it can be very useful to have a distribution that usually runs on an embedded ARM board such as the Raspberry Pi run right on your machine (that isn't ARM) using a virtual machine.

QEMU provides excellent support for emulation of the ARM architecture (both 32 and 64-bit) and can emulate many different real ARM boards.

Why not use QEMU's "raspi2" machine for emulation?

QEMU comes with a raspi2 machine. It emulates the GPU's framebuffer, HWRNG, UART, GPIO and SD controller.

Spot something missing? It doesn't implement USB, which makes it useless for headless and graphical use as you can plug in neither a network connection nor a keyboard or mouse.

(Update 2021-01-08: QEMU has apparently added raspi USB support in versions 5.1.0 onward, so you could skip much of the setup detailed here if doing this from scratch.)

If you still want to use it, this guide will only help you halfway but here are the qemu parameters:

-M raspi2b -kernel kernel7l.img -dtb bcm2709-rpi-2-b.dtb -append "root=/dev/mmcblk0 rw console=ttyAMA0"

The plan

Instead of (poorly) emulating a real piece of hardware, QEMU also has a virt machine [1] that is designed for virtualization. It gives you a modern system with PCI and also works out-of-the-box with Linux without providing a Device Tree (QEMU generates one internally).

The most straightforward way of getting network and disk into such a VM is to use virtio-net and virtio-disk respectively, which is what we'll be doing.

Since virtio requires kernel support, chances are the Raspberry Pi kernel wouldn't work anyway, so we'll be using a different one.

I picked Arch Linux ARM's armv7 kernel from here, though any other should work just as well provided it comes with the appropriate modules. To load the virtio modules during boot we'll require an initramfs, but more on that later.

Extracting Raspbian's root filesystem into a virtual disk image

Start by downloading Raspbian from the Raspberry Pi website, then run the script below or follow the steps manually.

The script will create a copy of the image file, expand the image and its partition to 10 gigabytes, mount the partition using a loop device and make two adjustments:

  • Remove both SD card partitions from /etc/fstab, these don't exist inside the VM and we will be mounting the rootfs ourselves

  • Disable several startup services that do not work inside the VM

After unmounting the partition it will convert the filesystem into a qcow2 format image for use with QEMU.

#!/bin/bash -e
input=2020-02-13-raspbian-buster-lite.img
[ -f $input ]

mkdir mnt
cp --reflink=auto $input source.img
truncate -s 10G source.img
echo ", +" | sfdisk -N 2 source.img
dev=$(sudo losetup -fP --show source.img)
[ -n "$dev" ]
sudo resize2fs ${dev}p2
sudo mount ${dev}p2 ./mnt -o rw
sudo sed '/^PARTUUID/d' -i ./mnt/etc/fstab
sudo rm -f \
        ./mnt/etc/systemd/system/multi-user.target.wants/{hciuart,dphys-swapfile}.service \
        ./mnt/etc/rc?.d/?01{resize2fs_once,rng-tools,rng-tools-debian}
sudo umount ./mnt
sudo chmod a+r ${dev}p2
qemu-img convert -O qcow2 ${dev}p2 rootfs.qcow2
sudo losetup -d $dev
rm source.img; rmdir mnt

The kernel and initramfs

Kernel

Conveniently the linux-armv7 package is just a tar archive, so you can extract the kernel executable using:

tar -xvf linux-armv7*.pkg.tar.xz --strip-components=1 boot/zImage

Making an initramfs

Since virtio support is not compiled into the kernel and the root filesytem is missing modules for the exact kernel we'll be using (maybe copying them would've been easier?), we need to write an initramfs that can load these modules prior to mounting the rootfs.

Fortunately the Gentoo Wiki has a great article on making a custom one yourself. The basic idea is to extract the required kernel modules into the initramfs, whose init script loads the modules, mounts the root filesystem and actually boots.

The script shown below does the following steps:

  • Extract kernel modules from package

  • Delete some that we won't be needing and take a lot of space (optional)

  • Download and install a statically-linked busybox executable

  • Create the init script

  • Pack contents into a cpio archive as required by the Linux kernel

Using a virtio disk and network adapter requires loading the virtio-pci, virtio-blk, virtio-net modules. If you need any more the init script can easily be changed accordingly.

#!/bin/bash -e
pkg=$(echo linux-armv7-*.pkg.tar.xz)
[ -f "$pkg" ]

mkdir initrd; pushd initrd
mkdir bin dev mnt proc sys
tar -xaf "../$pkg" --strip-components=1 usr/lib/modules
rm -rf lib/modules/*/kernel/{sound,drivers/{net/{wireless,ethernet},media,gpu,iio,staging,scsi}}
wget https://www.busybox.net/downloads/binaries/1.31.0-defconfig-multiarch-musl/busybox-armv7l -O bin/busybox
cat >init <<"SCRIPT"
#!/bin/busybox sh
busybox mount -t proc none /proc
busybox mount -t sysfs none /sys
busybox mount -t devtmpfs none /dev

for mod in virtio-pci virtio-blk virtio-net; do
        busybox modprobe $mod
done

busybox mount -o rw /dev/vda /mnt || exit 1

busybox umount /proc
busybox umount /sys
busybox umount /dev

exec busybox switch_root /mnt /sbin/init
SCRIPT
chmod +x bin/busybox init
bsdtar --format newc --uid 0 --gid 0 -cf - -- * | gzip -9 >../initrd.gz
popd; rm -r initrd

Booting the virtual machine

With the initramfs built, we have all parts needed to actually run the VM: [2]

qemu-system-arm -M virt,highmem=off -m 2048 -smp 4 -kernel zImage -initrd initrd.gz \
        -drive file=rootfs.qcow2,if=virtio -nic user,model=virtio \
        -append "console=ttyAMA0" -nographic
After roughly a minute of booting you should be greeted by
Raspbian GNU/Linux 10 raspberrypi ttyAMA0 and a login prompt.

Further steps

This virtualization approach should work for just about any ARM/Linux distribution. I have tested it with Raspbian, Void Linux and Arch Linux ARM (whose rootfs even works without any modifications).

To ensure the kernel performs as expected beyond basic tasks, it's a good idea to extract the modules from the linux-armv7 package into the guest rootfs.

As with any VM, you can use the full extent of QEMU's features to e.g.:

  • attach an USB controller (-device usb-ehci or nec-usb-xhci)

  • ..SCSI controller (-device virtio-scsi)

  • ..Audio input/output (-device usb-audio)

  • or even enable graphical output (-device VGA)

AArch64?

With a few adjustments in the right places, this guide also works to emulate an AArch64 kernel and userland. I wrote this up in a newer article.


Opening a shell inside non-systemd nspawn containers

If you try to open shell inside a container that runs e.g. Alpine Linux using machinectl, the following not very descriptive error will appear:

# machinectl shell vpn
Failed to get shell PTY: Protocol error

The reason for this is simply that the container is not running systemd.

Because systemd-nspawn just uses Linux namespaces [1], nsenter can alternatively be used to access the container. For this, we'll need the PID of the init process inside the container:

# systemctl status systemd-nspawn@vpn
● systemd-nspawn@vpn.service - Container vpn
   Loaded: loaded (/lib/systemd/system/systemd-nspawn@.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-08-11 19:49:19 UTC; 6 months 3 days ago
 Main PID: 795 (systemd-nspawn)
   Status: "Container running."
   CGroup: /machine.slice/systemd-nspawn@vpn.service
           ├─payload
           │ ├─ 797 /sbin/init
           │ ├─1028 /sbin/syslogd -t
           [...]
In this case the PID of init is 797, you can then spawn a login shell inside the container:

nsenter -t 797 -a /bin/sh -l

Depending on the namespaces the container is (or isn't) joined to it can be necessary to specify the appropriate flags manually after consulting the nsenter manpage:

nsenter -t 797 -{m,u,i,n,p,U,C} -a /bin/sh -l

All in all, this can be turned into a nice alias for your .bashrc:

function center ()
{
  [ -z "$1" ] && { echo "Usage: center <name>" >&2; return 1; }
  pid=$(sed -n 2p "/sys/fs/cgroup/pids/machine.slice/systemd-nspawn@$1.service/tasks")
  [ -z "$pid" ] && { echo "Container not running" >&2; return 1; }
  nsenter -t $pid -a /bin/sh -l
}

QEMU Configuration & Usage

This will cover some QEMU options I have found useful beyond the basics.

Machine Type (x86)

-M q35 configures a more modern chipset to be emulated. The Q35 chipset supports PCI-e and includes an AHCI controller [1].

UEFI
Get UEFI support by replacing the bios using -bios ./OVMF-pure-efi.fd or OVMF-with-csm.fd if legacy boot is desired.
OVMF can be downloaded from https://www.kraxel.org/repos/jenkins/edk2/ (pick edk2.git-ovmf-x64-...). bsdtar can extract the rpms.
Many distributions also offer a matching ovmf package in their repos.

If you need UEFI settings (such as display resolution) to be persisted, a copy of OVMF_VARS (one per VM) needs to be provided too:

-drive file=./OVMF_CODE-pure-efi.fd,format=raw,if=pflash,readonly=on \
-drive file=./OVMF_VARS-pure-efi.fd,format=raw,if=pflash
Attaching disk images using VirtIO

Use -drive file=disk.img,if=virtio for improved disk performance. Windows guests require additional drivers [2] to use this.

Disabling disk cache flush
If you are installing a VM to quickly test something, disabling flushing of write cache to disk can speed up the process immensely:
-drive file=foobar.qcow2,if=virtio,cache=unsafe (or cache.no-flush=on)

Caution: Never use this in a production environment or with any VMs or data you care about. The guest OS will be tricked into believing it has safely written data to disk, when in reality it could be lost at any moment.

Attaching raw disks
-drive file=/dev/sdb,if=virtio,format=raw,cache=none

When attaching entire disks, partitions or logical volumes cache=none is a good idea.

Share host directory to guest

-drive file=fat:/path/to/dir,snapshot=on creates a read-only virtual FAT-formatted disk image from the given directory.

Multiple CD-ROM images
-drive file=X.iso,index=0,media=cdrom -drive file=Y.iso,index=1,media=cdrom

The index=N parameter is optional but can be used to explicitly order drives.

Bridged Network Adapter
-netdev bridge,br=br0,id=mynet -device virtio-net-pci,netdev=mynet
short syntax: -nic bridge,br=br0,model=virtio
For virtio, Windows needs additional drivers [2].
Aside from virtio-net-pci QEMU also supports emulating real cards such as:

e1000e (Intel 82574L GbE) which is the default, e1000 (Intel 82540EM GbE) or rtl8139 (Realtek RTL-8139C 10/100M)

CPU type
The default is -cpu qemu64.
To get the full CPU feature set in the guest use -cpu host or the appropriate family, e.g. -cpu Haswell.
Alternatively, flags can also be enabled individually: -cpu qemu64,+ssse3,+sse4.2,+avx,+avx2

-cpu kvm64 is legacy and should not be used [3].

VNC

-display vnc=localhost:1,lossy=on starts VNC on port 5901 (no authentication, but localhost only) with JPEG compression enabled to save bandwidth.

io_uring

io_uring is supported by Linux 5.1 and newer (on the host) and can further raise disk performance: -drive file=disk.qcow2,aio=io_uring

USB Input Devices
-usb -device usb-tablet -device usb-kbd attaches attaches keyboard and tablet (as mouse) via USB instead of PS/2.

This improves mouse support especially when using VNC and makes grabbing unnecessary in the GUI.

Port forwarding with User networking
When using -nic user (default) the hostfwd=PROTO::HPORT-:PORT option can be used to forward connections to the guest.

e.g. -nic user,model=virtio,hostfwd=tcp::2222-:22

VGA driver
-vga qxl offers improved performance over the default (std). Windows needs drivers, again [2].

3D acceleration for Linux guests is possible with -vga virtio [4].

Serial console
-serial pty connects the serial port to a PTY, which can then be interacted with using screen.
Alternatively when -nographic is used, the QEMU monitor and serial get multiplexed to stdio.

Ctrl-A c can then be used to switch between the monitor/serial [5].

Monitor console
With either -nographic or -monitor stdio, QEMU will make its monitor console available in the terminal.
It can also be accessed when using the GUI via "View" > "compatmonitor0".

The monitor console provides access to lots of functionality including snapshots, migration, device hotplug and debugging.

  • stop pauses the VM, cont resumes execution again

  • info registers shows the current values of CPU registers

  • x/x ADDRESS prints the hexadecimal value at the given address in guest memory

  • x/20i ADDRESS disassembles 20 instructions at the given address

Detailed explanations can be found in the relevant documentation.

Emulated SCSI controller
(because it's possible, not because it's useful)

-device lsi,id=lsi -drive file=somewhere.img,if=none,id=disk0 -device scsi-hd,drive=disk0,bus=lsi.0

Further reading