Virtualizing Raspbian (or any ARM/Linux distro) headless using QEMU
For testing or development it can be very useful to have a distribution that usually runs on an embedded ARM board such as the Raspberry Pi run right on your machine (that isn't ARM) using a virtual machine.
QEMU provides excellent support for emulation of the ARM architecture (both 32 and 64-bit) and can emulate many different real ARM boards.
Why not use QEMU's "raspi2" machine for emulation?
raspi2 machine. It emulates the GPU's framebuffer, HWRNG, UART, GPIO and SD controller.Spot something missing? It doesn't implement USB, which makes it useless for headless and graphical use as you can plug in neither a network connection nor a keyboard or mouse.
(Update 2021-01-08: QEMU has apparently added raspi USB support in versions 5.1.0 onward, so you could skip much of the setup detailed here if doing this from scratch.)
-M raspi2b -kernel kernel7l.img -dtb bcm2709-rpi-2-b.dtb -append "root=/dev/mmcblk0 rw console=ttyAMA0"
The plan
Instead of (poorly) emulating a real piece of hardware, QEMU also has a virt
machine [1] that is designed for virtualization. It gives you a modern system
with PCI and also works out-of-the-box with Linux without providing a Device Tree
(QEMU generates one internally).
The most straightforward way of getting network and disk into such a VM is to
use virtio-net and virtio-disk respectively, which is what we'll be doing.
I picked Arch Linux ARM's armv7 kernel from here, though any other should
work just as well provided it comes with the appropriate modules.
To load the virtio modules during boot we'll require an initramfs, but more on that later.
Extracting Raspbian's root filesystem into a virtual disk image
Start by downloading Raspbian from the Raspberry Pi website, then run the script below or follow the steps manually.
The script will create a copy of the image file, expand the image and its partition to 10 gigabytes, mount the partition using a loop device and make two adjustments:
Remove both SD card partitions from
/etc/fstab, these don't exist inside the VM and we will be mounting the rootfs ourselvesDisable several startup services that do not work inside the VM
After unmounting the partition it will convert the filesystem into a qcow2
format image for use with QEMU.
#!/bin/bash -e input=2020-02-13-raspbian-buster-lite.img [ -f $input ] mkdir mnt cp --reflink=auto $input source.img truncate -s 10G source.img echo ", +" | sfdisk -N 2 source.img dev=$(sudo losetup -fP --show source.img) [ -n "$dev" ] sudo resize2fs ${dev}p2 sudo mount ${dev}p2 ./mnt -o rw sudo sed '/^PARTUUID/d' -i ./mnt/etc/fstab sudo rm -f \ ./mnt/etc/systemd/system/multi-user.target.wants/{hciuart,dphys-swapfile}.service \ ./mnt/etc/rc?.d/?01{resize2fs_once,rng-tools,rng-tools-debian} sudo umount ./mnt sudo chmod a+r ${dev}p2 qemu-img convert -O qcow2 ${dev}p2 rootfs.qcow2 sudo losetup -d $dev rm source.img; rmdir mnt
The kernel and initramfs
Kernel
linux-armv7 package is just a tar archive, so you can extract the kernel executable using:tar -xvf linux-armv7*.pkg.tar.xz --strip-components=1 boot/zImage
Making an initramfs
Since virtio support is not compiled into the kernel and the root filesytem is missing modules for the exact kernel we'll be using (maybe copying them would've been easier?), we need to write an initramfs that can load these modules prior to mounting the rootfs.
Fortunately the Gentoo Wiki has a great article on making a custom one yourself. The basic idea is to extract the required kernel modules into the initramfs, whose init script loads the modules, mounts the root filesystem and actually boots.
The script shown below does the following steps:
Extract kernel modules from package
Delete some that we won't be needing and take a lot of space (optional)
Download and install a statically-linked busybox executable
Create the init script
Pack contents into a
cpioarchive as required by the Linux kernel
Using a virtio disk and network adapter requires loading the virtio-pci,
virtio-blk, virtio-net modules. If you need any more the
init script can easily be changed accordingly.
#!/bin/bash -e pkg=$(echo linux-armv7-*.pkg.tar.xz) [ -f "$pkg" ] mkdir initrd; pushd initrd mkdir bin dev mnt proc sys tar -xaf "../$pkg" --strip-components=1 usr/lib/modules rm -rf lib/modules/*/kernel/{sound,drivers/{net/{wireless,ethernet},media,gpu,iio,staging,scsi}} wget https://www.busybox.net/downloads/binaries/1.31.0-defconfig-multiarch-musl/busybox-armv7l -O bin/busybox cat >init <<"SCRIPT" #!/bin/busybox sh busybox mount -t proc none /proc busybox mount -t sysfs none /sys busybox mount -t devtmpfs none /dev for mod in virtio-pci virtio-blk virtio-net; do busybox modprobe $mod done busybox mount -o rw /dev/vda /mnt || exit 1 busybox umount /proc busybox umount /sys busybox umount /dev exec busybox switch_root /mnt /sbin/init SCRIPT chmod +x bin/busybox init bsdtar --format newc --uid 0 --gid 0 -cf - -- * | gzip -9 >../initrd.gz popd; rm -r initrd
Booting the virtual machine
With the initramfs built, we have all parts needed to actually run the VM: [2]
qemu-system-arm -M virt,highmem=off -m 2048 -smp 4 -kernel zImage -initrd initrd.gz \ -drive file=rootfs.qcow2,if=virtio -nic user,model=virtio \ -append "console=ttyAMA0" -nographic
Raspbian GNU/Linux 10 raspberrypi ttyAMA0 and a login prompt.Further steps
This virtualization approach should work for just about any ARM/Linux distribution. I have tested it with Raspbian, Void Linux and Arch Linux ARM (whose rootfs even works without any modifications).
To ensure the kernel performs as expected beyond basic tasks, it's a good idea
to extract the modules from the linux-armv7 package into the guest rootfs.
As with any VM, you can use the full extent of QEMU's features to e.g.:
attach an USB controller (
-device usb-ehciornec-usb-xhci)..SCSI controller (
-device virtio-scsi)..Audio input/output (
-device usb-audio)or even enable graphical output (
-device VGA)
AArch64?
With a few adjustments in the right places, this guide also works to emulate an AArch64 kernel and userland. I wrote this up in a newer article.