X32 is an ABI for Linux that uses the x86-64 instruction set but 32-bit longs and pointers (this is called ILP32), thereby limiting the memory for a single process to 4 GiB. Compared to amd64 it offers significant memory savings and unlike plain i386 it can make use of all registers and extensions also available to 64-bit code.

• A Debian netinst CD for the amd64 architecture: https://www.debian.org/CD/netinst/

• A computer or VM with x86-64 compatible CPU

• An internet connection on the machine you are installing

mount --bind /dev /target/dev
mount --bind /proc /target/proc
mount --bind /sys /target/sys
chroot /target

passwd root
echo "/dev/sda1 / ext4 rw 0 1" >/etc/fstab

apt install -y debian-ports-archive-keyring
cat >/etc/apt/sources.list
deb [arch=x32] http://ftp.ports.debian.org/debian-ports unstable main
deb [arch=amd64] http://deb.debian.org/debian unstable main
^D
cat >/etc/apt/preferences.d/amd64
Package: linux-image*:amd64
Pin: release b=amd64
Pin-Priority: 500

Package: *
Pin: release b=amd64
Pin-Priority: -1
^D
dpkg --add-architecture amd64
apt update

cat >/etc/systemd/network/eth.network
[Match]
Name=en*
[Network]
DHCP=ipv4
^D
systemctl enable --now systemd-networkd

tasksel install standard

eval $(tasksel -t install standard | tr ' ' '\n' | egrep -v 'isc-dhcp-client|^bind9-' | tr '\n' ' ')

For testing or development it can be very useful to have a distribution that usually runs on an embedded ARM board such as the Raspberry Pi run right on your machine (that isn't ARM) using a virtual machine.

QEMU provides excellent support for emulation of the ARM architecture (both 32 and 64-bit) and can emulate many different real ARM boards.

#!/bin/bash -e
input=2020-02-13-raspbian-buster-lite.img
[ -f $input ]

mkdir mnt
cp --reflink=auto $input source.img
truncate -s 10G source.img
echo ", +" | sfdisk -N 2 source.img
dev=$(sudo losetup -fP --show source.img)
[ -n "$dev" ]
sudo resize2fs ${dev}p2
sudo mount ${dev}p2 ./mnt -o rw
sudo sed '/^PARTUUID/d' -i ./mnt/etc/fstab
sudo rm -f \
        ./mnt/etc/systemd/system/multi-user.target.wants/{hciuart,dphys-swapfile}.service \
        ./mnt/etc/rc?.d/?01{resize2fs_once,rng-tools,rng-tools-debian}
sudo umount ./mnt
sudo chmod a+r ${dev}p2
qemu-img convert -O qcow2 ${dev}p2 rootfs.qcow2
sudo losetup -d $dev
rm source.img; rmdir mnt

#!/bin/bash -e
pkg=$(echo linux-armv7-*.pkg.tar.xz)
[ -f "$pkg" ]

mkdir initrd; pushd initrd
mkdir bin dev mnt proc sys
tar -xaf "../$pkg" --strip-components=1 usr/lib/modules
rm -rf lib/modules/*/kernel/{sound,drivers/{net/{wireless,ethernet},media,gpu,iio,staging,scsi}}
wget https://www.busybox.net/downloads/binaries/1.31.0-defconfig-multiarch-musl/busybox-armv7l -O bin/busybox
cat >init <<"SCRIPT"
#!/bin/busybox sh
busybox mount -t proc none /proc
busybox mount -t sysfs none /sys
busybox mount -t devtmpfs none /dev

for mod in virtio-pci virtio-blk virtio-net; do
        busybox modprobe $mod
done

busybox mount -o rw /dev/vda /mnt || exit 1

busybox umount /proc
busybox umount /sys
busybox umount /dev

exec busybox switch_root /mnt /sbin/init
SCRIPT
chmod +x bin/busybox init
bsdtar --format newc --uid 0 --gid 0 -cf - -- * | gzip -9 >../initrd.gz
popd; rm -r initrd

qemu-system-arm -M virt,highmem=off -m 2048 -smp 4 -kernel zImage -initrd initrd.gz \
        -drive file=rootfs.qcow2,if=virtio -nic user,model=virtio \
        -append "console=ttyAMA0" -nographic

If you try to open shell inside a container that runs e.g. Alpine Linux using machinectl, the following not very descriptive error will appear:

# machinectl shell vpn
Failed to get shell PTY: Protocol error

The reason for this is simply that the container is not running systemd.

Because systemd-nspawn just uses Linux namespaces [1], nsenter can alternatively be used to access the container. For this, we'll need the PID of the init process inside the container:

# systemctl status systemd-nspawn@vpn
● systemd-nspawn@vpn.service - Container vpn
   Loaded: loaded (/lib/systemd/system/systemd-nspawn@.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-08-11 19:49:19 UTC; 6 months 3 days ago
 Main PID: 795 (systemd-nspawn)
   Status: "Container running."
   CGroup: /machine.slice/systemd-nspawn@vpn.service
           ├─payload
           │ ├─ 797 /sbin/init
           │ ├─1028 /sbin/syslogd -t
           [...]

nsenter -t 797 -a /bin/sh -l

Depending on the namespaces the container is (or isn't) joined to it can be necessary to specify the appropriate flags manually after consulting the nsenter manpage:

All in all, this can be turned into a nice alias for your .bashrc:

function center ()
{
  [ -z "$1" ] && { echo "Usage: center <name>" >&2; return 1; }
  pid=$(sed -n 2p "/sys/fs/cgroup/pids/machine.slice/systemd-nspawn@$1.service/tasks")
  [ -z "$pid" ] && { echo "Container not running" >&2; return 1; }
  nsenter -t $pid -a /bin/sh -l
}

This will cover some QEMU options I have found useful beyond the basics.

Machine Type (x86)

-M q35 configures a more modern chipset to be emulated. The Q35 chipset supports PCI-e and includes an AHCI controller [1].

UEFI

Get UEFI support by replacing the bios using -bios ./OVMF-pure-efi.fd or OVMF-with-csm.fd if legacy boot is desired.

OVMF can be downloaded from https://www.kraxel.org/repos/jenkins/edk2/ (pick edk2.git-ovmf-x64-...). bsdtar can extract the rpms.

Many distributions also offer a matching ovmf package in their repos.

If you need UEFI settings (such as display resolution) to be persisted, a copy of OVMF_VARS (one per VM) needs to be provided too:

-drive file=./OVMF_CODE-pure-efi.fd,format=raw,if=pflash,readonly=on \
-drive file=./OVMF_VARS-pure-efi.fd,format=raw,if=pflash

Attaching disk images using VirtIO

Use -drive file=disk.img,if=virtio for improved disk performance. Windows guests require additional drivers [2] to use this.

Disabling disk cache flush

If you are installing a VM to quickly test something, disabling flushing of write cache to disk can speed up the process immensely:

-drive file=foobar.qcow2,if=virtio,cache=unsafe (or cache.no-flush=on)

Caution: Never use this in a production environment or with any VMs or data you care about. The guest OS will be tricked into believing it has safely written data to disk, when in reality it could be lost at any moment.

Attaching raw disks

-drive file=/dev/sdb,if=virtio,format=raw,cache=none

When attaching entire disks, partitions or logical volumes cache=none is a good idea.

Share host directory to guest

-drive file=fat:/path/to/dir,snapshot=on creates a read-only virtual FAT-formatted disk image from the given directory.

Multiple CD-ROM images

-drive file=X.iso,index=0,media=cdrom -drive file=Y.iso,index=1,media=cdrom

The index=N parameter is optional but can be used to explicitly order drives.

Bridged Network Adapter

-netdev bridge,br=br0,id=mynet -device virtio-net-pci,netdev=mynet

short syntax: -nic bridge,br=br0,model=virtio

For virtio, Windows needs additional drivers [2].

Aside from virtio-net-pci QEMU also supports emulating real cards such as:

e1000e (Intel 82574L GbE) which is the default, e1000 (Intel 82540EM GbE) or rtl8139 (Realtek RTL-8139C 10/100M)

CPU type

The default is -cpu qemu64.

To get the full CPU feature set in the guest use -cpu host or the appropriate family, e.g. -cpu Haswell.

Alternatively, flags can also be enabled individually: -cpu qemu64,+ssse3,+sse4.2,+avx,+avx2

-cpu kvm64 is legacy and should not be used [3].

VNC

-display vnc=localhost:1,lossy=on starts VNC on port 5901 (no authentication, but localhost only) with JPEG compression enabled to save bandwidth.

io_uring

io_uring is supported by Linux 5.1 and newer (on the host) and can further raise disk performance: -drive file=disk.qcow2,aio=io_uring

USB Input Devices

-usb -device usb-tablet -device usb-kbd attaches attaches keyboard and tablet (as mouse) via USB instead of PS/2.

This improves mouse support especially when using VNC and makes grabbing unnecessary in the GUI.

Port forwarding with User networking

When using -nic user (default) the hostfwd=PROTO::HPORT-:PORT option can be used to forward connections to the guest.

e.g. -nic user,model=virtio,hostfwd=tcp::2222-:22

VGA driver

-vga qxl offers improved performance over the default (std). Windows needs drivers, again [2].

3D acceleration for Linux guests is possible with -vga virtio [4].

Serial console

-serial pty connects the serial port to a PTY, which can then be interacted with using screen.

Alternatively when -nographic is used, the QEMU monitor and serial get multiplexed to stdio.

Ctrl-A c can then be used to switch between the monitor/serial [5].

Monitor console

With either -nographic or -monitor stdio, QEMU will make its monitor console available in the terminal.

It can also be accessed when using the GUI via "View" > "compatmonitor0".

The monitor console provides access to lots of functionality including snapshots, migration, device hotplug and debugging.

• stop pauses the VM, cont resumes execution again

• info registers shows the current values of CPU registers

• x/x ADDRESS prints the hexadecimal value at the given address in guest memory

• x/20i ADDRESS disassembles 20 instructions at the given address

Detailed explanations can be found in the relevant documentation.

Emulated SCSI controller

(because it's possible, not because it's useful)

-device lsi,id=lsi -drive file=somewhere.img,if=none,id=disk0 -device scsi-hd,drive=disk0,bus=lsi.0

Emulated USB drive

This may not seem useful at first look but will help if what you're running on the VM refuses to operate on "system" disks and requires a removable drive.

-drive file=whatever.qcow2,if=none,id=disk1 -device usb-storage,drive=disk1