Throughout this article we will study the FreeBSD boot process. FreeBSD’s boot process is very robust and well thought out, but it differs slightly depending on your system architecture, filesystem (UFS2 or ZFS), partitioning scheme (GPT or MBR), and whether the system boots under UEFI or legacy BIOS (also known as CSM).
BIOS
BIOS—also known as legacy boot or CSM mode—is the old school 16-bit system boot mechanism that is still supported on most computers. BIOS still works pretty well for the FreeBSD boot process, but it’s a bit slower than UEFI, and doesn’t support booting from NVMe devices, BIOS mode supports MBR partitions, FreeBSD disk slices, and the newer GPT partitioning scheme as well.
UEFI
All modern amd64 [1] systems provide UEFI boot, and most of them also offer the legacy BIOS mode. Some even allow you to configure one as primary, and the other as fallback if the first method fails.
UEFI mode is also available on many arm64 systems, and is required to achieve ARM’s ServerReady certification.
Some arm64 and riscv/riscv64 (for RISC-V architectures) systems also allow booting with U-Boot bootloader instead of UEFI firmware - FreeBSD supports both of these methods. The RISC-V systems can be also booted with OpenSBI bootloader which superseded the BBL (Berkley Boot Loader) from the FreeBSD project. The FreeBSD team is currently working on implementing UEFI support on RISC-V systems.
The FreeBSD Boot Process
The FreeBSD boot process can be quite involved, and supports many subtly different components. To make the process easier to follow, we’ve broken this article down into two sections. The first covers the road to the FreeBSD loader(8) that provides the iconic “Beastie Menu”, and the second covers what happens after we exit the loader and boot the selected FreeBSD kernel.
The Road to the FreeBSD loader(8)
FreeBSD has several branching paths that lead to loading the FreeBSD kernel and then passing the rest of the boot process to PID 1 - the init(8) system. Which exact path it follows depends on system architecture, boot method, partition scheme, and filesystem.
Before we go into the details of each booting mode, first let’s list a simplified breakdown of all of them:
BIOS/ MBR/UFS
+-> MBR from 'Boot Device' BIOS disk | MBR
+-> boot0 | STAGE 0
+-> boot1 | STAGE 1
+-> boot2 | STAGE 2
+-> loader | STAGE 3
+-> kernel | KERNEL
+-> init | INIT
BIOS/ MBR/ZFS
+-> MBR from 'Boot Device' BIOS disk | MBR
+-> boot0 | STAGE 0
+-> boot1 | STAGE 1
+-> zfsboot | STAGE 2
+-> zfsloader | STAGE 3
+-> kernel | KERNEL
+-> init | INIT
BIOS/ GPT/UFS
+-> GPT from 'Boot Device' BIOS disk | GPT
+-> pmbr | STAGE 0
+-> gptboot | STAGE 1 + STAGE 2
+-> loader | STAGE 3
+-> kernel | KERNEL
+-> init | INIT
BIOS/ GPT/ZFS
+-> GPT from 'Boot Device' BIOS disk | GPT
+-> pmbr | STAGE 0
+-> gptzfsboot | STAGE 1 + STAGE 2
+-> zfsloader (analogous to loader) | STAGE 3
+-> kernel | KERNEL
+-> init | INIT
UEFI/GPT/MBR/UFS/ZFS
+-> GPT/MBR from 'Boot Device' BIOS disk | GPT/MBR
+-> UEFI | STAGE 0
+-> boot1.efi (/efi/boot/boot${ARCH}.efi) | STAGE 1 + STAGE 2
+-> loader.efi | STAGE 3
+-> kernel | KERNEL
+-> init | INIT
UEFI/GPT/MBR/UFS/ZFS (13.0 and later)
+-> GPT/MBR from 'Boot Device' BIOS disk | GPT/MBR
+-> UEFI | STAGE 0
+-> loader.efi (/efi/FreeBSD/loader.efi) | STAGE 1-3
+-> kernel | KERNEL
+-> init | INIT
The ARCH variable above is one of the following values depending on your system: x86 - x64 - arm - aa64
Now that you have a basic overview of the boot process, let’s examine the details of each one.
BIOS/Legacy using MBR and UFS
The oldest way of booting FreeBSD is with a Master Boot Record (MBR) partition table and the UFS filesystem. Let’s assume that your boot device is a SATA disk, /dev/ada0. It includes one primary MBR partition (called a slice in the FreeBSD vernacular) visible as /dev/ada0s1. That slice will need to have the active flag set. Inside that slice you can find the BSD partitions that are created with the bsdlabel(8) tool. The root partition that would be read during the boot process will be /dev/ada0s1a.
Booting the BIOS with MBR requires several stages of boot process due to size constraints. The BIOS loads and executes the first sector of the disk. This is called Stage 0 and uses the /boot/boot0 file. This file contains 446 bytes of assembly code, with the rest of the 512-byte sector containing the actual partition table (this is why MBR is limited to 4 partitions). After it loads and executes, this 446-byte of program parses the partition table and displays a simple boot menu based on the partition types found, as shown below:
F1 FreeBSD
F2 Win
Default: F1
Additionally, you can use /boot/boot0sio in place of boot0, and it will print this menu to the serial console as well. However, if your machine doesn’t have a serial console, this may cause the machine to hang during boot, so it is not the default.
The next part—reading and executing /boot/boot1 contents from the first sector of the selected partition (slice)—is Stage 1, and it’s generally a combination of the /boot/boot1 (512 bytes) and /boot/boot2 (7680 bytes) files which reside on the first 16 512-byte sectors (16 x 512 bytes), which UFS purposely does not use.
The main task of boot1 is to load the next boot stage, which is a little more complex. It loads the BTX server, which then loads its boot2 client, and then another client called loader.
Next, we move on to Stage 2. The boot2 defines and initializes an important data structure called bootinfo, passes it to the loader, then finally passes it to the FreeBSD kernel—but more on that later. The boot2 is a small program with just enough knowledge of the UFS filesystem to find the /boot/loader and /boot/kernel/kernel files—pretty impressive, for 7680 bytes of code.
Stage 3 then executes the loader which finds the kernel to boot. As explained earlier, the loader is also a BTX client. Configuration files /boot/loader.rc and /boot/loader.conf are processed and loaded. The main task of loader is to boot the FreeBSD kernel, but you can select which one to boot in the loader. When the kernel is loaded into memory, kernel initialization takes place.
BIOS/Legacy using MBR and ZFS
Booting the ZFS filesystem from the MBR partition scheme is a bit different. At Stage 0, boot0 loads a different boot1 file. boot1 then finds the 64KiB zfsboot file (analogous to boot2) at an offset of 1 MiB into the ZFS partition—this is a predetermined blank space in the ZFS on disk format.
This means that at Stage 1 and Stage 2 simultaneously the zfsboot code understands the ZFS file system and loads zfsloader, which provides an interactive menu similar to that of loader for the UFS filesystem. Much like loader, zfsloader then loads kernel and boots the rest of the FreeBSD OS.
This is Stage 3 at which zfsloader is loaded and running. It is now possible to select a different ZFS Boot Environment to boot from.
BIOS/Legacy using GPT and UFS
Now, let’s examine a slightly more modern boot process, using the GPT partition scheme but still using the older BIOS (also known as Legacy or CSM) boot mode. Although our disk is partitioned using GPT, it still contains a Protective MBR (called pmbr in short). The purpose of pmbr is to keep operating systems that do not understand GPT partitioning from misidentifying the disk as unpartitioned.
The pmbr boots the FreeBSD system in a similar way to the MBR method with one 512-byte sector that is read and executed - we can call this Stage 0.
The dedicated boot partition for GPT (type freebsd-boot) contains the remaining bootcode—, gptldr which is analogous to boot1, and gptboot in place of boot2—which serves as Stage 1and Stage 2 respectively.
The pmbr locates and reads from this boot partition and executes it. For the UFS filesystem, the gptboot file is used. Again, Stage 1 is a 512 byte assembly program that just loads and executes Stage 2.gptboot contains enough code to understand UFS and optionally GELI encryption, and is able to locate, load and launch the loader from the UFS filesystem.
At Stage 3 when loader is loaded and running, one can also select different kernels to boot from.
BIOS/Legacy using GPT and ZFS
Booting from GPT and ZFS is largely similar to booting from GPT and UFS2. First, the pmbr boots the FreeBSD system in a similar way to the MBR method with one 512-byte sector that is read and executed—we can call this Stage 0.
The dedicated boot partition for GPT (again type freebsd-boot) contains the bootcode—gptldr as an analog for boot1 and zfsboot as the analogue for boot2— again referred to as Stage 1 and Stage 2 respectively.
The pmbr reads from this boot partition and executes it. For the ZFS filesystem, the gptzfsboot file contains enough knowledge of ZFS and optionally GELI encryption to decrypt, load, and launch the zfsloader from the default ZFS dataset.
When zfsloader is loaded and running, we have reached Stage 3 and can select different ZFS Boot Environment and kernels to boot from.
UEFI using GPT/MBR and UFS/ZFS
Now let’s talk about the 'most modern' way of booting the FreeBSD system - with Unified Extensible Firmware Interface, (UEFI). UEFI is faster than the old BIOS method, because it runs in 64bit mode as soon as possible.
Some background on this: at first there was Intel EFI implementation for their Itanium (IA64) architecture. Later, that work was taken and 'unified' to be used on other architectures - thus the Unified at the beginning of the name. Of course, this is a very short version of its history—the rabbit hole goes considerably deeper, but this is enough for now.
FreeBSD supports the UEFI boot mode very well. Of course, as with other boot methods several stages are involved. These steps look largely the same whether the boot disk is partitioned GPT or MBR.
At Stage 0 the UEFI firmware runs at power up and searches for the OS loader in the FAT32 filesystem on an EFI File System Partition (ESP). The UEFI ESP partition has C12A7328-F81F-11D2-BA4B-00A0C93EC93B UUID.
Depending on the architecture, it may be one of the following:
ARCH WIDE PATH
i386 32bit /efi/boot/bootx86.efi
amd64 64bit /efi/boot/bootx64.efi
arm 32bit /efi/boot/bootarm.efi
arm64 64bit /efi/boot/bootaa64.efi
By default, FreeBSD installs boot1.efi as bootx64.efi file. Executing it means we are at Stage 1 and Stage 2 steps.
The boot1.efi uses the following sequence to determine the root filesystem for the FreeBSD boot process:
· for ZFS it will search for ZFS bootable pools.
· for UFS it will search for bootable UFS partitions.
The partition is recognized as bootable if boot1.efi can load loader.efi from it. If both UFS and ZFS partitions exist on the same device, the ZFS partition is preferred. After UEFI executes bootx64.efi (which is boot1.efi file), we’ve reached Stage 3. We now load loader.efi, and can select the ZFS Boot Environment and kernel we wish to boot from.
Once the boot environment and kernel are selected, loader.efi loads the FreeBSD kernel.
Kernel and Multi User Mode
Just before loader boots the selected kernel, it reads and loads settings from the /boot/device.hints file. After selecting the kernel and or ZFS Boot Environment to boot from, it starts the kernel and begins to probe hardware for devices and initialize them.
The kernel then passes control to /sbin/init (with PID 1, of course) which mounts the filesystems.
The init(8) process also takes care of other things—such as terminal ports found in the ttys(5) file—and executes getty(8) processes, which in turn initialize login(1) commands.
With that done, init(8) continues the boot process on the road to Multi User Mode, in which it starts configuring the resources rc(8) of the FreeBSD system. The resource configuration defaults are read from /etc/defaults/rc.conf file, and system specific details are loaded from the /etc/rc.conf file.
Then, the filesystems from /etc/fstab are mounted. Network interfaces are addressed. Finally, services and daemons are started using the startup scripts located in /etc/rc.d and /usr/local/etc/rc.d, with rcorder(8) making certain that services are started in the correct order.
nextboot(8)
FreeBSD comes with the nextboot(8) feature that allows you to specify alternate kernel and boot flags for the next reboot cycle. But only once,for the very next boot process.
This is achieved by loader(8) preferring to load kernel information from the /boot/nextboot.conf file if present. Once the machine is rebooted, nextboot.conf is automatically removed, reverting the system to its previous 'permanent' configuration.
For ZFS filesystems, nextboot stores metadata in a designated place in the ZFS pool label, since the early boot code can’t handle writing directly to the pool. The nextboot metadata is erased as soon as it is read, so it only impacts the next single boot after it’s been written.
To boot the CUSTOM FreeBSD kernel once, issue the following nextboot(8) command:
# nextboot -k CUSTOMTo force the Single User Mode along with booting the TEST kernel do this:
# nextboot -o "-s" -k TESTYou can also remove any existing nextboot(8) configuration with this option:
# nextboot -DThe bootme/bootonce/bootfailed Flags
FreeBSD implements several extra boot flags. These are bootme, bootonce, and bootfailed. The bootme flag is used to specify which partition should be used for the boot process. The bootonce flag is used with bootme flag to boot the specified partition once only and then fail back to what was configured permanently. The FreeBSD loader(8) achieves that by removing these flags.The last flag—bootfailed— is used to show that the partition failed to boot. You can use the gpart(8) tool to manipulate these flags as needed. For example:
# gpart set -a bootme -i 2 ada0These flags are very useful if you have more than one UFS root partition with some other version of the FreeBSD system. You can then set the bootme flag to boot from that partition instead.
Example Commands to Manipulate the Boot Process
The boot code at MBR can be changed/updated with these commands:
# fdisk -B
# fdisk -B -b /boot/boot0 ada0
# boot0cfg -B
# boot0cfg -s 1 -b /boot/boot0 ada0
Installation process for the zfsboot on MBR slice.
# gpart create -s mbr ada0
# gpart add -t freebsd ada0
# gpart bootcode -b /boot/boot0 ada0
# gpart set -a active -i 1 ada0
# dd if=/dev/zero of=/dev/ada0s1 count=2
# dd if=/boot/zfsboot of=/dev/ada0s1 count=1
# dd if=/boot/zfsboot of=/dev/ada0s1 iseek=1 oseek=1024
Similar commands for GPT below:
UFS # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0
ZFS # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
Creating UEFI ESP partition manually.
# gpart add -a 4K -t efi -s 200M ada0
# newfs_msdos /dev/ada0s1
# mount_msdosfs /dev/ada0s1 /mnt
# mkdir -p /mnt/efi/boot
# cp /boot/boot1.efi /mnt/efi/boot/bootx64.efi
Additional Resources
Here are some interesting FreeBSD resources on the boot process topic that you may also find useful:
- FreeBSD Architecture Handbook - Chapter 1 - Bootstrapping and Kernel Initialization
- FreeBSD Handbook - Chapter 13 - FreeBSD Booting Process
- boot(8)
- boot.config(5)
- boot0cfg(8)
- boot1.efi(8)
- efibootmgr(8)
- efivar(8)
- fdisk(8)
- gptboot(8)
- gptzfsboot(8)
- zfsboot(8)
- zfsbootcfg(8)
- loader(8)
- loader.efi(8)
- nextboot(8)
- fastboot(8)
Embedded