The FreeBSD Boot Process

The FreeBSD Boot Process

Robust But Powerful – Adjust Your FreeBSD Boot Process According to Your System Architecture

Throughout this article we will study the FreeBSD boot process. FreeBSD’s boot process is very robust and well thought out, but it differs slightly depending on your system architecture, filesystem (UFS2 or ZFS), partitioning scheme (GPT or MBR), and whether the system boots under UEFI or legacy BIOS (also known as CSM).


BIOS

BIOS—also known as legacy boot or CSM mode—is the old school 16-bit system boot mechanism that is still supported on most computers. BIOS still works pretty well for the FreeBSD boot process, but it’s a bit slower than UEFI, and doesn’t support booting from NVMe devices, BIOS mode supports MBR partitions, FreeBSD disk slices, and the newer GPT partitioning scheme as well.

UEFI

All modern amd64 [1] systems provide UEFI boot, and most of them also offer the legacy BIOS mode. Some even allow you to configure one as primary, and the other as fallback if the first method fails.

UEFI mode is also available on many arm64 systems, and is required to achieve ARM’s ServerReady certification. 

Some arm64 and riscv/riscv64 (for RISC-V architectures) systems also allow booting with U-Boot bootloader instead of UEFI firmware – FreeBSD supports both of these methods. The RISC-V systems can be also booted with OpenSBI bootloader which superseded the BBL (Berkley Boot Loader) from the FreeBSD project. The FreeBSD team is currently working on implementing UEFI support on RISC-V systems.

The FreeBSD Boot Process

The FreeBSD boot process can be quite involved, and supports many subtly different components. To make the process easier to follow, we’ve broken this article down into two sections. The first covers the road to the FreeBSD loader(8) that provides the iconic “Beastie Menu”, and the second covers what happens after we exit the loader and boot the selected FreeBSD kernel.

The Road to the FreeBSD loader(8)

FreeBSD has several branching paths that lead to loading the FreeBSD kernel and then passing the rest of the boot process to PID 1 – the init(8) system. Which exact path it follows depends on system architecture, boot method, partition scheme, and filesystem.

You might also be interested in

Get more out of your FreeBSD development

Kernel development is crucial to many companies. If you have a FreeBSD implementation or you’re looking at scoping out work for the future, our team can help you further enable your efforts.

Before we go into the details of each booting mode, first let’s list a simplified breakdown of all of them:

BIOS/ MBR/UFS

  +-> MBR from 'Boot Device' BIOS disk          | MBR
    +-> boot0                                   | STAGE 0
      +-> boot1                                 | STAGE 1
        +-> boot2                               | STAGE 2
          +-> loader                            | STAGE 3
            +-> kernel                          | KERNEL
              +-> init                          | INIT

BIOS/ MBR/ZFS
  +-> MBR from 'Boot Device' BIOS disk          | MBR
    +-> boot0                                   | STAGE 0
      +-> boot1                                 | STAGE 1
        +-> zfsboot         | STAGE 2
          +-> zfsloader    | STAGE 3
            +-> kernel                          | KERNEL
              +-> init                          | INIT

BIOS/ GPT/UFS
  +-> GPT from 'Boot Device' BIOS disk          | GPT
    +-> pmbr                                    | STAGE 0
      +-> gptboot                               | STAGE 1 + STAGE 2
        +-> loader                              | STAGE 3
          +-> kernel                            | KERNEL
            +-> init                            | INIT

BIOS/ GPT/ZFS
  +-> GPT from 'Boot Device' BIOS disk          | GPT
    +-> pmbr                                    | STAGE 0
      +-> gptzfsboot                            | STAGE 1 + STAGE 2
        +-> zfsloader (analogous to loader)     | STAGE 3
          +-> kernel                            | KERNEL
            +-> init                            | INIT

UEFI/GPT/MBR/UFS/ZFS
  +-> GPT/MBR from 'Boot Device' BIOS disk      | GPT/MBR
    +-> UEFI                                    | STAGE 0
      +-> boot1.efi (/efi/boot/boot${ARCH}.efi) | STAGE 1 + STAGE 2
        +-> loader.efi                          | STAGE 3
          +-> kernel                            | KERNEL
            +-> init                            | INIT

UEFI/GPT/MBR/UFS/ZFS (13.0 and later)
  +-> GPT/MBR from 'Boot Device' BIOS disk      | GPT/MBR
    +-> UEFI                                    | STAGE 0
      +-> loader.efi (/efi/FreeBSD/loader.efi)  | STAGE 1-3
        +-> kernel                              | KERNEL
          +-> init                              | INIT

The ARCH variable above is one of the following values depending on your system: x86 – x64 – arm – aa64

Now that you have a basic overview of the boot process, let’s examine the details of each one.

BIOS/Legacy using MBR and UFS

The oldest way of booting FreeBSD is with a Master Boot Record (MBR) partition table and the UFS filesystem. Let’s assume that your boot device is a SATA disk, /dev/ada0. It includes one primary MBR partition (called a slice in the FreeBSD vernacular) visible as /dev/ada0s1. That slice will need to have the active flag set. Inside that slice you can find the BSD partitions that are created with the bsdlabel(8) tool. The root partition that would be read during the boot process will be /dev/ada0s1a.

Booting the BIOS with MBR requires several stages of boot process due to size constraints. The BIOS loads and executes the first sector of the disk. This is called Stage 0 and uses the /boot/boot0 file. This file contains 446 bytes of assembly code, with the rest of the 512-byte sector containing the actual partition table (this is why MBR is limited to 4 partitions). After it loads and executes, this 446-byte of program parses the partition table and displays a simple boot menu based on the partition types found, as shown below:

F1 FreeBSD
F2 Win

Default: F1

Additionally, you can use /boot/boot0sio in place of boot0, and it will print this menu to the serial console as well. However, if your machine doesn’t have a serial console, this may cause the machine to hang during boot, so it is not the default.

The next part—reading and executing /boot/boot1 contents from the first sector of the selected partition (slice)—is Stage 1, and it’s generally a combination of the /boot/boot1 (512 bytes) and /boot/boot2 (7680 bytes) files which reside on the first 16 512-byte sectors (16 x 512 bytes), which UFS purposely does not use.

The main task of boot1 is to load the next boot stage, which is a little more complex. It loads the BTX server, which then loads its boot2 client, and then another client called loader.

Next, we move on to Stage 2. The boot2 defines and initializes an important data structure called bootinfo, passes it to the loader, then finally passes it to the FreeBSD kernel—but more on that later. The boot2 is a small program with just enough knowledge of the UFS filesystem to find the /boot/loader and /boot/kernel/kernel files—pretty impressive, for 7680 bytes of code.

Stage 3 then executes the loader which finds the kernel to boot. As explained earlier, the loader is also a BTX client. Configuration files /boot/loader.rc and /boot/loader.conf are processed and loaded. The main task of loader is to boot the FreeBSD kernel, but you can select which one to boot in the loader. When the kernel is loaded into memory, kernel initialization takes place.

BIOS/Legacy using MBR and ZFS

Booting the ZFS filesystem from the MBR partition scheme is a bit different. At Stage 0boot0 loads a different boot1 file. boot1 then finds the 64KiB zfsboot file (analogous to boot2) at an offset of 1 MiB into the ZFS partition—this is a predetermined blank space in the ZFS on disk format.

This means that at Stage 1 and Stage 2 simultaneously the zfsboot code understands the ZFS file system and loads zfsloader, which provides an interactive menu similar to that of loader for the UFS filesystem. Much like loader, zfsloader  then loads kernel and boots the rest of the FreeBSD OS.

This is Stage 3 at which zfsloader is loaded and running. It is now possible to select a different ZFS Boot Environment to boot from.

BIOS/Legacy using GPT and UFS

Now, let’s examine a slightly more modern boot process, using the GPT partition scheme but still using the older BIOS (also known as Legacy or CSM) boot mode. Although our disk is partitioned using GPT, it still contains a Protective MBR (called pmbr in short). The purpose of pmbr is to keep operating systems that do not understand GPT partitioning from misidentifying the disk as unpartitioned.

The pmbr boots the FreeBSD system in a similar way to the MBR method with one 512-byte sector that is read and executed – we can call this Stage 0.

The dedicated boot partition for GPT (type freebsd-boot) contains the remaining bootcode—, gptldr which is analogous to boot1, and gptboot in place of boot2—which serves as Stage 1and Stage 2 respectively.

The pmbr locates and reads from this boot partition and executes it. For the UFS filesystem, the gptboot file is used. Again, Stage 1 is a 512 byte assembly program that just loads and executes Stage 2.gptboot contains enough code to understand UFS and optionally GELI encryption, and is able to locate, load and launch the loader from the UFS filesystem. 

At Stage 3 when loader is loaded and running, one can also select different kernels to boot from.

BIOS/Legacy using GPT and ZFS

Booting from GPT and ZFS is largely similar to booting from GPT and UFS2. First, the pmbr boots the FreeBSD system in a similar way to the MBR method with one 512-byte sector that is read and executed—we can call this Stage 0.

The dedicated boot partition for GPT (again type freebsd-boot) contains the bootcode—gptldr as an analog for boot1 and zfsboot as the analogue for boot2— again referred to as Stage 1 and Stage 2 respectively.

The pmbr reads from this boot partition and executes it. For the ZFS filesystem, the gptzfsboot file contains enough knowledge of ZFS and optionally GELI encryption to decrypt, load, and launch the zfsloader from the default ZFS dataset. 

When zfsloader is loaded and running, we have reached Stage 3 and can select different ZFS Boot Environment and kernels to boot from.

UEFI using GPT/MBR and UFS/ZFS

Now let’s talk about the ‘most modern’ way of booting the FreeBSD system – with Unified Extensible Firmware Interface, (UEFI). UEFI is faster than the old BIOS method, because it runs in 64bit mode as soon as possible. 

Some background on this: at first there was Intel EFI implementation for their Itanium (IA64) architecture. Later, that work was taken and ‘unified’ to be used on other architectures – thus the Unified at the beginning of the name. Of course, this is a very short version of its history—the rabbit hole goes considerably deeper, but this is enough for now.

FreeBSD supports the UEFI boot mode very well. Of course, as with other boot methods several stages are involved. These steps look largely the same whether the boot disk is partitioned GPT or MBR.

At Stage 0 the UEFI firmware runs at power up and searches for the OS loader in  the FAT32 filesystem on an EFI File System Partition (ESP). The UEFI ESP partition has C12A7328-F81F-11D2-BA4B-00A0C93EC93B UUID.

Depending on the architecture, it may be one of the following:

   ARCH  WIDE   PATH
   i386  32bit  /efi/boot/bootx86.efi
  amd64  64bit  /efi/boot/bootx64.efi
    arm  32bit  /efi/boot/bootarm.efi
  arm64  64bit  /efi/boot/bootaa64.efi

By default, FreeBSD installs boot1.efi as bootx64.efi file. Executing it means we are at Stage 1 and Stage 2 steps.

The boot1.efi uses the following sequence to determine the root filesystem for the FreeBSD boot process:

·  for ZFS it will search for ZFS bootable pools.

·  for UFS it will search for bootable UFS partitions.

The partition is recognized as bootable if boot1.efi can load loader.efi from it. If both UFS and ZFS partitions exist on the same device, the ZFS partition is preferred. After UEFI executes bootx64.efi (which is boot1.efi file), we’ve reached Stage 3. We now load loader.efi, and can select the ZFS Boot Environment and kernel we wish to boot from. 

Once the boot environment and kernel are selected, loader.efi loads the FreeBSD kernel.

Kernel and Multi User Mode

Just before loader boots the selected kernel, it reads and loads settings from the /boot/device.hints file. After selecting the kernel and or ZFS Boot Environment to boot from, it starts the kernel and begins to probe hardware for devices and initialize them.

The kernel then passes control to /sbin/init (with PID 1, of course) which mounts the filesystems.

The init(8) process also takes care of other things—such as terminal ports found in the ttys(5) file—and executes getty(8) processes, which in turn initialize login(1) commands. 

With that done, init(8) continues the boot process on the road to Multi User Mode, in which it starts configuring the resources rc(8) of the FreeBSD system. The resource configuration defaults are read from /etc/defaults/rc.conf file, and system specific details are loaded from the /etc/rc.conf file.

Then, the filesystems from /etc/fstab are mounted. Network interfaces are addressed. Finally, services and daemons are started using the startup scripts located in /etc/rc.d and /usr/local/etc/rc.d, with rcorder(8) making certain that services are started in the correct order.

nextboot(8)

FreeBSD comes with the nextboot(8) feature that allows you to specify alternate kernel and boot flags for the next reboot cycle. But only once,for the very next boot process. 

This is achieved by loader(8) preferring to load kernel information from the /boot/nextboot.conf file if present. Once the machine is rebooted, nextboot.conf is automatically removed, reverting the system to its previous ‘permanent’ configuration.

For ZFS filesystems, nextboot stores metadata in a designated place in the ZFS pool label, since the early boot code can’t handle writing directly to the pool. The nextboot metadata is erased as soon as it is read, so it only impacts the next single boot after it’s been written.

To boot the CUSTOM FreeBSD kernel once, issue the following nextboot(8) command:

# nextboot -k CUSTOM

To force the Single User Mode along with booting the TEST kernel do this:

# nextboot -o "-s" -k TEST

You can also remove any existing nextboot(8) configuration with this option:

# nextboot -D

The bootme/bootonce/bootfailed Flags

FreeBSD implements several extra boot flags. These are bootmebootonce, and bootfailed.  The bootme flag is used to specify which partition should be used for the boot process. The bootonce flag is used with bootme flag to boot the specified partition once only and then fail back to what was configured permanently. The FreeBSD loader(8) achieves that by removing these flags.The last flag—bootfailed— is used to show that the partition failed to boot. You can use the gpart(8) tool to manipulate these flags as needed. For example:

# gpart set -a bootme -i 2 ada0

These flags are very useful if you have more than one UFS root partition with some other version of the FreeBSD system. You can then set the bootme flag to boot from that partition instead.

Example Commands to Manipulate the Boot Process

The boot code at MBR can be changed/updated with these commands:

# fdisk -B
# fdisk -B -b /boot/boot0 ada0
# boot0cfg -B
# boot0cfg -s 1 -b /boot/boot0 ada0

Installation process for the zfsboot on MBR slice.

# gpart create -s mbr ada0
# gpart add -t freebsd ada0
# gpart bootcode -b /boot/boot0 ada0
# gpart set -a active -i 1 ada0
# dd if=/dev/zero of=/dev/ada0s1 count=2
# dd if=/boot/zfsboot of=/dev/ada0s1 count=1
# dd if=/boot/zfsboot of=/dev/ada0s1 iseek=1 oseek=1024

Similar commands for GPT below:

UFS # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0
ZFS # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

Creating UEFI ESP partition manually.

# gpart add -a 4K -t efi -s 200M ada0
# newfs_msdos /dev/ada0s1
# mount_msdosfs /dev/ada0s1 /mnt
# mkdir -p /mnt/efi/boot
# cp /boot/boot1.efi /mnt/efi/boot/bootx64.efi

Additional Resources

Here are some interesting FreeBSD resources on the boot process topic that you may also find useful:

·  FreeBSD Architecture Handbook – Chapter 1 – Bootstrapping and Kernel Initialization

·  FreeBSD Handbook – Chapter 13 – FreeBSD Booting Process

·  boot(8)

·  boot.config(5)

·  boot0cfg(8)

·  boot1.efi(8)

·  efibootmgr(8)

·  efivar(8)

·  fdisk(8)

·  gptboot(8)

·  gptzfsboot(8)

·  zfsboot(8)

·  zfsbootcfg(8)

·  loader(8)

·  loader.efi(8)

·  nextboot(8)

·  fastboot(8)

Like this article? Share it!

You might also be interested in

Getting expert FreeBSD advice is as easy as reaching out to us!

At Klara, we have an entire team dedicated to helping you with your FreeBSD projects. Whether you’re planning a FreeBSD project, or are in the middle of one and need a bit of extra insight, we’re here to help!

More on this topic

Contributing to Open Source Beyond Software Development

Open source projects thrive because of the community built around them. However, non-coding contributions are frequently overlooked and under appreciated.

Let’s examine how non-developer contributors enhance user experience, improve bug reporting, and influence feature requests, all while becoming advocates and evangelists for your open source project.

FreeBSD and OpenZFS

Part 2: Tuning Your FreeBSD Configuration for Your NAS

Building your own NAS isn’t just about having the right storage configuration. It starts with the right hardware, the right OS setup, and finally going through the right choice for your storage – OpenZFS. In this edition of our 4-part article series on how to build your own NAS we discuss about fine tuning your FreeBSD OS for excellent NAS performance.

Improving Replication Security With OpenZFS Delegation

OpenZFS privilege delegation is an extremely powerful tool that enables system administrators to carefully provide unprivileged users the ability to manage ZFS datasets and zvols at an extremely precise level —with much finer control than would be possible with generic security tools like sudo or doas.

Tell us what you think!