Building Customized FreeBSD Images

FreeBSD does a great job of being self-contained and easy to build, requiring only a few simple make commands to compile the system from scratch. But when it comes to configuring and installing the system in a repeatable way, the path is not so straightforward.

Perhaps you are looking to build a pre-configured FreeBSD installation for your router, or a deeply customized image with packages and all, to be flashed to a fleet of embedded devices. So, how could this be done? This article will explore some of the ideas and tools required to start down this path, and by the end you should have the knowledge required to continue on your own.

Definition: for the purposes of this article, an “image” is a file containing the partitions and contents of a complete and bootable FreeBSD system, such that it can be written to a disk without modification. Examples include the USB memstick or VM images available on FreeBSD’s download page.

Basic Bootable Images

This section will show briefly how to assemble a bootable FreeBSD image – including a UFS root filesystem, swap partition, and UEFI bootloader – by hand. It is useful to understand the basics of this procedure to grasp how the later stages work.

The examples that follow assume the amd64 architecture, but most of what is presented here applies equally to other FreeBSD platforms.

Root filesystem

The root filesystem typically contains the majority of files required for a functioning FreeBSD system. This includes the kernel, system daemons and utilities, and configuration files.

Note: the following commands require superuser privileges. It is possible to generate an image without this, but the process is slightly different and not covered here.

This example begins by assuming you have completed a build of FreeBSD by running make buildworld and make buildkernel. With this done, we will begin by installing this system to a temporary directory:

$ export ROOTFSDIR=$HOME/rootfs
$ cd $HOME/freebsd-src
$ make DESTDIR=$ROOTFSDIR installworld
$ make DESTDIR=$ROOTFSDIR distribution
$ make DESTDIR=$ROOTFSDIR installkernel

Did you know?

65% of businesses running on FreeBSD have less incidents and faster to go-to-market of their products with professional support!

Our FreeBSD Support Offering

If you are unfamiliar with the distribution target and how it differs from installworld, it simply instructs the build system to install the default configuration files, such as the contents of /etc, to the target directory. Running this target on an existing FreeBSD system would overwrite the existing configuration, so it is not done as part of a normal upgrade procedure.⁽¹⁾

You should now have $ROOTFSDIR populated with a clean install of FreeBSD:

$ ls -la $ROOTFSDIR
total 103
drwxr-xr-x  18 root      wheel       22 Apr  9 03:36 .
drwxr-xr-x  46 root      wheel      102 Apr 28 11:21 ..
-rw-r--r--   2 root      wheel     1023 Apr  9 03:17 .cshrc
-rw-r--r--   2 root      wheel      507 Apr  9 03:17 .profile
drwxr-xr-x   2 root      wheel       46 Apr  9 03:17 bin
drwxr-xr-x  14 root      wheel       65 Apr  9 03:38 boot
-r--r--r--   1 root      wheel     6109 Apr  9 03:26 COPYRIGHT
dr-xr-xr-x   2 root      wheel        2 Apr  9 03:09 dev
drwxr-xr-x  27 root      wheel      103 Apr  9 03:26 etc
drwxr-xr-x   5 root      wheel       67 Apr  9 03:20 lib
drwxr-xr-x   3 root      wheel        5 Apr  9 03:17 libexec
drwxr-xr-x   2 root      wheel        2 Apr  9 03:09 media
drwxr-xr-x   2 root      wheel        2 Apr  9 03:09 mnt
drwxr-xr-x   2 root      wheel        2 Apr  9 03:09 net
dr-xr-xr-x   2 root      wheel        2 Apr  9 03:09 proc
drwxr-xr-x   2 root      wheel      150 Apr  9 03:19 rescue
drwxr-x---   2 root      wheel        7 Apr  9 03:26 root
drwxr-xr-x   2 root      wheel      137 Apr  9 03:24 sbin
lrwxr-xr-x   1 root      wheel       11 Apr  9 03:09 sys -> usr/src/sys       
drwxrwxrwt   2 root      wheel        2 Apr  9 03:09 tmp
drwxr-xr-x  14 root      wheel       14 Apr  9 03:36 usr
drwxr-xr-x  24 root      wheel       24 Apr  9 03:09 var

Looks good. Let’s give the machine a hostname, and generate an fstab(5):

$ echo "hostname=selfbuilt" > $ROOTFSDIR/etc/rc.conf
$ echo "/dev/gpt/rootfs  /         ufs     rw,noatime 1 1" > $ROOTFSDIR/etc/fstab
$ echo "/dev/gpt/esp     /boot/efi msdosfs rw,noatime 0 0" >> $ROOTFSDIR/etc/fstab
$ echo "/dev/gpt/swapfs  none      swap    sw         0 0" >> $ROOTFSDIR/etc/fstab

The exact details of this are somewhat specific to the example, but hopefully it is obvious that we intend to create and mount three different partitions: the UFS root filesystem, the EFI System Partition (ESP) and a swap partition.

At this point, any other customizations you’d like to make to the system can be performed on the contents of $ROOTFSDIR. This could be as simple as enabling some services in /etc/rc.conf, installing the ports tree to $ROOTFSDIR/usr/ports, or adding your ssh(1) public key to the known_hosts file. For more complex cases, you can chroot into the system to interact with it, e.g. to use adduser(8). Finally, you might choose to bootstrap pkg(7) and install some desired packages with pkg --chroot.⁽²⁾

Now that the system is set up with any desired tweaks, let’s create the UFS filesystem from the temporary directory. The preferred way to do this is with makefs(8), as follows:

$ makefs -B little \
    -o label=rootfs -o version=2 -o softupdates=1 \
    -D -s 10g \
    rootfs.ufs $ROOTFSDIR

This command creates a new file rootfs.ufs, containing a UFS partition made up of the contents of $ROOTFSDIR. This is the first piece of our final bootable system image.

Bootloader Partition

Next, we want to construct the ESP. This is a small, FAT-formatted partition containing a copy of FreeBSD’s EFI loader(8). This partition will enable the image to be booted from any firmware that supports UEFI. For the sake of brevity, legacy BIOS-based boot won’t be considered by this example.

The following will construct an ESP; again using makefs:

$ export EFIDIR=$HOME/efistage
$ mkdir -p $EFIDIR/efi/boot
$ cp $ROOTFSDIR/boot/loader.efi $EFIDIR/efi/boot/bootx64.efi
$ makefs -t msdos \
    -o fat_type=32 -o sectors_per_cluster=1 -o volume_label=EFISYS \
    -s 50m \
    efi.part $EFIDIR

The resulting file efi.part will be used in the next step.

Final Image

Now that we have constructed a root filesystem and an ESP, it’s time to put them together into the final image. For this, we will use mkimg(8). Our image will use a GPT partition scheme.

$ mkimg -s gpt -f raw \
    -p efi/esp:=efi.part \
    -p freebsd-swap/swapfs::2G \
    -p freebsd-ufs/rootfs:=rootfs.ufs \
    -o selfbuilt.img

This creates the final selfbuilt.img, containing a GPT partition table and our three partitions. It can be written to a disk or USB and booted on a real PC, or passed to a virtual machine.

Tools

Building images by hand as we have just seen can be useful for one’s understanding, but this isn’t something you want to repeat every time. However, these commands can easily be encapsulated and automated by a scripting language. Many of the official FreeBSD release images are produced using a set of Makefiles and shell scripts with steps very similar to what was presented.⁽³⁾

In this section, we will look at two existing tools that wrap this functionality and make customizing images easier. One is a well-tested set of scripts available in the FreeBSD source tree, and the other an in-development extension to FreeBSD’s favorite package building tool.

NanoBSD

nanobsd(8) is a set of shell scripts and configuration files available in the FreeBSD src tree, which provide a framework for building customized images. NanoBSD is intended specifically for creating FreeBSD-based appliances, and has been present in FreeBSD since version 6.0. A notable consumer of NanoBSD is the BSD Router Project.

It is worth noting that NanoBSD images are structured somewhat differently than a typical FreeBSD installation, as the root filesystem is read-only. This is intentional, the idea being that this helps reduce wear on persistent storage, since NanoBSD-based appliances are expected to be long-lasting installations that run 24/7 (or close to it). Configuration changes must be written to a special /cfg partition to persist across reboots. If these design decisions don’t fit your use-case, then NanoBSD might not be the right choice.

NanoBSD configuration files are shell scripts, containing variables that influence its build. The nanobsd(8) man page contains descriptions of these variables. A basic NanoBSD configuration file looks something like this:

#
# Custom config for a nanobsd appliance
#
NANO_NAME=nanobsd_custom          # Image name
NANO_KERNEL=GENERIC               # name of desired FreeBSD kernel config
NANO_ARCH=amd64
NANO_SRC=$HOME/freebsd-src        # path to FreeBSD src tree

# Set src.conf(5) options for the build and install
CONF_WORLD="
WITHOUT_CLANG=yes
WITHOUT_FTP=yes
WITHOUT_ZFS=yes
"

# shell functions can be registered to customize the image after install
cust_setup_rcconf() {
    sysrc -R ${NANO_WORLDDIR} sendmail_disable="YES"
    sysrc -R ${NANO_WORLDDIR} ifconfig_default="10.0.0.101/24"
    sysrc -R ${NANO_WORLDDIR} powerd_enable="YES"
}
customize_cmd cust_setup_rcconf

There are a couple of predefined custom commands, notably cust_install_files, which will include everything in the /tools/tools/nanobsd/Files directory in the final image.

NanoBSD configuration files can be made to do pretty much anything, but require some investment from the user. There are several example config files available under tools/tools/nanobsd/, which are worth some study while you develop your own. If shell scripting isn’t your thing, consider that the custom commands allow you to execute whatever you like. You could do the bulk of your customizations with a python script, for example.

Once your configuration file is set up, it is a simple matter to build it:

$ cd tools/tools/nanobsd
$ sh nanobsd.sh -c nanobsd_custom.cfg

The script will run the usual build and install steps, run the customization commands, and assemble the image file. This file will be available under $NANO_OBJ/nanobsd.$NANO_NAME, with the name $NANO_IMGNAME.img. In this case, that’s /usr/obj/nanobsd.nanobsd_custom/_disk.full.

Image Building with Poudriere

poudriere(8) is a versatile and well-known package building tool for FreeBSD. However, it is capable of more than just packages, including building the FreeBSD base system, and even generating FreeBSD images! In this section, we will take a brief look at how to get started with this feature.

Warning: at the time of writing, image building is still considered an alpha feature of poudriere, despite continuous development since 2015. As such, the interface is subject to change, and there may be bugs present. Regardless, this feature is functional and will eventually become stable, so forward-looking users can still perform fruitful experiments with this tool in its current state.

The full set of options is described by the poudriere-image(8) man page, which is short and worth skimming through before proceeding. Obviously, you’ll need the poudriere port/package installed for the steps that follow.

There are several different types, or flavors, of images that can be built by poudriere, selected with the -t argument. The examples that follow will create a usb type image, which has a layout resembling our hand-built image from earlier. There are also supported images with memory-backed filesystems, u-boot ready embedded images, raw UFS partitions, and more.

Before starting anything else, we must create a jail and ports tree:

$ poudriere jail -c -j 13_0_amd64 -K GENERIC -m ftp -v 13.0-RELEASE
$ poudriere ports -c -p latest -m git -B main

This creates a jail named 13_0_amd64 and ports tree named latest. The -m argument can be changed if you want to build/install from an alternate source, e.g. an existing customized src or ports tree. Important here is to specify -K <kernelconf>, which instructs the jail to include a kernel. This is required for the final image, which won’t be run as a jail.

To construct a basic image without further customizations, all that is needed is the command:

$ poudriere image -j 13_0_amd64 -p latest \
    -n poudriere_custom -h poudriere_custom \
    -s 10g -t usb

This will create poudriere_custom.img in the current directory, and this file should be immediately bootable.

One of the most compelling features of poudriere-image is easy integration with custom-built package sets. Let’s take a file with a list of desired ports:

$ cat pkglist.txt
editors/vim-tiny
shells/bash
sysutils/tmux
www/nginx

Using poudriere, we can compile this list of ports with whatever options or customizations we like, and then include them in the final image:

$ # Build the packages
$ poudriere bulk -j 13_0_amd64 -p latest -f pkglist.txt
$ # Generate the image again, this time with packages included
$ poudriere image -j 13_0_amd64 -p latest \
    -n poudriere_custom -h poudriere_custom \
    -s 10g -t usb \
    -f pkglist.txt

This is great, now the only thing missing is a way to make configuration changes to the system. This can be done in two ways.

First, the jail can start (poudriere jail -s), and you can log in to make changes or additions interactively. Once the jail is stopped (poudriere jail -k), these changes will be persistent. Alternatively, poudriere image allows inclusion of an ‘overlay’ directory in the final image by use of the -c argument. For example, one might add a customized /usr/local/etc/nginx/nginx.conf file to this overlay directory, providing the desired configuration for the webserver out of the box.

Together, these two functions provide everything needed to make any final customizations before generating and deploying your image.

Closing and Further Reading

Hopefully, this introduction to building custom FreeBSD images has been informative. Each of the methods presented here have their own strengths and weaknesses, so the best one to proceed with is up to you.

NanoBSD offers a powerful system for configuration, at the cost of some complexity, and rigidity in its use-case. Poudriere offers a more pleasant experience on the commandline, and integration with its package-building features, but it is not yet production-tested software. Building images by hand can be tedious and error-prone, but it is fundamental to the image-building tools of the future.

Clearly, this is only the tip of the iceberg of what can be done with these methods. Further tricks or customizations will depend on your use-case and your creativity. As a couple of final resources, you might look at this article on NanoBSD, or the BSD Router Project’s write-up about poudriere-image.

____________________________________

⁽¹⁾ Updating configuration files against an existing system is handled by etcupdate(8).

⁽²⁾ If you get No address record errors with pkg --chroot, you can copy the host system’s /etc/resolv.conf to the chroot as a quick workaround.

⁽³⁾ These are found in the release directory of the FreeBSD source tree.