Customizing the FreeBSD Kernel

In this article, we will be looking at the various ways to customize the build of the FreeBSD kernel and its loadable modules. Understanding this process is invaluable for making custom additions or tuning the kernel build for a specific piece of hardware.

What we'll cover in this article:

1. Kernel config file format and configuration
2. Kernel module Makefiles
3. Building out-of-tree modules

Readers will benefit from existing knowledge of the kernel build targets, i.e. make buildkernel and make installkernel.

A final note before we begin: we do not recommend stripping "unnecessary" drivers out of the kernel in most environments. In nearly any general-purpose environment, there are no tangible performance benefits from doing so, and a significant chance of causing future problems in the event of hardware replacement.

If you're building a very specific, tightly controlled hardware environment with high security requirements, it might be worth stripping drivers to theoretically reduce attack surface—but if you're not absolutely certain of what you're doing, it's best to leave the default driver environment intact.

   | For those interested in exploring deployment options for ZFS on Linux, check out our guide on deploying ZFS on Linux.

Kernel Configuration

FreeBSD provides a simple and readable configuration file format for the build of its kernel. The kernel configuration file allows you to:

• Set/unset kernel options, e.g. SMP support
• Include/exclude individual device drivers
• Include other config files
• Set make(1) variables for the kernel build

First and foremost, every kernel config must have a name, which is set in an ident stanza. It is recommended, but not required, that ident be configured at or very near the top of the file:

$ cat MYCONFIG
#
# MYCONFIG - custom kernel config
#

ident MYCONFIG

Options and device drivers can be controlled using the [no]options and [no]device keywords respectively. Consider the following config fragment:

...

options       DDB               # Enable the ddb(4) kernel debugger
device        nvme              # Include base NVME driver
nooptions     COMPAT_FREEBSD32  # Disable 32-bit compatibility shims

The first line specifies that we would like to compile in the kernel debugger ddb(4)—an option which is excluded from normal release builds of FreeBSD. The second line indicates that we wish to include the nvme(4) driver in our build of the kernel (changing device to nodevice would instead exclude it). The last line disables inclusion of 32-bit compatibility interfaces.

Due to the number of config lines required to build a minimal bootable kernel, trying to write a kernel config from scratch is typically a bad idea.

It may be tempting to start a new kernel config by making a copy of the existing GENERIC kernel and editing it, but this makes it difficult to focus on your own changes, and can lead to pain in the future when the default kernel config changes, and you must rebase to match.

For these reasons, the best practice is to include an existing config, such as GENERIC, and add overrides. This keeps the focus on your modifications to that base config, and reduces the pain and potential errors which might otherwise be introduced in a major rebase (or failure to rebase) later.

#
# MYKERNEL config file
#

include GENERIC
ident   MYKERNEL

nodevice isa                    # Don't need to support an ISA bus
nodevice cd                     # Don't need CD-ROM SCSI driver

This works as you might intuitively expect: config lines that follow the include directive override settings specified by the included file. So, ident MYKERNEL takes precedence over ident GENERIC, and nodevice isa wins out over device isa.

By using include, one can also ensure that future additions to GENERIC will automatically be picked up by a custom config. This makes porting across FreeBSD versions or tracking CURRENT less error-prone. This also provides a way to manage similar but distinct configs which share a common set of override options.

Finally, the makeoptions directive allows setting make(1) variables for the specific kernel build. A common example is setting DEBUG=-g, to generate debugging output for the object files.

Make Variables

Some useful make(1) variables that influence the kernel build process:

• INSTKERNNAME - the installed kernel name, which defaults to “kernel”, informing the install path $DESTDIR/boot/$INSTKERNNAME/.
• KERNCONF - set this to specify which kernel config(s) should be built. More than one value can be supplied, separated by spaces.
• KERNCONFDIR - this sets the directory in which kernel config files will be looked for. It may make sense to keep your custom config files outside of the FreeBSD source repository.

Building Kernel Modules

Drivers and functionality which will not be required in the main kernel configuration can still be provided in the form of loadable kernel modules.

By default, the set of supported kernel modules on a given platform are also compiled as part of buildkernel, but they can be compiled separately as well. The logic for building kernel modules lives in the sys/modules/ directory of the source tree, with each module having its own subdirectory.

Below is an example of a real module Makefile from the FreeBSD source tree, for if_bridge(4).

$ cat sys/modules/if_bridge/Makefile
# $FreeBSD$

.PATH:	${SRCTOP}/sys/net
KMOD=	if_bridge
SRCS=	if_bridge.c opt_inet.h opt_inet6.h opt_carp.h

.include <bsd.kmod.mk>

As you can see, this is a relatively simple module with a single C source file. Most of the Makefile “magic” is defined in the included file, bsd.kmod.mk. This provides standard targets, allowing you to build and install if_bridge(4) on its own.

Modules built in this standalone way will be installed to $KMODDIR, which is set to /boot/modules by default.

$ cd sys/modules/if_bridge/
$ make  
... build output ...
$ make install
install -T release -o root -g wheel -m 555   if_bridge.ko /boot/modules/
kldxref /boot/modules

Make Variables

Some useful make(1) variables that influence the build of kernel modules:

•	MODULES_OVERRIDE - a list of modules to build, instead of them all
•	WITHOUT_MODULES - a list of modules to exclude from the build
•	NO_MODULES - set to skip building modules altogether
•	KMODDIR - default standalone module install path

Out-of-tree Modules

If you wanted to add some custom functionality to the FreeBSD kernel in the form of a module, but wanted to keep it separate from the main source tree, what is the best way to do it? This section will explore how to set up and build out-of-tree kernel modules.

One of the niceties of the BSD make infrastructure is the ability to quickly write Makefiles without requiring in-depth knowledge of make(1). Just as the kernel modules in the source tree are able to include  bsd.kmod.mk, any FreeBSD install will provide a global set of make(1) includes under the /usr/share/mk directory.

This means that building a program, library, or kmod really only requires setting a few variables. Let us consider an example kernel module, aptly named example_kmod:

$ cd ~/work/example_kmod
$ ls
example_kmod.c
example_kmod_acpi.c
Makefile

So, we have a separate directory, containing a couple of source files and a Makefile. Let's look at its contents:

KMOD=    example_kmod

SRCS=    opt_acpi.h
SRCS+=   example_kmod.c            # Main kmod functionality
SRCS+=   example_kmod_acpi.c       # ACPI attachment logic

.include <bsd.kmod.mk>

This Makefile alone is enough to make && make install, the same as if it were part of the main source tree.

If one wanted to manage multiple custom modules under one repository separate from the main FreeBSD source tree, it is as simple as creating a top-level Makefile:

# Top-level Makefile. Builds all specified subdirectories.

SUBDIR+=    example_kmod
SUBDIR+=    custom_device_kmod

.include <bsd.subdir.mk>

For a file structure such as this, you can simply navigate into the repo directory, and a single make will build all your modules:

/usr/home/mhorne/repo/
  example_kmod/
    example_kmod.c
    example_kmod_acpi.c
    Makefile
  custom_device_kmod/
    custom_device.c
    Makefile
  Makefile

This is easy enough, but it would be better if these subdirectories could be included as part of the buildworld/installworld process. For this, there are the $LOCAL_MODULES and $LOCAL_MODULES_DIR variables.

These variables allow for kmod source files located under $LOCAL_MODULES_DIR to be automatically picked up as part of the make buildkernel process. $LOCAL_MODULES will automatically be set to include all $LOCAL_MODULES_DIR subdirectories, unless it is manually overridden.

Note: make sure to also specify WITH_LOCAL_MODULES=yes, since that is not a default when cross-compiling!

This system is used automatically if you install the FreeBSD drm-kmod graphics drivers package on your laptop, to automatically rebuild the module to keep it in sync with your kernel.

Putting it all together

As a final example, let’s consider how we can set up a kernel and module build with a custom config and a set of out-of-tree modules.

Assuming our src tree is located at $HOME/work/freebsd-src, and out-of-tree modules at $HOME/work/repo as in the previous section. The MYKERNEL config is located in this directory as well, keeping FreeBSD's own /src pristine.

Further Reading and Examples

Hopefully, this has been helpful in demonstrating some of the flexibility of the FreeBSD build system, and how it can be leveraged by projects that need to manage their own additions or customizations to the kernel.

The kernel configuration file format is described in detail by the config(5) man page. There are also several kernel configs in the /src tree, along with the various NOTES files. These describe all of the possible config options, as well as dependencies between options and/or devices.

systat will provide information for network interfaces (via ifstat) and for network protocols (providing icmp, icmp6, ip, ip6, sctp and tcp).

Finally, there are other useful make(1) variables that influence the kernel build process. They are documented in a few places, the useful man pages being make(1), build(7), make.conf(5), and src.conf(5).