kernel development

Customizing the FreeBSD Kernel

Customizing the FreeBSD Kernel

Learn more about customizing the build of the FreeBSD kernel and its loadable modules

In this article, we will be looking at the various ways to customize the build of the FreeBSD kernel and its loadable modules. Understanding this process is invaluable for making custom additions or tuning the kernel build for a specific piece of hardware.

What we’ll cover in this article:

  1. Kernel config file format and configuration
  2. Kernel module Makefiles
  3. Building out-of-tree modules

Readers will benefit from existing knowledge of the kernel build targets, i.e. make buildkernel and make installkernel.

Kernel Configuration

FreeBSD provides a simple and readable configuration file format for the build of its kernel. This section will walk through the basics of this format.

A kernel configuration file allows you to:

  • Set/unset kernel options, e.g. SMP support
  • Include/exclude individual device drivers
  • Include other config files
  • Set make(1) variables for the kernel build

First and foremost, every kernel config must have a name. It is recommended, but not required, that this be placed near the top.

$ cat MYCONFIG
#
# MYCONFIG - custom kernel config
#

ident MYCONFIG

Options and device drivers can be controlled using the [no]options and [no]device keywords respectively. Consider the following config fragment:

...

options       DDB               # Enable the ddb(4) kernel debugger
device        nvme              # Include base NVME driver
nooptions     COMPAT_FREEBSD32  # Disable 32-bit compatibility shims

The first line specifies that we would like to compile in the kernel debugger ddb(4); something normal excluded from release builds of FreeBSD. The second indicates that we wish to include the nvme(4) driver in our build of the kernel. The last line disables inclusion of 32-bit compatibility interfaces, useful when this compatibility is known not to be required. Attentive readers should also be able to see how device drivers can be explicitly excluded from a kernel config.

Due to the number of config lines required to build a minimal bootable kernel, it is not recommended to write a kernel config from scratch. It may be tempting to start by copying the existing GENERIC kernel, but this can lead to pain in the future. Instead, the best practice is to include an existing config, such as GENERIC, and make modifications.

#
# MYKERNEL config file
#

include GENERIC
ident   MYKERNEL

nodevice isa                    # Don't need to support an ISA bus
nodevice cd                     # Don't need CD-ROM SCSI driver

This works as you might intuitively expect: config lines that follow the include directive override settings specified by the included file. So, ident MYKERNEL gets precedence over ident GENERIC, and nodevice isa wins out over device isa.

By using include, one can also ensure that future additions to GENERIC will automatically be picked up by a custom config. This makes porting across FreeBSD versions or tracking CURRENT less error-prone. It also provides a way to manage similar but distinct configs, that share a common base set of options.

Finally, the makeoptions directive allows setting make(1) variables for the specific kernel build. A common example is setting DEBUG=-g, to generate debugging output for the object files.

Make Variables

Some useful make(1) variables that influence the kernel build process:

  • INSTKERNNAME – the installed kernel name, which defaults to “kernel”, informing the install path $DESTDIR/boot/$INSTKERNNAME/.
  • KERNCONF – set this to specify which kernel config(s) should be built. More than one value can be supplied, separated by spaces.
  • KERNCONFDIR – directory in which kernel config files will be looked for. It may make sense to keep your custom config files outside of the FreeBSD source repository.

Building Kernel Modules

For drivers and functionality not required to be part of the main kernel configuration, they can still be provided in the form of loadable kernel modules.

By default, the set of supported kernel modules on a given platform are also compiled as part of buildkernel, but they can be compiled separately as well. The logic for building kernel modules lives in the sys/modules/ directory of the source tree, with each module having its own subdirectory.

Below is an example of a real module Makefile from the FreeBSD source tree, for if_bridge(4).

$ cat sys/modules/if_bridge/Makefile
# $FreeBSD$

.PATH:	${SRCTOP}/sys/net
KMOD=	if_bridge
SRCS=	if_bridge.c opt_inet.h opt_inet6.h opt_carp.h

.include <bsd.kmod.mk>

As you can see, this is a relatively simple module with a single C source file. Most of the Makefile “magic” is defined in the included file, bsd.kmod.mk. This provides standard targets, allowing you to build and install if_bridge(4) on its own. Modules that are built in this standalone way will be installed to $KMODDIR, /boot/modules by default.

$ cd sys/modules/if_bridge/
$ make  
... build output ...
$ make install
install -T release -o root -g wheel -m 555   if_bridge.ko /boot/modules/
kldxref /boot/modules

Make Variables

Some useful make(1) variables that influence the build of kernel modules:

•	MODULES_OVERRIDE - a list of modules to build, instead of them all
•	WITHOUT_MODULES - a list of modules to exclude from the build
•	NO_MODULES - set to skip building modules altogether
•	KMODDIR - default standalone module install path

Out-of-tree Modules

If you wanted to add some custom functionality to the FreeBSD kernel in the form of a module, but wanted to keep it separate from the main source tree, what is the best way to do it? This section will explore how to set up and build out-of-tree kernel modules.

One of the niceties of the BSD make infrastructure is the ability to quickly write Makefiles without requiring in-depth knowledge of make(1). Just as the kernel modules in the source tree are able to include bsd.kmod.mk, any FreeBSD install will provide a global set of make(1) includes, under the /usr/share/mk directory. What this means is that building a program, library, or kmod, really only requires setting a few variables.

Let us consider an example kernel module, aptly named example_kmod.

$ cd ~/work/example_kmod
$ ls
example_kmod.c
example_kmod_acpi.c
Makefile

So, we have a separate directory, containing a couple of source files and a Makefile. Let us look at the Makefile contents:

KMOD=    example_kmod

SRCS=    opt_acpi.h
SRCS+=   example_kmod.c            # Main kmod functionality
SRCS+=   example_kmod_acpi.c       # ACPI attachment logic

.include <bsd.kmod.mk>

This Makefile alone is enough to make && make install, the same as if it were part of the main source tree.If one had multiple custom modules they wanted to manage under one repository, separate from the main FreeBSD source tree, it is as simple as creating a top-level Makefile, like the following:

# Top-level Makefile. Builds all specified subdirectories.

SUBDIR+=    example_kmod
SUBDIR+=    custom_device_kmod

.include <bsd.subdir.mk>

For a file structure such as this:

/usr/home/mhorne/repo/
  example_kmod/
    example_kmod.c
    example_kmod_acpi.c
    Makefile
  custom_device_kmod/
    custom_device.c
    Makefile
  Makefile

You could navigate into the repo directory, and simple execute make to build all of your modules.

This is easy enough, but it would be better if these subdirectories could be included as part of the buildworld/installworld process. For this, there are the $LOCAL_MODULES and $LOCAL_MODULES_DIR variables. These variables allow for kmod source files located under the local modules directory, defined by $LOCAL_MODULES_DIR to be automatically picked up as part of the make buildkernel process. $LOCAL_MODULES will automatically be set to all $LOCAL_MODULES_DIR subdirectories, unless it is manually overridden.

Note: make sure to also specify WITH_LOCAL_MODULES=yes, as it is not a default when cross-compiling.

This system is used automatically if you install the FreeBSD drm-kmod graphics drivers package on your laptop, to automatically rebuild the module to keep it in sync with your kernel.

[user@freebsd-client] $ iperf3 -c 10.0.4.2 -u -b 1G
Connecting to host 10.0.4.2, port 5201
[  5] local 10.0.4.1 port 16765 connected to 10.0.4.2 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   119 MBytes  1000 Mbits/sec  85586  
[  5]   1.00-2.00   sec   119 MBytes  1000 Mbits/sec  85587  
[  5]   2.00-3.00   sec   119 MBytes  1.00 Gbits/sec  85642  
[  5]   3.00-4.00   sec   119 MBytes  1000 Mbits/sec  85608  
[  5]   4.00-5.00   sec   119 MBytes  1.00 Gbits/sec  85626  
[  5]   5.00-6.00   sec   119 MBytes  1000 Mbits/sec  85590  
[  5]   6.00-7.00   sec   119 MBytes  1000 Mbits/sec  85607  
[  5]   7.00-8.00   sec   119 MBytes  1.00 Gbits/sec  85621  
[  5]   8.00-9.00   sec   119 MBytes  1000 Mbits/sec  85610  
[  5]   9.00-10.00  sec   119 MBytes  1000 Mbits/sec  85725  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.16 GBytes  1000 Mbits/sec  0.000 ms  0/856202 (0%)  sender
[  5]   0.00-10.10  sec  1.01 GBytes   859 Mbits/sec  0.003 ms  113542/856202 (13%)  receiver

iperf Done.

Putting it all together

As a final example, let’s consider how we can set up a kernel and module build with a custom config and a set of out-of-tree modules.

Assuming our src tree located at $HOME/work/freebsd-src, and out-of-tree modules at $HOME/work/repo as in the previous section. The MYKERNEL config is located in this directory as well, keeping the FreeBSD src pristine.

Further Reading and Examples

Hopefully, this has been helpful in demonstrating some of the flexibility of the FreeBSD build system, and how it can be leveraged by projects that need to manage their own additions or customizations to the kernel.

The kernel configuration file format is described in detail by the config(5) man page. There are also several kernel configs in the src tree, along with the various NOTES files. These describe all of the possible config options, as well as dependencies between options and/or devices.

systat will provide information for network interfaces (via ifstat) and for network protocols (providing icmp, icmp6, ip, ip6, sctp and tcp).

Finally, there are other useful make(1) variables that influence the kernel build process. They are documented in a few places, the useful man pages being make(1), build(7), make.conf(5), and src.conf(5).

Like this article? Share it!

You might also be interested in

Get more out of your FreeBSD development

Kernel development is crucial to many companies. If you have a FreeBSD implementation or you’re looking at scoping out work for the future, our team can help you further enable your efforts.

More on this topic

Performance observability

FreeBSD Performance Observability

Performance observability is a powerful feature that highly supports FreeBSD. In this article, we’re showing you how to take advantage of tools that are specifically built for and with an operating system: tools which understand and are built into the operating system’s kernel structures. Learn about how to gather the information you need in order to get the most out of your system, determine your operational baselines, and find and resolve performance bottlenecks.

DTrace Network Probes

DTrace landed in FreeBSD 7.1 in 2009. DTrace is an observability framework that enables dynamic tracing of programs, and the FreeBSD Kernel. It offers an incredible view into the operation of programs, and is an excellent tool for debugging and performing analysis of complex software. Using the full power of FDT probes requires familiar knowledge with how things are implemented. SDT probes and specific providers like ip, tcp and udp offer insights into the internal behavior of the kernel without requiring a high level of knowledge of the underlying code.

Building Customized FreeBSD Images

Did you know that Poudriere is not only a package building tool for FreeBSD, but it can also generate FreeBSD images? You can also use NanoBSD or, perhaps, assemble a bootable FreeBSD image by hand. This article will explore some of the ideas and tools required to start down the path of building customized FreeBSD images, and by the end you should have the knowledge required to continue on your own.

One Comment on “Customizing the FreeBSD Kernel

  1. Pingback: Valuable News – 2021/03/15 | 𝚟𝚎𝚛𝚖𝚊𝚍𝚎𝚗

Tell us what you think!