Your Comprehensive Guide to rc(8): FreeBSD Services and Automation

FreeBSD has a very stable and well thought out init and services system called rc(8). 

According to the FreeBSD man pages, the rc(8) utility first appeared in the 4.0BSD release on 1980. That was 42 years ago. It has of course been modernized since then—But before we get into that, let’s first take a quick overview of its history.

The History Of rc(8)

The classic boot process was simple—the BSD kernel started init(8), which then passed control to the /etc/rc script before starting a getty(8) process to manage each virtual console. The /etc/rc script then called /etc/netstart to get network configuration up and running, and later /etc/rc called /etc/rc.local to start daemons that were not a part of the base system.

In 2000, NetBSD began modernizing this simple /etc/rc subsystem. First, NetBSD introduced an /etc/rc.d directory to contain separate scripts for each service. Second, they introduced the rcorder(8) command to determine the order in which these services would be started at system boot.

FreeBSD imported these improvements three years later, in FreeBSD 5.0-RELEASE, with improvements and bugfixes continuing to be added ever since. 

Although the rc.d system may not seem very polished by modern standards, it’s simple and rugged, unlikely to crash absent a serious sysadmin error or filesystem corruption. Although there are arguments to be made for more “modern” init systems like systemd(1), I prefer and recommend the stability, simplicity, and readability of BSD’s more traditional init system.

This article will try to bring you up to speed on FreeBSD services management, and offer options for automation that you may find useful in your cloud-like environments.

FreeBSD Services

FreeBSD services that utilize the rc(8) init system are based on the POSIX-compliant /bin/sh (Bourne) shell. Below, you can see a very simple start/stop script example:

#!/bin/sh
#
# PROVIDE: myutility
# REQUIRE: DAEMON sshd netif
# KEYWORD: shutdown

. /etc/rc.subr

name=myutility
rcvar=myutility_enable

command="/usr/local/sbin/myutility"

load_rc_config $name

#
# DO NOT CHANGE THESE DEFAULT VALUES HERE
# SET THEM IN THE /etc/rc.conf FILE
#
myutility_enable=${myutility_enable-"NO"}
pidfile=${myutility_pidfile-"/var/run/myutility.pid"}

run_rc_command "$1"

This start/stop script allows the FreeBSD init system to properly manage a “myutility” application installed by the system administrator. The path to the app’s binary is /usr/local/sbin/myutility, its “friendly” name within rc is “myutility,” and we can enable it in /etc/rc.conf by adding the line “myutility_enable=YES”.

Enabling this app by adding myutility_enable="YES" to /etc/rc.conf causes this service to be started when the system enters 'multiuser' mode. (It would not be started when the system is booted to single-user mode, which is akin to “safe mode” in other operating systems.)

This myutility service will start after the netif and sshd services are already started. The order that FreeBSD system services will be started in is determined by the rcorder(8) command mentioned earlier. As you can see in the script above, each service script also includes the /etc/rc.subr file, which contains functions (subroutines) commonly used by system service shell scripts.

FreeBSD will start the services in the order determined by rcorder(8). This is one of the rc(8) init system’s strengths—it’s simple to determine and configure the order in which services start, whereas Linux’s more complex systemd(1) init may boot a bit faster, but be considerably more difficult to troubleshoot.

The rcorder(8) command takes a list of the startup scripts as an argument. On a typical FreeBSD system there are two paths that are used for these startup scripts—the FreeBSD base system path at /etc/rc.d, and the Third Party Applications path at /usr/local/etc/rc.d. You may 'query' their order separately or all at once – it's up to you. 

Let’s take a look at some examples below. In these examples, we use the 2> /dev/null redirection to keep the output clean—without that redirection, we’d also see some debug noise.

# rcorder /etc/rc.d/* 2> /dev/null | head
/etc/rc.d/sysctl
/etc/rc.d/natd
/etc/rc.d/dhclient
/etc/rc.d/dumpon
/etc/rc.d/hostid
/etc/rc.d/ddb
/etc/rc.d/ccd
/etc/rc.d/gbde
/etc/rc.d/geli
/etc/rc.d/swap

# rcorder /usr/local//etc/rc.d/* 2> /dev/null | head
/usr/local//etc/rc.d/xdm
/usr/local//etc/rc.d/wireguard
/usr/local//etc/rc.d/webcamd
/usr/local//etc/rc.d/virtual_oss
/usr/local//etc/rc.d/vboxwebsrv
/usr/local//etc/rc.d/vboxwatchdog
/usr/local//etc/rc.d/vboxnet
/usr/local//etc/rc.d/dbus
/usr/local//etc/rc.d/rsyncd
/usr/local//etc/rc.d/powerdxx

# rcorder /etc/rc.d/* /usr/local/etc/rc.d/* 2> /dev/null | head -50
/etc/rc.d/sysctl
/etc/rc.d/dhclient
/etc/rc.d/natd
/etc/rc.d/dumpon
/etc/rc.d/ddb
/etc/rc.d/hostid
/etc/rc.d/geli
/etc/rc.d/ccd
/etc/rc.d/gbde
/etc/rc.d/swap
/etc/rc.d/zpool
/etc/rc.d/zfskeys
/etc/rc.d/fsck
/etc/rc.d/growfs
/etc/rc.d/zvol
/etc/rc.d/root
/etc/rc.d/sppp
/etc/rc.d/hostid_save
/etc/rc.d/mdconfig
/etc/rc.d/serial
/etc/rc.d/mountcritlocal
/etc/rc.d/zfsbe
/etc/rc.d/tmp
/etc/rc.d/zfs
/etc/rc.d/var
/etc/rc.d/cfumass
/etc/rc.d/cleanvar
/etc/rc.d/FILESYSTEMS
/etc/rc.d/iovctl
/etc/rc.d/ipsec
/usr/local/etc/rc.d/vboxnet
/etc/rc.d/ugidfw
/etc/rc.d/autounmountd
/etc/rc.d/ldconfig
/etc/rc.d/kldxref
/etc/rc.d/adjkerntz
/etc/rc.d/mixer
/etc/rc.d/hostname
/etc/rc.d/ip6addrctl
/etc/rc.d/ippool
/etc/rc.d/rctl
/etc/rc.d/netoptions
/etc/rc.d/opensm
/etc/rc.d/random
/etc/rc.d/geli2
/etc/rc.d/kld
/usr/local/etc/rc.d/uuidd
/etc/rc.d/ipfilter
/etc/rc.d/ipnat
/etc/rc.d/addswap

As you probably guessed, the FreeBSD Base System services are usually started before any Third Party Applications are. This ensures that the base system itself is up and responsive as soon as possible. Starting the base system in full first also keeps third party apps’ startup scripts shorter and simpler, since they don’t need to manually specify dependency services included and started in the base system itself.

When rcorder(8) is executed against Third Party Applications only, it may not show their 'real' startup order—this is because they need the base system’s foundation before they can start.

Enabling and Disabling Services

There are many ways in which you can enable or disable a certain service. The most obvious way is to just open rc.conf in your favorite  text ${EDITOR} and add myservice_enable=YES to enable automatic startup of the myservice service at system startup. While obvious , this simple method is also somewhat error prone. If you accidentally introduce syntax errors into /etc/rc.conf, such as one too many or too few quote marks, you will render your FreeBSD system unbootable.

There are some ways to check your /etc/rc.conf edits afterwards. The most simple and obvious is using the /bin/sh interpreter to check it. Let’s see what happens if we introduce a syntax error into rc.conf, then attempt to run it manually:

# sh /etc/rc.conf
/etc/rc.conf: YES: not found

# grep ntpd /etc/rc.conf
ntpd_enable= YES

As you can see above, there is 'unsupported' whitespace in the line enabling the NTPd daemon. Since rc.conf is a simple /bin/sh  script, it must adhere to the Bourne shell’s simple rules for variable assignment. Manually running rc.conf with /bin/sh ourselves allowed us to find and resolve the error now, rather than when the system refuses to boot later! 

Let’s look at another syntax error commonly introduced when hand editing the /etc/rc.conf file:

# sh /etc/rc.conf
/etc/rc.conf: 94: Syntax error: Unterminated quoted string

# cat -n /etc/rc.conf | grep -C 1 94
    93
    94      ntpd_enable=YES"
    95

As we can see above, line 94 of /etc/rc.conf attempts to close quotation marks it never opened.

Let's check one more typical syntax error:

# sh /etc/rc.conf
/etc/rc.conf: ntpd_enable: not found

# cat -n /etc/rc.conf | grep -C 1 94
    93
    94      ntpd_enable =YES
    95  

Once again, we’ve been bitten by unsupported whitespace after the  ntpd_enable keyword. 

Now, let’s talk about how we can avoid all of these syntax errors by using service(8) or  sysrc(8) to introduce changes to rc.conf, rather than hand-editing them in.

On FreeBSD the service(8) command is most commonly used to start and stop services, but it can also permanently—and syntax-safely— enable and/or disable those services in rc.conf, using its enable and disable  arguments.  Here are some examples:

To enable the ntpd(8) service:

# service ntpd enable   
ntpd enabled in /etc/rc.conf

# tail -1 /etc/rc.conf
ntpd_enable="YES"

To disable the ntpd(8) service:

# service ntpd disable
ntpd disabled in /etc/rc.conf

# tail -1 /etc/rc.conf     
ntpd_enable="NO"

The sysrc(8) command is another way to safely edit service definitions in rc.conf. Many services have additional options beyond just being enabled, for example you can specify a different configuration file that should be used:

Configuring the ntpd(8) service with sysrc(8):

# sysrc ntpd_config="/etc/my_ntp.conf"
ntpd_config: /etc/ntp.conf -> /etc/my_ntp.conf

# tail -1 /etc/rc.conf
ntpd_config="/etc/my_ntp.conf"

Some applications run services which must be added to the /etc/rc.conf before starting the application. These applications usually install their own startup script in the /usr/local/etc/rc.d directory, which defines these dependencies and ensures that they’re met prior to application start. See Starting Services for more information. 

Now that FreeBSD includes rc.d, configuration of application startup is easier and provides more features. Using the keywords discussed in Managing Services in FreeBSD, applications can be set to start after certain other services and extra flags can be passed through /etc/rc.conf in place of hard coded flags in the startup script. 

In the next several sections, we’ll look at modern options for starting FreeBSD applications.

(Extremely) Oldschool Services

Some services don’t manage their own internet connections, relying instead on the  so called 'Super Server,' inetd(8). 

Inetd isn’t something most admins will encounter frequently, and if you’re feeling overwhelmed, you can safely ignore it. For those who’d like to fall down the rabbit hole—or find ways to avoid having to write network code themselves!—we have you covered; all the details you’ll need are in our article Modern inetd in FreeBSD.

Using cron(8) to Start Services

Some services only need to run intermittently, rather than all the time. This is where cron(8) comes handy. This approach has many advantages, since the cron(8) daemon runs processes as the owner of the crontab(5) file used to start them. 

Most people think of cron as a way to run jobs at a given hour of the day, but remember—cron also supports a @reboot target, which may be used to run the specified job when the cron(8) daemon itself starts—typically, during the FreeBSD system boot process itself. 

When running cron(8) scripts, keep in mind that they are executed in a much more limited environment than interactive user logins typically enjoy. You can make sure that the limited cron environment is sufficient for your job to execute as shown below: 

# env -i SHELL=/bin/sh PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
HOME=/home/klara LOGNAME=klara /home/klara/bin/yourscript.sh

This env command replicates the environment a cron script (and its launched processes) started on behalf of the klara user would experience. If yourscript.sh does not work properly when started this way, it won’t work properly when started from cron either—with the most common problem being the much shorter PATH environment variable, and the most common fix being hardcoding full paths into yourscript.sh itself. 

Force Service to Start

Sometimes you want to start a service, even when it isn’t enabled on the system itself. There’s nothing wrong with that approach for a one-off test, as long as you know what you’re doing.  

To start a disabled service, you must add a  one  prefix to the usual start or stop commands. Here is an example: 

# service ntpd start
Cannot 'start' ntpd. Set ntpd_enable to YES in /etc/rc.conf or use 'onestart' instead of 'start'.

# service ntpd onestart
Starting ntpd. 

Although we’re not actually using sudo, the effect is similar to the classic 'sudo make me a sandwitch' XKCD comic. Specifying onestart instead of simply start tells the system to go ahead and start the service this time, whether it normally would or not. 

If you ever forget how to manually start a service, you don’t need to guess—the presence of the rcvar variable in an rc(8) script’s comments lets you know which lines would work to start or stop it: 

# service sshd rcvar
# sshd : Secure Shell Daemon
#
sshd_enable="YES"
#   (default: "")

# /etc/rc.d/sshd rcvar
# sshd : Secure Shell Daemon
#
sshd_enable="YES"
#   (default: "")

In the above example, we can see that sshd may be started with either service sshd start or /etc/rc.d/sshd start,  and stopped by using the argument stop instead of start. 

There is no effective difference between either method from a system administrator’s point of view: service myservice start and /etc/rc.d/myservice start accomplish the same thing.

Loading Kernel Modules

Historically, all FreeBSD kernel modules needed to be loaded by loader(8) before the system booted, in the following syntax as shown below:

# grep load /boot/loader.conf
  geom_eli_load=YES
  zfs_load=YES
  aio_load=YES

Loading all needed modules this way is simple, but it comes at a price—FreeBSD system boot time is MUCH slower if all modules are loaded this way, as it forces them to load sequentially rather than in parallel, and the loader’s implementation of reading from the filesystem is much more naive. 

It’s better to only load FreeBSD kernel modules that are required during early boot (storage and filesystem drivers) using the /boot/loader.conf file, and load user-installed kernel modules during the later phase of boot using the kld_list stanza in /etc/rc.conf.  

This way, your boot process still loads all necessary kernel modules—but system boot time improves, since kld_list can load them in parallel after the base system is already up, rather than one by one as it comes online 

We can see some examples of dynamically loaded kernel modules using kld_list  in /etc/rc.conf below: 

# grep kld /etc/rc.conf
  kld_list="${kld_list} /boot/modules/i915kms.ko"
  kld_list="${kld_list} fusefs coretemp sem cpuctl ichsmb cuse linux linux64"
  kld_list="${kld_list} libiconv cd9660_iconv msdosfs_iconv udf_iconv"

You’ll still get kernel messages during the boot sequence showing that your kernel modules were loaded; they modules—and their status messages—simply fire up after the base system itself is up and running.

Autoloading module: i915kms
Autoloading module: fusefs
Autoloading module: coretemp
Autoloading module: sem
Autoloading module: cpuctl
(...)

Automation for the Cloud

Besides configuring your FreeBSD systems to enable/disable kernel modules and system services, there are automation tools which can make your life easier. 

FreeBSD comes with many *.d directories, in both in the Base System /etc prefix and the /usr/local/etc  prefix used for third party applications.  

Here are the directories we are talking about: 

# find /etc -name \*\.d -maxdepth 1
/etc/cron.d
/etc/jail.conf.d
/etc/newsyslog.conf.d
/etc/pam.d
/etc/rc.conf.d
/etc/rc.d
/etc/syslog.d

# find /usr/local/etc -name \*\.d -maxdepth 1
/usr/local/etc/bash_completion.d
/usr/local/etc/libmap.d
/usr/local/etc/logrotate.d
/usr/local/etc/man.d
/usr/local/etc/newsyslog.conf.d
/usr/local/etc/pam.din
/usr/local/etc/profile.d
/usr/local/etc/rc.d
/usr/local/etc/smartd_warning.d
/usr/local/etc/sudoers.d
/usr/local/etc/syslog.d

These directories allow you to insert small fragments of configuration that get included as if all of the lines were present in the main configuration file. This makes it simple for packages to automatically enable log rotation by installing a file in /usr/local/etc/newsyslog.conf.d/ rather than needing to programmatically edit your main /etc/newsyslog.conf file. It also makes it easier for automation tools like cloudinit or ansible to install different sudo rules by places files in the sudoers.d directory, rather than trying to template all of the different permutations into a single file. 

There is one caveat with the rc.conf.d/ directories, rather than reading every one of the fragments each time a service is started, only the fragment with the name that matches the service is read. So when doing service sshd start only rc.conf and /etc/rc.conf.d/sshd is read, while all other rc.conf.d/ files are ignored. 

Ansible Orchestration

You can also use Ansible to enable/disable FreeBSD services with its community.general.sysrc module. Here are some examples:

# ADD gif0 INTERFACE TO THE cloned_interfaces
- name: add gif0 interface
  community.general.sysrc:
    name: cloned_interfaces
    state: value_present
    value: "gif0"


# ENABLE nginx SERVICE IN www JAIL
- name: enable nginx in www jail
  community.general.sysrc:
    name: nginx_enable
    value: "YES"
    jail: www

The sysrc module makes it easy to have ansible safely edit your services configuration for you, rather than needing to install various config fragment, or manually parse and modify the configuration file. 

There are several other Ansible modules that may be valuable when configuring a system, including the beadm and pkgng modules.

Conclusion

FreeBSD’s rc system provides a simple yet powerful way to control, manage, and configure the services that are started when the system boots. Keeping the service management system simple enough that it is easy to reason about and extend, but powerful enough that it supports modern automation techniques and doesn’t hamper your deployments. This provides a simple way to manage the entire system via a single interface.