Controlling Resource Limits with rctl in FreeBSD

One of the leading tyre manufacturers says that "Power is nothing without control." ... and they are right. FreeBSD’s slogan is "The Power to Serve"—but FreeBSD also comes with mature and well thought-out limits that you can use to control many aspects of this stable operating system.

Starting with the Basics

The tool (and service) to configure and enforce resource limits is the rctl(8) command and the /etc/rc.d/rctl service. The service(8) itself is very simple and just applies resource limits that are configured in the rctl.conf(5) file - which is located in /etc/rctl.conf.

You can use rctl to manage consumption limits on many system resources, including but not limited to CPU utilization, process and shared memory, and total number of threads or processes.

For filesystem limits, you can throttle the reads and writes in bytes per second or in operations per second (IOPS). All these options give a very large spectrum of possibilities on what you can do with resource limits on FreeBSD.

Below is the complete list of all the resources rctl controls:

       RESOURCE  [UNIT]     DESCRIPTION                                                  cputime  (s)        CPU time
       datasize  (bytes)    data size
      stacksize  (bytes)    stack size
   coredumpsize  (bytes)    core dump size
      memoryuse  (bytes)    resident set size
   memorylocked  (bytes)    locked memory
        maxproc  (count)    number of processes
      openfiles  (count)    file descriptor table size
     vmemoryuse  (bytes)    address space limit
pseudoterminals  (count)    number of PTYs
        swapuse  (bytes)    swap space that may be reserved/used
           nthr  (count)    number of threads
     msgqqueued  (count)    number of queued SysV messages
       msgqsize  (bytes)    SysV message queue size
          nmsgq  (count)    number of SysV message queues
           nsem  (count)    number of SysV semaphores
         nsemop  (count)    number of SysV semaphores modified in single semop(2)           nshm  (count)    number of SysV shared memory segments
        shmsize  (bytes)    SysV shared memory size
      wallclock  (s)        wallclock time
           pcpu  (count%)   %CPU in percents of single CPU core
        readbps  (bytes/s)  filesystem reads
       writebps  (bytes/s)  filesystem writes
       readiops  (ops/s)    filesystem reads
      writeiops  (ops/s)    filesystem writes

Now that you know which resources of what subjects you can control – it's time to define what actions you can engage on them.

   ACTION  DESCRIPTION                                          
     deny  deny allocation - not supported for:
           cputime/wallclock/readbps/writebps/readiops/writeiops
     log   log warning to console
     devctl  send notification to devd(8) using:
           - system = "RCTL"
           - subsystem = "rule"
           - type = "matched"
     signame  send signal (sigterm/sighup) to offending process
           see signal(3) man page for list of all 33 signals
     throttle   slow down process execution - only supported for:
           readbps/writebps/readiops/writeiops

Each rctl(8) rule has the following syntax:

subject:subject-id:resource:action=value

The syntax is the same whether invoked from the command line using the rctl(8) command, or in the rctl.conf(5) file.

Enable and Reboot

The default GENERIC FreeBSD kernel comes with support for the rctl(8) resource limits compiled in—but although available, it is disabled by default.

% grep -e RCTL -e RACCT /usr/src/sys/amd64/conf/GENERIC
options         RACCT                       # Resource accounting framework
options         RACCT_DEFAULT_TO_DISABLED   # Set kern.racct.enable=0 by default
options         RCTL                        # Resource limits

To enable it you need to add kern.racct.enable=1 to the /boot/loader.conf file, then reboot your system for changes to take effect.

# cat << BSD >> /boot/loader.conf

# ENABLE RCTL/RACCT RESOURCE LIMITS
  kern.racct.enable=1

BSD

# grep -i -C 1 RACCT /boot/loader.conf

# ENABLE RCTL/RACCT RESOURCE LIMITS
  kern.racct.enable=1

If you forget to enable rctl in loader.conf, you will see the message below:

# rctl
RACCT/RCTL present, but disabled; enable using kern.racct.enable=1 tunable

If you’re using a custom-compiled FreeBSD kernel built without the RACCT and RCTL options, you will see the following message instead:

# rctl
RACCT/RCTL support not present in kernel; see rctl(8) for details

If you see this message, you will need to switch to a kernel which supports rctl—or go back into your own custom kernel source, enable those options, then rebuilt it again.

After booting a kernel with rctl support and kern.racct.enable=1 line in /boot/loader.conf, the rctl(8) command will work fine.

If you do not have any rctl rules defined, you will get an empty output as shown below:

# rctl
#

For security’s sake, it’sa good idea to restrict permissions on /etc/rctl.conf by setting chmod 600 /etc/rctl.conf. This keeps unprivileged users from snooping on—or altering!—the limits you’ve set.

Now we have working Resource Limits on FreeBSD.

Manage Rules

In this section, we will see how to add/remove and list the rules. Here is a brief description of the rctl(8) flags:

  -a rule    adds rule
  -l filter  filter rules
     filter  list rules
  -r filter  remove rules matching filter
  -u filter  display utilization matching filter rule
  -h         show human readable output
  -n         displays user IDs numerically

Keep in mind that if you add a new rule using rctl(8)on the command line, it will not be automatically written to /etc/rctl.conf. To make that new rule permanent, you need to add it manually:

# rctl -a 'user:hast:maxproc:deny=100/user'

# rctl | grep user:hast:maxproc
user:hast:maxproc:deny=100

# rctl user:hast
user:hast:maxproc:deny=100

# grep user:hast:maxproc /etc/rctl.conf
(no match/no output)

# echo 'user:hast:maxproc:deny=100/user' >> /etc/rctl.conf

# grep user:hast:maxproc /etc/rctl.conf
user:hast:maxproc:deny=100/user

When listing and/or filtering rules, keep in mind that it is sometimes better to just display all rules and grep(1) for needed information, because the filter syntax for matching rules is quite strict.

# rctl -l user:hast
rctl: failed to get limits for 'user:tests': Operation not supported

# rctl user:tests
user:tests:maxproc:deny=10

# rctl user
rctl: malformed rule 'user': missing subject

# rctl | grep user
user:tests:maxproc:deny=10
user:pulse:maxproc:deny=20

To remove rules you need to use the -r flag as specified in the rctl(8) man page—but if you would like to remove all rules for a specified user then you can use the following command.

# rctl -r user:username

Now let's assume we have configured some rules in /etc/rctl.conf,and loaded them up using the command service rctl start. Here is an example rctl ruleset:

# cat /etc/rctl.conf | sort
jail:klara:maxproc:deny=100
jail:klara:memoryuse:deny=1G
jail:klara:pcpu:deny=10
jail:klara:vmemoryuse:deny=1G
jail:syncthing:maxproc:deny=100
jail:syncthing:memoryuse:deny=1G
jail:syncthing:pcpu:deny=25
jail:syncthing:vmemoryuse:deny=1G
jail:websrv:maxproc:deny=100
jail:websrv:memoryuse:deny=512M
jail:websrv:openfiles:deny=1024
jail:websrv:pcpu:deny=50
user:hast:maxproc:deny=100/user
user:pulse:maxproc:deny=20/user
user:tests:maxproc:deny=10/user

# rctl | sort
jail:klara:maxproc:deny=100
jail:klara:memoryuse:deny=1073741824
jail:klara:pcpu:deny=10
jail:klara:vmemoryuse:deny=1073741824
jail:syncthing:maxproc:deny=100
jail:syncthing:memoryuse:deny=1073741824
jail:syncthing:pcpu:deny=25
jail:syncthing:vmemoryuse:deny=1073741824
jail:websrv:maxproc:deny=100
jail:websrv:memoryuse:deny=536870912
jail:websrv:openfiles:deny=1024
jail:websrv:pcpu:deny=50
user:hast:maxproc:deny=100
user:pulse:maxproc:deny=20
user:tests:maxproc:deny=10

It may be handy to use human readable flag -h to make the rctl(8) output look more like the format seen in /etc/rctl.conf:

# rctl -h | sort
jail:klara:maxproc:deny=100
jail:klara:memoryuse:deny=1024M
jail:klara:pcpu:deny=10
jail:klara:vmemoryuse:deny=1024M
jail:syncthing:maxproc:deny=100
jail:syncthing:memoryuse:deny=1024M
jail:syncthing:pcpu:deny=25
jail:syncthing:vmemoryuse:deny=1024M
jail:websrv:maxproc:deny=100
jail:websrv:memoryuse:deny=512M
jail:websrv:openfiles:deny=1024
jail:websrv:pcpu:deny=50
user:hast:maxproc:deny=100
user:pulse:maxproc:deny=20
user:tests:maxproc:deny=10

You may also parse these rules with the rctl(8) command like in the examples below:

# rctl user:
user:tests:maxproc:deny=10
user:hast:maxproc:deny=100
user:pulse:maxproc:deny=20

# rctl jail:klara
jail:klara:pcpu:deny=10
jail:klara:vmemoryuse:deny=1073741824
jail:klara:memoryuse:deny=1073741824
jail:klara:maxproc:deny=100

# rctl -h jail:klara
jail:klara:pcpu:deny=10
jail:klara:vmemoryuse:deny=1024M
jail:klara:memoryuse:deny=1024M
jail:klara:maxproc:deny=100

Keep in mind to add a colon after the subject—if you forget the colon, rctl will fail to properly filter the rules.

# rctl -h jail
rctl: malformed rule 'jail': missing subject

# rctl -h jail:
jail:klara:pcpu:deny=10
jail:klara:vmemoryuse:deny=1024M
jail:klara:memoryuse:deny=1024M
jail:klara:maxproc:deny=100
jail:syncthing:pcpu:deny=25
jail:syncthing:vmemoryuse:deny=1024M
jail:syncthing:memoryuse:deny=1024M
jail:syncthing:maxproc:deny=100
jail:websrv:memoryuse:deny=512M
jail:websrv:maxproc:deny=100
jail:websrv:openfiles:deny=1024
jail:websrv:pcpu:deny=50

FreeBSD Jails and rctl

FreeBSD Jails can also be contained within required limits. For example, you can limit a specified Jail to only use 80% of a single CPU core and 1 GB RAM with the following rules:

# rctl -a jail:JAILNAME:pcpu:deny=80
# rctl -a jail:JAILNAME:memoryuse:deny=1G

You may also view a Jail’s current resource usage with the following command:

# service jail onestart klara
Starting jails: klara.

# jls
   JID  IP Address      Hostname                      Path
     1  10.0.0.200      klara                         /jail/klara

# rctl -u jail:klara
cputime=0
datasize=741376
stacksize=2113536
coredumpsize=0
memoryuse=19050496
memorylocked=0
maxproc=4
openfiles=160
vmemoryuse=63598592
pseudoterminals=0
swapuse=0
nthr=4
msgqqueued=0
msgqsize=0
nmsgq=0
nsem=0
nsemop=0
nshm=0
shmsize=0
wallclock=170
pcpu=0
readbps=0
writebps=0
readiops=0
writeiops=0

Once again, you may find using the human readable flag (-h) useful:

# rctl -h -u jail:klara
cputime=0
datasize=724K
stacksize=2064K
coredumpsize=0
memoryuse=18M
memorylocked=0
maxproc=4
openfiles=160
vmemoryuse=61M
pseudoterminals=0
swapuse=0
nthr=4
msgqqueued=0
msgqsize=0
nmsgq=0
nsem=0
nsemop=0
nshm=0
shmsize=0
wallclock=287
pcpu=0
readbps=0
writebps=0
readiops=0
writeiops=0

Knowing what a jail’s resource consumption looks like allows us to configure limits for it more intelligently.

Additional Resources

Here are FreeBSD resources about the Resource Limits that you may find also useful.

• FreeBSD Handbook - 14. Security - 14.13. Resource Limits
• FreeBSD Man Page - rctl(4)
• FreeBSD Man Page - rctl(8)
• FreeBSD Man Page - rctl.conf(5)