Explaining top(1) on FreeBSD

We all know and have at least once used the top(1) command to track information about our cpu and processes, but how many of you know what each field means? Today we will guide you through each of these fields. By default, top(1) displays the 'top' processes on each system and periodically updates this information every 2.0 seconds using the raw cpu use percentage to rank the processes in the list.

Default top(1) Output

This is how to default top(1) command output looks. We will use it as a base to describe each field and column.

last pid: 95139;  load averages:  0.21,  0.25,  0.25     up 17+06:17:12  21:01:34
22 processes:  1 running, 21 sleeping
CPU:  0.1% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.9% idle
Mem: 5968K Active, 285M Inact, 2918M Wired, 584K Buf, 552M Free
ARC: 101M Total, 32M MFU, 24M MRU, 392K Anon, 1286K Header, 44M Other
     12M Compressed, 52M Uncompressed, 4.46:1 Ratio
Swap: 1024M Total, 1024M Free
 
  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
95139 klarabsd      1  23    0    14M  3988K CPU3     3   0:01   0.42% top
91467 klarabsd      1  20    0    21M  9536K select   1   0:00   0.04% sshd
79666 root          1  20    0    13M  3024K select   2   5:05   0.02% mountd
66928 root          1  20    0    13M  2312K select   2   4:32   0.02% powerd
86590 ntpd          1  20    0    21M  5452K select   2   2:46   0.01% ntpd
88853 root          1  21    0   173M   146M select   1   2:29   0.00% smbd
62176 root          1  20    0    13M  2812K select   1   2:16   0.00% syslogd
88570 root          1  20    0    39M    16M select   0   0:41   0.00% nmbd
91145 root          1  20    0    13M  2728K nanslp   3   0:26   0.00% cron
80942 root          4  52    0    12M  2300K rpcsvc   0   0:05   0.00% nfscbd
83679 root         32  52    0    12M  2788K rpcsvc   1   0:05   0.00% nfsd
77923 root          1  20    0    13M  2680K select   1   0:04   0.00% rpcbind
90750 root          1  20    0    20M  8268K select   1   0:04   0.00% sshd
93654 root          2  20    0   136M   104M select   0   0:03   0.00% smbd
93831 root          1  20    0   131M   104M select   0   0:03   0.00% smbd
45020 root          1  20    0    11M  1488K select   0   0:01   0.00% devd
91568 klarabsd      1  29    0    15M  6196K pause    2   0:00   0.00% zsh
91342 root          1  41    0    21M  9508K select   3   0:00   0.00% sshd
  312 klarabsd      3  20    0    31M    12M select   0   0:00   0.00% mocp
83547 root          1  52    0    12M  3036K select   2   0:00   0.00% nfsd
 7478 root          1  52    0    12M  2196K pause    1   0:00   0.00% adjkerntz
80887 root          1  52    0    12M  2252K accept   0   0:00   0.00% nfscbd

PID/Load/Uptime/Time

The last pid: shows the PID of the last (most recent) process, which is 95139 in our example.

The load averages: field displays the load average of the system over the last 1, 5, and 15 minutes respectively. The topic of load average is generally very broad and it is calculated differently on various UNIX systems, so a separate article may be dedicated for just that. The up field shows the system uptime which is 17 days, (+) six hours, seventeen minutes and twelve seconds 06:17:12. The last field on the top right is the current time 21:01:34 displayed in HH:MM:SS format.

Processes States

We have now covered the first line of output from top(1). Time for the second line. This one contains information about the current count of processes and their state. On this system, there are 22 processes with 1 running and 21 sleeping processes.

CPU Use

The third line shows a summary breakdown of what is currently running on the system CPUs. The user field shows the usage of userspace processes. The nice field shows how much CPU time is taken by processes with different nice(1) priorities. The system field shows how much CPU time is used by the FreeBSD kernel. The interrupt field shows how much CPU time is delegated to interrupt support. Last but not least the idle field shows how much CPU time is unused or free.

Memory

The fourth line of the top(1) command marked Mem: shows a detailed summary of the memory information. The following physical memory stats contain this information:

Active shows the number of bytes used by actively running processes. Inact shows the number of bytes not recently used by a process (while the process itself may still remain active - just some parts of its memory has not been used recently). If the system needs more memory, then this inactive memory may be swapped out or freed to make room. The Laundry queue shows the amount of memory that is “dirty”, needing to be laundering before it can be reused. Data that has been modified since it was read from disk, or has not yet been stored on disk, cannot be easily recreated, so it must be written to swap or the modified file before that memory can be reused. For more details about how the inactive memory and the laundry system work, see Exploring Swap on FreeBSD . The Wired count shows the number of bytes wired down which, unlike the other types, CANNOT be swapped out. Buf shows the number of bytes used for buffers and caches. Instead of being swapped out, they will be just freed. Free shows the free memory.

To get the general idea of how much system memory is actually used versus free, here is a hint. The used memory is generally the sum of Active and Wired fields. As Inact/Laundry/Buf fields can be swapped out, they should all be treated as 'buffers' or 'cached' information in memory that can (and will) be freed when needed. FreeBSD, as any other clever UNIX system, follows the motto of 'Unused memory is wasted memory.' so when the memory is not needed, FreeBSD keeps it in case an inactive process will try to access it again.

ZFS ARC

The fifth line includes ZFS Adaptive Replacement Cache or, in short, ARC. This is the amount or RAM that ZFS uses as cache. We may select what should be the minimum and maximum ARC size in the /etc/sysctl.conf file with vfs.zfs.arc_min and vfs.zfs.arc_max parameters.

The ZFS ARC stats fields show the following information.

The Total field shows the number of wired bytes used for the ZFS ARC. The MRU field shows the number of ARC bytes holding Most Recently Used data. The MFU field shows the number of ARC bytes holding Most Frequently Used data. The Anon field shows the number of ARC bytes holding in flight data that does not fit into other categories. The Header field shows the number of ARC bytes holding headers. The Other field shows other miscellaneous ARC bytes needed for ZFS to operate properly.

The sixth line is a continuation of the ZFS ARC stats but, as ARC is compressed on FreeBSD, it shows stats of how much memory has been saved thanks to using the compression algorithm on all that ZFS cache. The Compressed field shows bytes of memory used by ARC caches. From all the memory used by ARC, the 12M(megabytes) is compressed and thanks to 4.46:1 compress Ratio displayed later on the right, it uses 'only' 12M instead of 52M (megabytes) shown in the Uncompressed field that will be used without compression.

SWAP

The remaining Swap: field on the seventh line, as you probably guessed, shows the SWAP information. Here, it is really simple - it just shows Total size of SWAP area and how much of it is actually Free.

The second part of the top(1) command is the list of processes. These are the displayed columns.

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND

This is the part where information about individual processes is displayed. The PID is the process id. The USERNAME shows the name of the process's owner. The PRI column shows the current priority of the process while NICE is the amount of nice(1) a process has. SIZE shows the text/data/stack of process. RES is the current amount of resident memory while SWAP is the amount of swap used - both in kilobytes. STATE is the current state of the process - it can START/SLEEP/STOP/ZOMB/WAIT/LOCK. When a process is waiting on something to happen, it will display the ‘waitchan’, describing what it is waiting for. There are many possible values for this, we might have to do an entire article on some of the common values. Keep in mind that RUN is shown as CPUn (for example, CPU3) on systems with multiple CPU cores. The C is the logical processor on which the process is currently running - this is visible only on SMP systems. The TIME column shows the number of seconds that the process used. The WCPU shows the weighted cpu percentage. The COMMAND shows the name of the command that the process is currently running.

Cores Monitoring

To just check which CPU is doing what, we can use the command below to display each CPU current usage stats. This is how it's going to look:

% top -P 0

### Output

last pid: 23091;  load averages:  0.66,  0.50,   up 0+01:12:58 912:44:49
107 processes: 2 running, 104 sleeping, 1 waiting
CPU 0:  1.6% user,  0.0% nice,  0.0% system,  0.0% interrupt, 98.4% idle
CPU 1:  2.4% user,  0.0% nice,  0.4% system,  0.0% interrupt, 97.3% idle
CPU 2:  0.0% user,  0.0% nice,  0.0% system,  1.2% interrupt, 98.8% idle
CPU 3:  1.1% user,  0.0% nice,  1.5% system,  0.0% interrupt, 97.3% idle
CPU 4:  0.4% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.2% idle
CPU 5:  2.7% user,  0.0% nice,  1.2% system,  0.0% interrupt, 96.1% idle
CPU 6:  0.4% user,  0.0% nice,  0.8% system,  0.0% interrupt, 98.8% idle
CPU 7:  1.2% user,  0.0% nice,  0.0% system,  0.0% interrupt, 98.8% idle
Mem: 1650M Active, 2263M Inact, 278M Laundry, 2623M Wired, 24G Free
ARC: 354M Total, 109M MFU, 92M MRU, 1508K Anon, 3657K Header, 147M Other
     87M Compressed, 313M Uncompressed, 3.58:1 Ratio
Swap: 1024M Total, 1024M Free

Interactive Mode Options

To avoid citing the whole top(1) man page, we will describe the 'most interesting' switches that we can toggle in the interactive (default) mode. No matter to what delay (with letter s) the refresh is set, we can always refresh the screen manually using the [CTRL]+[L] shortcut. We can also refresh the stats using the biggest 'key' on the keyboard - the more convenient [SPACEBAR]. The letter m will toggle between the 'cpu' mode (which is the default) and the 'io' mode. Just type  'cpu' or 'io' after pressing the m key. The letter S will toggle the display of system processes - which is very useful. We can also kill(1) a process using the letterk or renice(1) one with the letter r. For sorting the display of processes, use the letter o and then use one of these columns for the sorting order: cpu/res/size/time - the default is cpu. Using the capital P letter, we can switch to display per CPU stats instead of summary percentages of CPU usage. With capital J we can display processes owned only by a specific jail. The letter i can toggle displaying the idle processes.

The I/O Mode

While the default top(1) mode is CPU related, the top(1) on FreeBSD also has another mode - the I/O mode which is very useful in tracking processes that do much I/O. To start the top(1) in the I/O mode along with sensible sorting on the TOTAL column, use this command:

# top -S -m io -o total

The command above also includes the -S option to show the system processes because when we start the ZFS scrub process, it will not show as any separate process. However, with the -S option, we will see high I/O activity of the zfskern system process - this is where ZFS scrub does its work.

This is how the top(1) I/O mode looks like on FreeBSD.

last pid:  9572;  load averages:  0.40,  0.20,  0.17       up 17+21:37:30  12:21:52
75 processes:  2 running, 72 sleeping, 1 waiting
CPU:  0.9% user,  0.0% nice,  8.6% system,  3.4% interrupt, 87.1% idle
Mem: 32M Active, 298M Inact, 2922M Wired, 584K Buf, 512M Free
ARC: 191M Total, 58M MFU, 64M MRU, 7201K Anon, 3024K Header, 60M Other
     84M Compressed, 1179M Uncompressed, 14.01:1 Ratio
Swap: 1024M Total, 1024M Free
 
  PID USERNAME     VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT COMMAND
  806 root        1745     78      0    465      0    465  81.72% pkg
   19 root         104     26      0     38      0     38   6.68% zfskern
 9298 klarabsd      37     12     37      0      0     37   6.50% rsync
 1303 klarabsd      29      1     29      0      0     29   5.10% dd
    2 root           0      0      0      0      0      0   0.00% KTLS
    3 root           0      0      0      0      0      0   0.00% crypto
77923 root           0      0      0      0      0      0   0.00% rpcbind
64387 klarabsd       4      0      0      0      0      0   0.00% sshd
    4 root           0      0      0      0      0      0   0.00% crypto returns 0
 9572 klarabsd       6      0      0      0      0      0   0.00% rsync
    5 root           0      0      0      0      0      0   0.00% crypto returns 1
    6 root           0      0      0      0      0      0   0.00% crypto returns 2
68006 root           0      0      0      0      0      0   0.00% sshd
    7 root           0      0      0      0      0      0   0.00% crypto returns 3
93831 root           1      0      0      0      0      0   0.00% smbd
    8 root         147     54      0      0      0      0   0.00% cam
    9 root           0      0      0      0      0      0   0.00% soaiod1
91145 root           0      0      0      0      0      0   0.00% cron
   10 root           0      0      0      0      0      0   0.00% audit
   11 root        4429   4798      0      0      0      0   0.00% idle
71883 klarabsd       4      0      0      0      0      0   0.00% sshd
  427 root           0      0      0      0      0      0   0.00% pkg
68331 klarabsd       0      0      0      0      0      0   0.00% zsh

It has slightly different columns than the default CPU mode top(1) command. These are the displayed columns in the IO mode.

  PID USERNAME     VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT COMMAND

Let’s describe the fields we did not previously describe. VCSW means Voluntary Context SWitches and it is the type of context switch that happens when process yields before it has used its allotted time quantum. IVCSW means InVoluntary Context SWitches - it is the type of context switch that happens when the process uses all of its allocated time quantum and it is forcibly switched out of CPU by scheduler, to let other processes run. The READ column shows the read operations (read IOPS) and WRITE shows the write IOPS. FAULT shows how many faults there were and the TOTAL column is the sum of READ and WRITE columns. The PERCENT column shows the percentage of I/O activity for each process.

We Need to Go Deeper

Keep in mind that process (and I/O) monitoring does not end on top(1) command on FreeBSD. To get a better view of what happens in the FreeBSD system at CPU and/or IO level, you should also try at least one of the following commands:

• htop(1)
• gstat(1)
• systat(1)
• vmstat(1)

Summary

While top (1) is a complex tool that offers a lot of information, we are hopeful that this article will provide some insight into how to use top(1) more efficiently.