As commercial storage becomes increasingly expensive, more and more of the Education vertical is looking at Open Source solutions for storage. In this article, we discuss the value of OpenZFS for Universities and how system administrators can best leverage it to their benefit.
Explaining top(1) on FreeBSD
Explaining top(1) on FreeBSD
We all know and have at least once used the top(1) command to track information about our cpu and processes, but how many of you know what each field means? Today we will guide you through each of these fields. By default, top(1) displays the ‘top’ processes on each system and periodically updates this information every 2.0 seconds using the raw cpu use percentage to rank the processes in the list.
Default top(1) Output
This is how to default top(1) command output looks. We will use it as a base to describe each field and column.
last pid: 95139; load averages: 0.21, 0.25, 0.25 up 17+06:17:12 21:01:34 22 processes: 1 running, 21 sleeping CPU: 0.1% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.9% idle Mem: 5968K Active, 285M Inact, 2918M Wired, 584K Buf, 552M Free ARC: 101M Total, 32M MFU, 24M MRU, 392K Anon, 1286K Header, 44M Other 12M Compressed, 52M Uncompressed, 4.46:1 Ratio Swap: 1024M Total, 1024M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 95139 klarabsd 1 23 0 14M 3988K CPU3 3 0:01 0.42% top 91467 klarabsd 1 20 0 21M 9536K select 1 0:00 0.04% sshd 79666 root 1 20 0 13M 3024K select 2 5:05 0.02% mountd 66928 root 1 20 0 13M 2312K select 2 4:32 0.02% powerd 86590 ntpd 1 20 0 21M 5452K select 2 2:46 0.01% ntpd 88853 root 1 21 0 173M 146M select 1 2:29 0.00% smbd 62176 root 1 20 0 13M 2812K select 1 2:16 0.00% syslogd 88570 root 1 20 0 39M 16M select 0 0:41 0.00% nmbd 91145 root 1 20 0 13M 2728K nanslp 3 0:26 0.00% cron 80942 root 4 52 0 12M 2300K rpcsvc 0 0:05 0.00% nfscbd 83679 root 32 52 0 12M 2788K rpcsvc 1 0:05 0.00% nfsd 77923 root 1 20 0 13M 2680K select 1 0:04 0.00% rpcbind 90750 root 1 20 0 20M 8268K select 1 0:04 0.00% sshd 93654 root 2 20 0 136M 104M select 0 0:03 0.00% smbd 93831 root 1 20 0 131M 104M select 0 0:03 0.00% smbd 45020 root 1 20 0 11M 1488K select 0 0:01 0.00% devd 91568 klarabsd 1 29 0 15M 6196K pause 2 0:00 0.00% zsh 91342 root 1 41 0 21M 9508K select 3 0:00 0.00% sshd 312 klarabsd 3 20 0 31M 12M select 0 0:00 0.00% mocp 83547 root 1 52 0 12M 3036K select 2 0:00 0.00% nfsd 7478 root 1 52 0 12M 2196K pause 1 0:00 0.00% adjkerntz 80887 root 1 52 0 12M 2252K accept 0 0:00 0.00% nfscbd
The last pid: shows the PID of the last (most recent) process, which is 95139 in our example.
The load averages: field displays the load average of the system over the last 1, 5, and 15 minutes respectively. The topic of load average is generally very broad and it is calculated differently on various UNIX systems, so a separate article may be dedicated for just that. The up field shows the system uptime which is 17 days, (+) six hours, seventeen minutes and twelve seconds 06:17:12. The last field on the top right is the current time 21:01:34 displayed in HH:MM:SS format.
We have now covered the first line of output from top(1). Time for the second line. This one contains information about the current count of processes and their state. On this system, there are 22 processes with 1 running and 21 sleeping processes.
The third line shows a summary breakdown of what is currently running on the system CPUs. The user field shows the usage of userspace processes. The nice field shows how much CPU time is taken by processes with different nice(1) priorities. The system field shows how much CPU time is used by the FreeBSD kernel. The interrupt field shows how much CPU time is delegated to interrupt support. Last but not least the idle field shows how much CPU time is unused or free.
The fourth line of the top(1) command marked Mem: shows a detailed summary of the memory information. The following physical memory stats contain this information:
Active shows the number of bytes used by actively running processes. Inact shows the number of bytes not recently used by a process (while the process itself may still remain active – just some parts of its memory has not been used recently). If the system needs more memory, then this inactive memory may be swapped out or freed to make room. The Laundry queue shows the amount of memory that is “dirty”, needing to be laundering before it can be reused. Data that has been modified since it was read from disk, or has not yet been stored on disk, cannot be easily recreated, so it must be written to swap or the modified file before that memory can be reused. For more details about how the inactive memory and the laundry system work, see Exploring Swap on FreeBSD . The Wired count shows the number of bytes wired down which, unlike the other types, CANNOT be swapped out. Buf shows the number of bytes used for buffers and caches. Instead of being swapped out, they will be just freed. Free shows the free memory.
You might also be interested in
Get more out of your FreeBSD development
Kernel development is crucial to many companies. If you have a FreeBSD implementation or you’re looking at scoping out work for the future, our team can help you further enable your efforts.
To get the general idea of how much system memory is actually used versus free, here is a hint. The used memory is generally the sum of Active and Wired fields. As Inact/Laundry/Buf fields can be swapped out, they should all be treated as ‘buffers’ or ‘cached’ information in memory that can (and will) be freed when needed. FreeBSD, as any other clever UNIX system, follows the motto of ‘Unused memory is wasted memory.’ so when the memory is not needed, FreeBSD keeps it in case an inactive process will try to access it again.
The fifth line includes ZFS Adaptive Replacement Cache or, in short, ARC. This is the amount or RAM that ZFS uses as cache. We may select what should be the minimum and maximum ARC size in the /etc/sysctl.conf file with vfs.zfs.arc_min and vfs.zfs.arc_max parameters.
The ZFS ARC stats fields show the following information.
The Total field shows the number of wired bytes used for the ZFS ARC. The MRU field shows the number of ARC bytes holding Most Recently Used data. The MFU field shows the number of ARC bytes holding Most Frequently Used data. The Anon field shows the number of ARC bytes holding in flight data that does not fit into other categories. The Header field shows the number of ARC bytes holding headers. The Other field shows other miscellaneous ARC bytes needed for ZFS to operate properly.
The sixth line is a continuation of the ZFS ARC stats but, as ARC is compressed on FreeBSD, it shows stats of how much memory has been saved thanks to using the compression algorithm on all that ZFS cache. The Compressed field shows bytes of memory used by ARC caches. From all the memory used by ARC, the 12M(megabytes) is compressed and thanks to 4.46:1 compress Ratio displayed later on the right, it uses ‘only’ 12M instead of 52M (megabytes) shown in the Uncompressed field that will be used without compression.
The remaining Swap: field on the seventh line, as you probably guessed, shows the SWAP information. Here, it is really simple – it just shows Total size of SWAP area and how much of it is actually Free.
The second part of the top(1) command is the list of processes. These are the displayed columns.
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
This is the part where information about individual processes is displayed. The PID is the process id. The USERNAME shows the name of the process’s owner. The PRI column shows the current priority of the process while NICE is the amount of nice(1) a process has. SIZE shows the text/data/stack of process. RES is the current amount of resident memory while SWAP is the amount of swap used – both in kilobytes. STATE is the current state of the process – it can START/SLEEP/STOP/ZOMB/WAIT/LOCK. When a process is waiting on something to happen, it will display the ‘waitchan’, describing what it is waiting for. There are many possible values for this, we might have to do an entire article on some of the common values. Keep in mind that RUN is shown as CPUn (for example, CPU3) on systems with multiple CPU cores. The C is the logical processor on which the process is currently running – this is visible only on SMP systems. The TIME column shows the number of seconds that the process used. The WCPU shows the weighted cpu percentage. The COMMAND shows the name of the command that the process is currently running.
To just check which CPU is doing what, we can use the command below to display each CPU current usage stats. This is how it’s going to look:
% top -P 0 ### Output last pid: 23091; load averages: 0.66, 0.50, up 0+01:12:58 912:44:49 107 processes: 2 running, 104 sleeping, 1 waiting CPU 0: 1.6% user, 0.0% nice, 0.0% system, 0.0% interrupt, 98.4% idle CPU 1: 2.4% user, 0.0% nice, 0.4% system, 0.0% interrupt, 97.3% idle CPU 2: 0.0% user, 0.0% nice, 0.0% system, 1.2% interrupt, 98.8% idle CPU 3: 1.1% user, 0.0% nice, 1.5% system, 0.0% interrupt, 97.3% idle CPU 4: 0.4% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.2% idle CPU 5: 2.7% user, 0.0% nice, 1.2% system, 0.0% interrupt, 96.1% idle CPU 6: 0.4% user, 0.0% nice, 0.8% system, 0.0% interrupt, 98.8% idle CPU 7: 1.2% user, 0.0% nice, 0.0% system, 0.0% interrupt, 98.8% idle Mem: 1650M Active, 2263M Inact, 278M Laundry, 2623M Wired, 24G Free ARC: 354M Total, 109M MFU, 92M MRU, 1508K Anon, 3657K Header, 147M Other 87M Compressed, 313M Uncompressed, 3.58:1 Ratio Swap: 1024M Total, 1024M Free
Interactive Mode Options
To avoid citing the whole top(1) man page, we will describe the ‘most interesting’ switches that we can toggle in the interactive (default) mode. No matter to what delay (with letter s) the refresh is set, we can always refresh the screen manually using the [CTRL]+[L] shortcut. We can also refresh the stats using the biggest ‘key’ on the keyboard – the more convenient [SPACEBAR]. The letter m will toggle between the ‘cpu’ mode (which is the default) and the ‘io’ mode. Just type ‘cpu’ or ‘io’ after pressing the m key. The letter S will toggle the display of system processes – which is very useful. We can also kill(1) a process using the letterk or renice(1) one with the letter r. For sorting the display of processes, use the letter o and then use one of these columns for the sorting order: cpu/res/size/time – the default is cpu. Using the capital P letter, we can switch to display per CPU stats instead of summary percentages of CPU usage. With capital J we can display processes owned only by a specific jail. The letter i can toggle displaying the idle processes.
The I/O Mode
While the default top(1) mode is CPU related, the top(1) on FreeBSD also has another mode – the I/O mode which is very useful in tracking processes that do much I/O. To start the top(1) in the I/O mode along with sensible sorting on the TOTAL column, use this command:
# top -S -m io -o total
The command above also includes the -S option to show the system processes because when we start the ZFS scrub process, it will not show as any separate process. However, with the -S option, we will see high I/O activity of the zfskern system process – this is where ZFS scrub does its work.
This is how the top(1) I/O mode looks like on FreeBSD.
last pid: 9572; load averages: 0.40, 0.20, 0.17 up 17+21:37:30 12:21:52 75 processes: 2 running, 72 sleeping, 1 waiting CPU: 0.9% user, 0.0% nice, 8.6% system, 3.4% interrupt, 87.1% idle Mem: 32M Active, 298M Inact, 2922M Wired, 584K Buf, 512M Free ARC: 191M Total, 58M MFU, 64M MRU, 7201K Anon, 3024K Header, 60M Other 84M Compressed, 1179M Uncompressed, 14.01:1 Ratio Swap: 1024M Total, 1024M Free PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND 806 root 1745 78 0 465 0 465 81.72% pkg 19 root 104 26 0 38 0 38 6.68% zfskern 9298 klarabsd 37 12 37 0 0 37 6.50% rsync 1303 klarabsd 29 1 29 0 0 29 5.10% dd 2 root 0 0 0 0 0 0 0.00% KTLS 3 root 0 0 0 0 0 0 0.00% crypto 77923 root 0 0 0 0 0 0 0.00% rpcbind 64387 klarabsd 4 0 0 0 0 0 0.00% sshd 4 root 0 0 0 0 0 0 0.00% crypto returns 0 9572 klarabsd 6 0 0 0 0 0 0.00% rsync 5 root 0 0 0 0 0 0 0.00% crypto returns 1 6 root 0 0 0 0 0 0 0.00% crypto returns 2 68006 root 0 0 0 0 0 0 0.00% sshd 7 root 0 0 0 0 0 0 0.00% crypto returns 3 93831 root 1 0 0 0 0 0 0.00% smbd 8 root 147 54 0 0 0 0 0.00% cam 9 root 0 0 0 0 0 0 0.00% soaiod1 91145 root 0 0 0 0 0 0 0.00% cron 10 root 0 0 0 0 0 0 0.00% audit 11 root 4429 4798 0 0 0 0 0.00% idle 71883 klarabsd 4 0 0 0 0 0 0.00% sshd 427 root 0 0 0 0 0 0 0.00% pkg 68331 klarabsd 0 0 0 0 0 0 0.00% zsh
It has slightly different columns than the default CPU mode top(1) command. These are the displayed columns in the IO mode.
PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND
Let’s describe the fields we did not previously describe. VCSW means Voluntary Context SWitches and it is the type of context switch that happens when process yields before it has used its allotted time quantum. IVCSW means InVoluntary Context SWitches – it is the type of context switch that happens when the process uses all of its allocated time quantum and it is forcibly switched out of CPU by scheduler, to let other processes run. The READ column shows the read operations (read IOPS) and WRITE shows the write IOPS. FAULT shows how many faults there were and the TOTAL column is the sum of READ and WRITE columns. The PERCENT column shows the percentage of I/O activity for each process.
We Need to Go Deeper
Keep in mind that process (and I/O) monitoring does not end on top(1) command on FreeBSD. To get a better view of what happens in the FreeBSD system at CPU and/or IO level, you should also try at least one of the following commands:
While top (1) is a complex tool that offers a lot of information, we are hopeful that this article will provide some insight into how to use top(1) more efficiently.
Like this article? Share it!
You might also be interested in
Getting expert FreeBSD advice is as easy as reaching out to us!
At Klara, we have an entire team dedicated to helping you with your FreeBSD projects. Whether you’re planning a FreeBSD project, or are in the middle of one and need a bit of extra insight, we’re here to help!
Discover how OpenZFS can provide cost-effective and reliable storage for high-performance computing (HPC) workloads in this comprehensive write-up.
The most common category of ZFS questions is “how should I set up my pool?” Sometimes the question ends “… using the drives I already have” and sometimes it ends with “and how many drives should I buy.” Either way, today’s article can help you make sense of your options.