In Basics of ZFS Snapshot Management, we demonstrated how snapshots and clones can be used to access data from a given point in time. In this article, we’ll learn how ZFS boot environments (BEs) are used to boot the operating system itself into a previous point in time.
A ZFS boot environment is a bootable clone of the datasets needed to boot the operating system. Creating a BE before performing an upgrade provides a low-cost safeguard: if there is a problem with the update, the system can be rebooted back to the point in time before the upgrade.
The FreeBSD boot loader was rewritten for 12.0 to add BE support. Additionally, the default ZFS layout in the FreeBSD installer understands BEs.
This article demonstrates how to use the bectl utility to manage BEs and provides examples on how to update packages, apply security patches, and upgrade the operating system using BEs.
Using bectl(8)
Prior to FreeBSD 12.0, the sysutils/beadm port was available for managing BEs. This shell script was rewritten in C and included with the operating system as bectl beginning withFreeBSD 12.0. If you’ve used beadm in the past, the options are the same in bectl. bectl adds a few more options (such as jail support) which we won’t cover today.
This command lists available BEs. Here is an example from a freshly installed 12.2 system:
bectl list
BE Active Mountpoint Space Created
default NR / 906M 2021-06-10 04:29
By default, there is one BE named default. The N in the Active column indicates that this BE is active now; in other words, this is the currently booted BE. The R means that this BE is automatically selected on reboot, unless the boot process is interrupted.
The create option creates a new BE, which is a clone of the currently booted BE:
bectl create test
bectl list
BE Active Mountpoint Space Created
default NR / 906M 2021-06-10 04:29
test - - 8K 2021-06-10 05:11
The test BE has no Active letters as it is not the currently booted BE and it isn’t configured to be the default on next boot.
If I reboot this system, the boot menu now has a new 8. Boot Environments entry:
Press 8 or e to enter the BE sub-menu:
Press 2 as often as needed to toggle through the list of BEs. Once you find the one you want to boot into, press a or enterto boot into the selected BE. Here I’ve selected the test BE:
Once booted, the listing changes to:
bectl list
BE Active Mountpoint Space Created
default R / 906M 2021-06-10 04:29
test N - 404K 2021-06-10 05:11
Since the R is still on the default BE, the system will continue to boot into the default BE unless I interrupt the boot menu. To permanently set the test BE as the new default, use the activate switch:
bectl activate test
Successfully activated boot environment test
bectl list
BE Active Mountpoint Space Created
default - - 424K 2021-06-10 04:29
test NR / 906M 2021-06-10 05:11
Did you know?
Getting your ZFS infrastructure up to date has never been easier!
Our team provides consistent, expert advice tailored to your business.
In this example, I have a 12.2 laptop with a lot of packages installed and it’s been a while since I’ve updated them. I’ll start by creating a BE with a useful name:
bectl create pkg_update_`date +%y%m%d`
If I mount that BE and perform the package update within that BE, I can reboot into the new BE to test the updated packages. To mount the BE, specify its name:
bectl list | grep pkg
pkg_update_210610 - - 496K 2021-06-10 05:11
bectl mount pkg_update_210610
Successfully mounted pkg_update_210610 at /tmp/be_mount.EFae
Make note of the mount point (tmp/be_mount.EFae) and specify it to the pkg command to update the repos and perform the pkg upgrade:
pkg -r /tmp/be_mount.EFae update -f
Updating FreeBSD repository catalogue...
FreeBSD repository update completed. 30504 packages processed.
All repositories are up to date.
pkg -r /tmp/be_mount.EFae upgrade
Checking for upgrades (626 candidates): 100%
<snip>
Number of packages to be installed: 44
Number of packages to be upgraded: 394
Number of packages to be reinstalled: 37
The process will require 179 MiB more space.
1 GiB to be downloaded.
Proceed with this action? [y/N]: y
Once the upgrade is complete, boot into that BE to test the packages. Once you’re satisfied everything is working as expected, you can activate that BE.
Example: Deploying a Security Patch
The same process can be used to patch the operating system. In this example, I’ll update the default FreeBSD 12.2 installation to the latest patch level. Start by creating a BE, mounting it, and noting the name of the mount point:
bectl create security_update_`date +%y%m%d`
bectl list |grep security
security_update_210610 - - 8K 2021-06-10 06:15
bectl mount security_update_210610
Successfully mounted security_update_210610 at /tmp/be_mount.7pMM
Next, ensure that freebsd-update doesn’t use its default database in the currently booted BE by removing that directory:
rm -Rf /var/db/freebsd-update
Run the freebsd-update utility and specify the BE mount point as the basedir (-b) and the BE’s database as the workdir (-d). First, fetch the updates:
freebsd-update -b /tmp/be_mount.7pMM -d /tmp/be_mount.7pMM/var/db/freebsd-update fetch
<output snipped>
The following files will be removed as part of updating to
12.2-RELEASE-p8:
<SNIP> (use page down to read through and q to quit)
For this example, I’ll test updating the patched 12.2-RELEASE-p8 BE to 13.0. Again, create a new BE and mount it:
bectl create 13.0
bectl mount 13.0
Successfully mounted 13.0 at /tmp/be_mount.JO5Y
Specify the release version (-r 13.0-RELEASE) to upgrade toin the freebsd-update command:
freebsd-update -b /tmp/be_mount.JO5Y -d /tmp/be_mount.JO5Y/var/db/freebsd-update -r 13.0-RELEASE upgrade
<output snipped)
The following file will be removed, as it no longer exists in
FreeBSD 13.0-RELEASE: /etc/motd
Does this look reasonable (y/n)? y
<snip> (page down through list of files and q to quit)
To install the downloaded upgrades, run "/usr/sbin/freebsd-update install".
Next, repeat the command, this time with install:
freebsd-update -b /tmp/be_mount.JO5Y -d /tmp/be_mount.JO5Y/var/db/freebsd-update -r 13.0-RELEASE install
Installing updates...
Kernel updates have been installed. Please reboot and run
"/usr/sbin/freebsd-update install" again to finish installing updates.
Using the ZFS bootonce feature we developed at Klara, you can instruct the system to boot a different boot environment, one time only, with the -t (temporary) flag.
bectl activate -t 13.0
Reboot and the 13.0 BE will be loaded, and the temporary flag will be cleared. Since you are now rebooted into that BE, you don’t have to mount it and can just issue the install command to complete the installation process:
freebsd-update install
Once you are finished testing the upgrade, you can activate the BE.
bectl activate 13.0
If the upgrade isn’t working, or you don’t activate the 13.0 BE, rebooting will return to the previous default BE.
If Something Goes Wrong
Most OS upgrades “just work”. Occasionally you will hit a gotcha that requires configuration file edits after booting into the new OS version. Once these are resolved, you can activate the BE.
Sometimes you get the dreaded mountroot> prompt that halts the OS boot process. In this case, simply boot into a good BE, then mount the problematic BE, noting its mount point. As you research which update gotcha bit you, you can make any modifications within the mount point. For example, if the BE mount point is /tmp/be_mount.ON3E and you need to edit the BE’s fstab file, make your edits to /tmp/be_mount.ON3E/etc/fstab
Meet the author: Dru Lavigne
Dru Lavigne is a retired network and systems administrator, IT instructor, author, and international speaker. Dru is author of BSD Hacks, The Best of FreeBSD Basics, and The Definitive Guide to PC-BSD.
Getting expert FreeBSD advice is as easy as reaching out to us!
At Klara, we have an entire team dedicated to helping you with your FreeBSD projects. Whether you’re planning a FreeBSD project, or are in the middle of one and need a bit of extra insight, we’re here to help!
DTraceis a powerful tool for system administrators to diagnosis system issues without unduly impacting performance. DTrace became part of FreeBSD with the release of FreeBSD 7.1 in 2009—two years before Oracle began porting DTrace, and nine years before Oracle eventually solved the inherent CDDL vs GPL license conflict.
Pkgng became FreeBSD’s official package manager in FreeBSD 10 in 2014. Applications can be easily installed from either pkg—a system managing precompiled binary packages—or the ports tree, which automates building and installation of packages directly from their source code.
Early on, developers working on Unix created a set of ideals that acted as a roadmap for the programs they wrote. They didn’t always follow these ideals, but they set the tone for the Unix project. Keep programs simple, design programs to work together, test early and often – are only some of these ideals.…
I did not know this tool and how to use it, in this article I have learned how to do it. Another reason to continue with FreeBSD – recovering a system after a failed upgrade. Thank you. Great job.
While this is super cool, the upgrade from 13.0 to 13.1 failed for me:
I followed the instructions up to the point where I would reboot into 13.1 (“bectl activate -t 13.0”, I have replaced 13.0 with 13.1 for the RELEASE and for the BE of course).
After rebooting into 13.1, I couldn’t run the second “freebsd-update install” which would run on the currently running system, without mounting the BE. It would tell me that I have to fetch updates first, so apparently the system wasn’t aware that it was in the middle of upgrading to 13.1.
The reason might be that the new BE runs on the new kernel, but on the old Userland, without being able to proceed with the upgrade:
Checking userland and kernel versions:
Userland and kernel are not in sync
Userland version: 1300139
Kernel version: 1301000
Any ideas?
(BTW, in the meantime, I have rebooted into the old 13.0 BE, destroyed 13.1, and started from scratch, but this time directly booting into 13.1, and then performing the upgrade as usual.)
Pingback: Valuable News – 2021/07/19 | 𝚟𝚎𝚛𝚖𝚊𝚍𝚎𝚗
I did not know this tool and how to use it, in this article I have learned how to do it. Another reason to continue with FreeBSD – recovering a system after a failed upgrade. Thank you. Great job.
Pingback: Manipulating a Pool from the Rescue System | Klara Inc.
Pingback: Evaluating FreeBSD CURRENT for Production Use | Klara Inc.
Pingback: Valuable News – 2022/06/06 | 𝚟𝚎𝚛𝚖𝚊𝚍𝚎𝚗
While this is super cool, the upgrade from 13.0 to 13.1 failed for me:
I followed the instructions up to the point where I would reboot into 13.1 (“bectl activate -t 13.0”, I have replaced 13.0 with 13.1 for the RELEASE and for the BE of course).
After rebooting into 13.1, I couldn’t run the second “freebsd-update install” which would run on the currently running system, without mounting the BE. It would tell me that I have to fetch updates first, so apparently the system wasn’t aware that it was in the middle of upgrading to 13.1.
The reason might be that the new BE runs on the new kernel, but on the old Userland, without being able to proceed with the upgrade:
Checking userland and kernel versions:
Userland and kernel are not in sync
Userland version: 1300139
Kernel version: 1301000
Any ideas?
(BTW, in the meantime, I have rebooted into the old 13.0 BE, destroyed 13.1, and started from scratch, but this time directly booting into 13.1, and then performing the upgrade as usual.)