ZFS Orchestration Tools – Part 1: Snapshots

When using ZFS as part of your data storage solution, two compelling features are its fast and efficient snapshots, which we covered in Advanced ZFS Snapshots, and its facilities for efficient replication, which we previously covered in Introduction to ZFS Replication.

In those articles, we introduced the raw ZFS commands associated with these features and discussed how to use them manually. It is handy to be familiar with how to operate these tools manually; however, it is more useful to automate tasks based on them on a regular schedule. One option is to write your own scripts for common tasks, which has the advantage of maximum flexibility. Alternatively, existing ZFS snapshot tools are available, and using one of these can save time and help avoid costly mistakes.

In this article, we will review various ZFS snapshot tools and examine key aspects of their features and implementation relevant to choosing the right tool. The focus here is on snapshots, while Part 2 will cover replication and backups.

Regular Snapshots

Snapshots have a variety of uses, including forming the underpinnings of FreeBSD boot environments feature. With regular snapshots of data files, it can be easy to recover files that were accidentally deleted or modified. While some desktop environments provide a “Trash” folder as a protection against accidental deletions, it can be hard to resist the compulsion to empty it. Moreover, a trash folder only contains deleted files, which is no help if you want to go back to the previous content of a file that has been overwritten.

With regular snapshots in place, it is tempting to think of them as backups. While they do provide an easy means to recover individual files, they still reside on the same physical storage media and use the same operating system and filesystem as the live data. Backups are meant for disaster recovery, and ZFS snapshots won’t be of any help if your data center burns down.

ZFS Snapshot Tools

The table below lists some of the many ZFS snapshot tools available for managing snapshots. There’s considerable overlap in terms of the functionality these tools offer, but each tool varies in its specific focus. We’ve included details on whether they are available in FreeBSD ports, their programming language, license and the configuration file format.

We discussed zrep in a previous article, ZFS High Availability with Asynchronous Replication and zrep. This is something of an outlier among ZFS snapshot tools, as its functionality is primarily focused on failover between servers. Similarly, zfs-replicate shares that focus.

Triggering Snapshots

The traditional way to schedule automatic jobs on Unix is to use cron(8). On Linux, it can be more convenient to use a systemd timer, which allows the job to be managed as a service. Similarly, Solaris’ SMF supports periodic services. On FreeBSD, a simple and convenient option is to use the periodic(8) mechanism, which is designed for running system scripts on daily, weekly and monthly schedules.

Using Cron Jobs to Automate ZFS Snapshots

In an earlier article, FreeBSD Periodic Scripts, we covered this topic, including details that are relevant to ZFS snapshot tools —such as how to configure where output is sent.

After installing the zfs-periodic package, you need to modify /etc/periodic.conf to to enable it and add entries for each of duration:

daily_zfs_snapshot_enable="YES"
weekly_zfs_snapshot_enable="YES"
monthly_zfs_snapshot_enable="YES

It also needs a list of pools to snapshot. However, it doesn’t allow for more granular setups, such as snapshotting individual datasets. The default is tank, but if you have other pools, you may need to configure them. For example:

daily_zfs_snapshot_pools="zpool tank"

Another aspect you can configure is how many snapshots to keep before they are cleared. In this case, the defaults are seven snapshots are kept for daily and weekly intervals, and three for monthly. To keep a whole year’s worth of monthly snapshots, you would use:

monthly_zfs_snapshot_keep="12"

Automating Snapshots with Advanced ZFS Snapshot Utilities

While  zfs-periodic is simple and can be adequate for many use cases, it does have considerable limitations. It can be useful to have more frequent snapshots, and from a ZFS perspective, there are few technical reasons not to create them at a higher frequency. In practice, I set them to run every 15 minutes. It’s easy to add your own intervals via the periodic(8) system, so it is possible to overcome this limitation. To enable hourly snapshots, you need to add extra entries to /etc/crontab. For example:

0    *    *    *    *    root    periodic hourly

The periodic scripts are arranged in sub-directories of /etc/periodic and /usr/local/etc/periodic . A directory for a custom interval needs to exist containing scripts for each job. For hourly snapshots, you don’t need to be concerned with this because /usr/local/etc/periodic/hourly/000.zfs-snapshot already exists.

The underlying zfs-snapshot script that this calls supports both hourly and yearly snapshots. Adapting it to further intervals would be trivial with basic shell scripting  knowledge, but it may be better to consider one of the alternative tools instead.

Comparing ZFS Snapshot Tools for Scheduling

Another tool that integrates with the periodic(8) system is zfsnap, with both version 1 and 2 available in FreeBSD ports. It takes the unique approach of encoding the exact “time-to-live” of each snapshot in its name, which makes it able to support arbitrary intervals. However, configuration changes to how long snapshots should last won’t apply retrospectively to old snapshots.

For arbitrary intervals with zfsnap, you will need to use individual cron(5) entries. This approach is commonly used by many of tools for triggering jobs. Some example entries include:

5    * * * * zfsnap snapshot -a 5d -r zpool
*/5  * * * * zap snap 1d
*/15 * * * * pyznap snap >> /var/log/pyznap.log 2>&1
*/5  * * * * zfs-snapshot-mgmt
*/15 * * * * flock -n /var/run/sanoid/cron-take.lock -c "TZ=UTC sanoid --take-snapshots"

Note that /etc/crontab has an additional column for the user, typically root.

Finally, rather than using cron, zrepl and znapzend take the approach of running as a continuous service and doing their own scheduling.

Configuring Datasets

The preceding examples of cron entries highlight some differences in how you need to configure different tools. While zfsnap needs multiple cron entries for different datasets and intervals, pyznap needs only a single invocation, as the datasets and number of snapshots to keep are set up in a configuration file. Zap, on the other hand,takes the approach of using ZFS properties to configure datasets. This makes it easy to add and remove datasets.

For example, to enable snapshots for my home directory, I would simply run:

zfs set zap:snap=on tank/home/opk

zfstools also follows this approach to configuration but additionally has support for special handling for datasets containing databases so you might use:

zfs set com.sun:auto-snapshot=postgresql tank/db/postgres

Sanoid and zrepl, both of which use configuration files, allow you to define pre- and post-script hooks to cover this same requirement.

Pruning old Snapshots

Allowing snapshots to build up soon becomes unwieldy, even if ZFS performance is not overly affected. Pruning old snapshots also helps free up disk space. We already touched on this for zfs-periodic, where you can configure how many snapshots to keep.

If you have a mix of data, you may want to apply different pruning policies. Sanoid not only allows for separate configuration for each dataset but also supports templates to share common configuration settings. The following is a configuration example from the sanoid documentation for a template named production:

[data/home]
use_template = production
[template_production]
frequently = 0
hourly = 36
daily = 30
monthly = 3
yearly = 0
autosnap = yes
autoprune = yes

Znapzend and zfs_mgmt support this functionality while keeping to ZFS properties for configuration. This means the property stores the same information expressed in a condensed form.

In the case of znapzend, these are typically set up using the znapzendzetup, but the result is stored in a ZFS property with the key org.znapzend:src_plan and a value such as:

1days=>1hours,1weeks=>1days,4weeks=>1weeks,1years=>1months,5years=>1years

Instead of only using a basic count, it is possible to use other criteria when deciding on snapshots to prune. For example, zfs-autosnapshot considers the size of snapshots when determining which to remove.

Clearing up old snapshots can take rather longer than creating them. Many tools separate these two tasks and often recommend doing the cleanup less frequently. This avoids issues where snapshot creation is delayed while waiting for old snapshots to be deleted.

Using holds to Preserve Snapshots

Regardless of how you create and clean up snapshots, you may occasionally want to keep a particular snapshot beyond its scheduled removal time. For this purpose, it can be helpful to use a hold. For example:

zfs hold keep tank/home/opk@202312

The word keep here is a tag we define, and you should replace it with something more meaningful.

If you have other software, such as tape backups or malware scanners, it can be prudent to pass them a snapshot so they view consistent, unchanging files. Using a hold ensures that the dataset won’t disappear before such software has completed its work.

Retrieving Data from Snapshots

Regular snapshots can be helpful in a variety of situations beyond just restoring accidentally deleted files. In a recent case, a snapshot served me as a valuable source of information when investigating some regression tests that are run nightly. I was able to verify whether a problem was new or had only started at a specific time. For this, it is helpful to have easy access to the data across multiple snapshots.

There are times when you may need to do a full rollback of a dataset, or clone a snapshot, or send it elsewhere, but usually, simple access to the data is all that you need. You can mount an individual snapshot, for example:

mount -t zfs tank/home/opk@20241219_14 /mnt

However, ZFS provides easy access to snapshots via a special hidden .zfs directory. This directory is not shown in any directory listings, but you can navigate to it from the top-level of any dataset. Snapshots can be found in a snapshot subdirectory:

cd .zfs/snapshot

You can then use ls to list files, along with cp, rsync, diff, or whatever else you like to recover and inspect the data.

We also covered httm (Hot-Tub Time Machine) in an earlier article, httm –is Your ZFS Turn-Back-Time Method. It provides an easy-to-use interface for discovering files in snapshots.

Manipulating Snapshots as an Unprivileged User

With ZFS, you can delegate permissions for actions such as creating snapshots to unprivileged users. This means that automatic snapshots don’t necessarily need to be run as root. You might create a system user to run a snapshot tool or perhaps allow users to create and remove snapshots for their own data.

Creating snapshots requires the snapshot permission. To delegate permissions to myself for my home directory, I would need the following:

zfs allow opk snapshot tank/home/opk

While this allows the specified user to create snapshots, removing them is still restricted. Since snapshots are descendent file systems, we need to use the -d option when delegating permissions for the snapshots. In addition to the destroy permission, the mount permission is also needed:

>zfs allow -d opk destroy,mount tank/home/opk

Note that this also applies to other descendent file systems, such as child datasets. In more complex setups, you may find that additional root permissions are required that this method doesn’t resolve.

Conclusion: Choosing the Right ZFS Snapshot Tool

There are a variety of ZFS snapshot tools available for managing basic snapshots of your ZFS datasets. If that’s all you need, simpler tools such as zfs-periodic or zap are excellent choices. However, other factors might influence your choice, such as a preference for using a configuration file or ZFS properties for setup. Additionally, you may have further needs, such as support for hooks to quiesce your database.

A particularly common requirement is to extend it to a full backup solution by integrating with ZFS’s send and receive features for replication. This is the area that we will explore in the next article.

At Klara, we specialize in OpenZFS solutions, helping businesses optimize their ZFS storage strategy—from fine-tuning snapshots to designing efficient backup and replication workflows. Contact us to discuss how we can help streamline your storage management.