In the first part of this article, we introduced a range of available tools for automating common tasks related to ZFS. Where the focus of that first article was on snapshots, in this article we will primarily consider replication. For a full list of the tools and their links, refer back to the ZFS Orchestration Tools – Part 1: Snapshots.
Efficient ZFS Replication with Snapshots
Automating ZFS data replication is useful wherever the same data sets are regularly transferred to the same target locations. This is common for backups and for maintaining secondary servers ready for failover. Using ZFS to send and receive manually is not as simple as scp or rsync, so some of the tools also provide a simple interface for one-off transfers. However, the performance and efficiency improvements of the ZFS replication feature make it worth the additional setup.
While automatic snapshots were our focus in the previous article, snapshots are critical to ZFS replication because they are the atomic unit of change that zfs send uses to track the current state and the difference in the version of the data on the remote system. The technology behind ZFS snapshots allows for efficient transfers of only incremental changes. Unlike rsync, ZFS doesn’t have to compare the blocks on both sides to know what has changed. Using the snapshots, it can easily generate a list of every changed block between a pair of snapshots using only the birth times of the blocks in question.
Push vs. Pull: Choosing the Right ZFS Replication Strategy
One of the most powerful aspects of ZFS replication is that it follows the UNIX tradition of combining tools that do one thing well with UNIX pipes. It is common to use ssh to handle data transfer and orchestrate the remote commands, but you can simply run both zfs send and zfs receive locally. This can be useful for backing up data to removable media.
Failure Scenarios and Synchronization Considerations
There are several factors that go into deciding if a system should “push” changes to the backup/secondary server, or if that remote server should “pull” the changes over. One consideration is the failure scenario. If the primary is down, and the secondary switches out of read-only mode and accepts new changes, it is easier to disable a “pull” on the secondary. Then, resync the primary when it is back up.
However, if the primary is pushing, as soon as it comes back up, it will try to push changes to the secondary, and either fail (the default), or overwrite the divergence on the secondary (force mode).
Security Implications of Push vs. Pull
On the flip side, it is better security practice to design your permissions model such that each server is able to push to the backup, rather than the backup machine having privileged access to every machine on your network. In the event the backup server is compromised, it shouldn’t be possible to use it as a steppingstone to take over every other server in your infrastructure.
Choosing the Right Approach for Your Setup
When backing up data to a remote system, pushing the data may seem to be the most natural choice. However, having the backup system invoke the entire process – pulling the data, may be easier to orchestrate. This tends to cope better with the situation where the backup system is down because the primary system won’t be repeatedly failing to contact it. When the backup server comes back online, it can catch up with all of the missing changes at its own pace, rather than having every remote machine trying to push changes in at once.
There are situations where a push-based setup makes more sense. For example, when backing up a system that isn’t permanently turned on such as a laptop or where you need to traverse NAT to reach the backup system.
ZFS Replication Transfer Steps with Syncoid
Let’s start by considering a simple manual transfer without any configuration. sanoid comes with a separate syncoid tool for replication, and a simple command to push a copy of the local system’s home directories to a remote server would be as follows:
syncoid -r zroot/usr/home [email protected]:tank/backup/home
The one option we’ve used is -r to recursively transfer the child datasets. It is helpful that syncoid hides much of the underlying complexity, but we’ll now break this down into steps so we can have a fuller understanding and introduce points for further discussion. What that command does is as follows:
- Opens an ssh connection to the remote system. syncoid uses an initial ssh connection in master mode, which it later reuses so that you only need to authenticate once. It uses this connection for each remote command and for the data transfer.
- Checks for the existence of the source and target datasets along with the availability of commands and enabled ZFS pool features on both local and remote systems.
- Gets the value of the syncoid:sync ZFS property on the source dataset. This property can be manually set on datasets to limit which hosts it will send the dataset to. Some tools make more extensive use of properties for configuration.
- Checks for existing snapshots on the source dataset. On the first run of this simple example, it found none. It also checks for resume tokens and can handle resumption of interrupted partial sends.
Creates a snapshot on the source dataset named with the client’s hostname and a timestamp. In this case, it would be something like: zroot/usr/home@syncoid_client.example.com_2025-01-19:20:06:40-GMT00:00.
- Simulates sending the snapshot with the -n option to zfs send. This serves only to estimate the size of the data transfer for the progress bar but can take a noticeable amount of time.
- It is now ready to send the snapshot. The remaining steps run as a Unix pipeline so it will execute in parallel. First is zfs send, which isn’t passed any options in this initial simple example.
- syncoid uses lzop for data compression. This is a file compressor that is designed for speed over compression ratio. The option to select alternative compression tools exists if you prefer.
- pv provides a progress bar, using the size acquired in step 6. The pv tool does also have some basic support for buffering, however…
- mbuffer adds buffering with a default block size of 128k and buffer size of 16M. Use of mbuffer is common, but other tools have different sizes either as their defaults or mentioned as examples in their documentation, so there isn’t any common consensus as to what sizes are a good choice. However, the sizes are invariably configurable.
- Via ssh to the remote host, the first step is the lzop decompression step piped to zfs receive. This includes options to enable resumption of an interrupted receive and for an initial rollback, which is necessary to ensure the receive is successful if the backup has changed since the last run. The likelihood of this happening can be minimized by setting the target dataset to be readonly, which is not done by default.
Optimizing Incremental Replication for Faster Transfers
Aside from the intended transfer of the dataset, one further outcome of these steps is that snapshots remain in place on both the source and destination. We need these so that a subsequent replication can transfer only the incremental changes. If we were to run the same manual replication command a second time it should complete more quickly. In terms of the steps taken, the first difference comes in step 3 where it would now find snapshots on both the local and remote system from the first transfer. Having found these, it then uses an incremental send with -I to update the remote system. Finally, as an additional step, syncoid destroys the initial starting snapshot on both the source and destination systems.
The basic steps outlined are much the same as any of the tools. I chose syncoid for this example because it is a comprehensive tool where the work is done using a combination of other tools like ssh.
Choosing a Connection Method for ZFS Replication
As it’s common to already have ssh available and configured, it’s a good default choice. It provides encryption, which is important anywhere the network is not private. However, if performance is a concern, then there are better options.
Making direct TCP connections is often done using tools like netcat (nc) and socat, which are well known and likely what you would expect to use. However, mbuffer is a common choice because it handles both network connections and buffering. When using a direct connection, we need to arrange for a process to be listening on the remote system. For this purpose, syncoid and znapzend use ssh to invoke a remote netcat. Meanwhile, zrepl has a long-running server process, so it is able to listen on a TCP socket directly. zrepl also supports the use of TLS for encryption, including features like TLS client certificates.
The use of a single ssh connection by syncoid is notable, especially if you expect to type in passwords or key passphrases manually. zrepl side-steps this issue by using a Go ssh implementation.
Optimizing Compression for ZFS Replication
As we saw earlier, syncoid explicitly uses lzop to compress the data stream. Most other tools don’t include a compression step, but it is generally possible to configure additional pipeline stages.
If you have compressed datasets, it is better to take advantage of this and retain the existing ZFS on-disk compression by adding -c to the send options used. Even in this case, there may be some advantage to using lzop during the transfer process. If your systems contain large numbers of small files, compression can improve efficiency. The replication stream format itself can be compressed in addition to the file data.
With encrypted datasets, it may be worth disabling any compression applied when sending those datasets because encryption breaks the patterns that compression relies on, meaning the data will not compress. You need the option -w to zfs send to send the encrypted data in raw form with the encryption retained. This has the added benefit that it is possible to back up encrypted datasets without ever decrypting the data or even having access to the key.
Snapshot Management in ZFS Replication
With replication, you may find that you have two sets of snapshots: the periodic snapshots covered in part one and the snapshots used as the starting point for transfers. Some tools, such as zxfer, don’t automatically retain snapshots. That is tidier but less convenient for later incremental transfers. The intention is to use it in combination with periodic snapshots.
It is common for tools to have functionality covering both automatic snapshots and replication as there are benefits in the two systems understanding each of their snapshots. Having the periodic snapshots available on the backup system may be convenient, but it doesn’t work well to create them there. An incoming replication has to roll back to the snapshot that marks the starting point of any changes (the newest common snapshot between the source and destination), wiping out any locally created snapshots in the process. ZFS replication can send intermediary snapshots which can include the periodic snapshots created on the source. Later removing these intermediary snapshots doesn’t cause any problems for a subsequent replication. However, you do need to run the periodic snapshot tool on the backup system with its configuration set to disable the creation of snapshots but enable pruning.
Methods and Best Practices for Configuration
There are three basic approaches to configuration:
- Command-line arguments
- A configuration file
- ZFS properties
Most of the simpler tools such as bzfs and zxfer take the first approach. Unlike the companion sanoid tool, syncoid doesn’t use a configuration file. This approach has the advantage of simplicity. Typically, you would use cron jobs with arguments giving the datasets to backup. In that list of cron jobs or a small wrapper script you essentially have a configuration file, just using sh syntax rather than something like yaml. It can be helpful that simple transfers can be tested from the command line. The tool is also usable for one-off transfers.
Examples of tools using a configuration file include pyznap and zrepl. It is slightly more complicated to get started with zrepl because a configuration file is needed for a single basic transfer. A configuration file is easy to maintain under version control and provides a full overview of the complete setup in a single place.
zap, zfs_autobackup, and znapzend use ZFS properties. Properties can be convenient as it is easy to add new datasets into backups and when a dataset is removed, the associated configuration is removed with it. Properties allow for an easier and more direct way to interrogate the configuration for a single dataset.
Advanced Features in ZFS Replication
If you are concerned about a particular feature that is associated with options to zfs send or zfs receive it is never difficult to ensure the necessary options are passed on. This includes things like whether to send intermediary snapshots or properties and handling compressed or encrypted datasets.
Snapshot Holds and Bookmarks
Some tools create holds on snapshots during the transfer. This helps ensure a successful transfer by keeping the common snapshot between the source and destination. Even if the transfer is interrupted and must be resumed, the snapshot remains intact. You may also see a feature for using a bookmark instead of a snapshot on the source system.
Snapshots and Storage Considerations
A snapshot consumes disk space whenever data is changed or deleted. This happens because snapshots retain a copy of the modified or removed data. As a result, you can still retrieve that data later.
A bookmark is only a reference to a point in time. With a bookmark, it is still possible to create a send stream to update a remote system. In the case where you deleted files, zfs send only needs to tell the remote system which data blocks to free. It isn’t necessary to retain the contents of the deleted files on the source system. The use case for this would be if your source system has limited available storage or if it is only possible to do the replication infrequently.
Resuming Interrupted Transfers
ZFS allows you to resume a transfer that was interrupted. This can be useful when resending the data following a rollback would take a long time. This is one area where OpenZFS and Solaris ZFS deviate so take care to follow the right documentation. Support for resuming interrupted transfers is absent in zxfer and znapzend.
Bandwidth Limiting and Logging
One feature advertised by zrepl is bandwidth limiting if you need to be considerate in the use of a shared network link. The pv command used by sanoid and bzfs has a --rate-limit option so the functionality is available with them and in principle also with any other tool that lets you insert custom pipeline commands.
One fairly ubiquitous feature that we’ve not mentioned until now is support for logging. This can be invaluable when a replication does fail. You want to make sure you get notified when backups are not completing correctly, so you can remedy it before it is too late.
Finding the Right Fit for Your ZFS Backup Strategy
There is no shortage of tools for handling both periodic snapshots and replication with ZFS. In terms of functionality, they have a lot of commonalities. I found all of the tools somewhat clumsy when dealing with backups to removable USB disks. More could be done to provide tooling around detecting which disks are currently attached. Often, tools are let down by the paucity of their documentation. But that isn’t a criticism I could make here. Other than the lack of man pages, several tools are well-documented.
The choice of tool primarily comes down to personal taste on factors like how they are configured or on the trade-off between a simple and understandable tool versus a more complex and featureful tool. zrepl is perhaps the most powerful of the tools while sanoid does a good job of being capable while still feeling simple and approachable. I like the convenient use of ZFS properties for configuration of zap and znapzend.
Embedded