Key Article Takeaways
- Annual DR testing leaves too much room for unnoticed changes. OpenZFS upgrades, OS updates, and replication script changes can affect recovery between tests.
- Real-world failures often remain hidden until recovery is attempted. Klara has encountered inaccessible encrypted replicas, six-week replication gaps, and a four-day restore against an eight-hour RTO.
- Quarterly drills expose weaknesses before an incident. Testing a different dataset each quarter helps validate recovery, measure actual restore times, and identify necessary DR improvements.
Your ZFS replica is probably the most trusted dataset you own and the least tested. zfs send | zfs receive runs every hour, the job exits 0, and the dashboard stays green. Nobody mounts the copy. Nobody loads its keys. Nobody times how long it takes to bring a database back online given just the replica dataset.
When we ask how often the restore path gets exercised, the usual answer is once a year, for the audit. That was a thin schedule five years ago. Today the systems on both ends of the replication change every month, and the incidents your DR scheme was designed to guard against unfold in minutes.
Why Once a Year Stopped Working
Every OpenZFS release, OS update, firmware flash, and edit to the replication script touches the restore path. The primary keeps running, so nothing alerts. The breakage shows up when you need the replica. Change your systems monthly and test recovery yearly, and you are carrying eleven months of untested changes into every incident.
The pace has only gotten faster now. CrowdStrike's 2026 Global Threat Report puts average eCrime breakout time at 29 minutes, and the fastest at 27 seconds. Anthropic's Project Glasswing used a frontier model to find thousands of zero-days this year, in every major OS. That means more patches, and more change to every one of the systems your recovery depends on.
The malware or malefactor deleting your data may also hold valid credentials. In July 2025, an AI coding agent at Replit deleted a production database during a code freeze. On ZFS, an agent with destroy privileges that decides a pool at 92% needs more free space can run:
zfs destroy -r tank/db@%
That removes every snapshot of the dataset, including the incremental base your replication depends on. The next zfs send -i finds no common snapshot on the target, and you are looking at a full resend while your RPO gets stretched.
How a ZFS Restore Path Breaks Quietly
None of these issues trips an alert on the primary. All of them only surface the day you need the replica.
- The incremental base disappears. A pruning policy that removes snapshots as they age without considering if they are replicated, a cleanup script, or an agent removes the last snapshot both sides share. Now replication is broken, and the only way to fix it is with a force flag, removing the entire dataset from the DR side and resending it. You can prevent this by putting a hold on any critical snapshots with zfs hold repl tank/db@base, or create a bookmark with zfs bookmark tank/db@base tank/db#base. An incremental can be sent from a bookmark even after the snapshot itself is gone.
- Feature flags drift. Every time you look at zpool status, it is nagging you about unactivated features. But when you run zpool upgrade on the primary, it enables features that the DR host's OpenZFS release cannot understand. Pin both pools with the compatibility property, and upgrade the DR side first. This ensures everything works, with minimal impact on production, and ensures that once production is upgraded, there is no gap where the DR host doesn’t understand the new feature.
- Properties don't travel. A plain zfs send of a snapshot carries data, not dataset properties. Without -p or -R, the replica keeps its own compression, encryption, caching, and mountpoint settings, and the restored database runs on settings production has never used.
- The replica isn't pristine. Someone mounts it to check a file and writes to it. The next incremental receive fails until you use -F, which rolls those changes back. Prevent this by applying best practices: set readonly=on on the replica and leave it there.
- Permissions grow. The replication account picked up destroy during a migration, and nobody took it away. Run zfs allow tank/db and audit each permission the account has, and make sure it has a good reason to have it. More granular permissions are available in newer OpenZFS, including send:raw and send:encrypted that can be used to ensure that only the encrypted versions of datasets are replicated, preventing any accidental leak of sensitive data.
What We've Seen When the Drill Finally Runs
The Replica Nobody Could Unlock
Encrypted datasets were replicated with raw sends (zfs send -w), which is the right call: the data stays encrypted in transit and at rest on the target. The wrapping key lived in a keyfile on the primary, on the same pool it protected. When the primary was gone, the replica was intact and unreadable. zfs load-key had nothing to load. Make sure the key exists somewhere beyond the production environment, where the primary node's failure cannot impact it. Also, you should have a break-glass copy of the key somewhere else as well. This is the type of issue you will only find while running a drill.
Replication That Stopped in August
An OS upgrade via repave on the target changed the SSH host key. The replication job failed on every run after that, and the error went to a log nobody read. The business had signed off on a one-hour RPO. The newest snapshot on the replica was six weeks old. Comparing the newest snapshot on each side, which only takes a single command, would have caught it the first day.
The Restore That Worked, Slowly
A 40 TB pool replicated cleanly to an offsite target. The drill restored a dataset to an isolated host, and the data was perfect. Then someone did the math for the whole pool. Sending 40 TB back over a 1 Gbit/s link at a realistic 110 MB/s takes about 101 hours, a little over four days. The documented RTO was eight hours. Nothing was broken. The plan was simply impossible, and only a timed restore drill made that visible. The usefulness of a drill is proportional to its realism. Conduct your drill under the same circumstances that you will have to run the real thing. No peeking at the production host or its config. No access to the internal wiki. Everything is down, and you are building from scratch.
The Quarterly ZFS Drill
We recommend running these drills every quarter, and again whenever either end of the replication gets an OpenZFS or OS upgrade. Those are the changes that break a restore path without anyone noticing.
Start by picking one production dataset and writing down the RPO and RTO you've promised for it. Pick a different one next quarter. The dataset that restores in five minutes rarely tells you anything; pick something important.
Then check how far behind the replica really is. Run these commands on both sides and compare the newest snapshot:
zfs list -t snapshot -o name,creation -s creation tank/db | tail -1
If the gap is bigger than your RPO, stop and fix replication before going any further.
Now do the restore on the DR host, from the replica. Start a timer to see what your real RTO actually is. A clone of the newest replicated snapshot moves no data and exercises the copy you would actually recover from:
zfs load-key drpool/db
zfs clone drpool/db@hourly-1019-2300 drpool/drill
Load that key from wherever you escrow it. If the only copy lives on the primary, you've found your first problem.
Point the application at the clone. Only stop the timer when the application answers real requests. A successful mount doesn't count. Now we have the real number for your RTO for this dataset.
While that runs, check the protections on the replica. Run zpool scrub drpool and read zpool status -v drpool once it finishes. zfs holds on your replication base snapshot should show the holds you have set up. zfs allow tank/db should not show the replication account having the destroy permission or anything else beyond what it needs.
Then do the full-pool math. Divide zfs get -Hp used tank by the real throughput you measured on the replication link. If that number is bigger than your RTO, more testing won't fix it. The design has to change.
Write down the date, the dataset, the replication lag, the real restore time and everything that broke or went less smoothly than it should have. Then zfs destroy drpool/drill and put the next drill on the calendar.
ZFS DR Testing Takeaways
- Test recovery at least as often as you change the systems it depends on. For most environments, that means at least every quarter.
- Restore from the replica. A local snapshot proves the snapshot works. It says nothing about your DR copy. Keep that copy readonly=on.
- Keep encryption keys for raw-send replicas somewhere the primary's failure cannot disrupt.
- Alert on replication lag, not just on job failures. Compare the newest snapshots on both sides, and alert if the creation time of the newest snapshot on the DR side falls behind your RPO.
- Protect replication base snapshots with holds or bookmarks, and keep destroy away from replication accounts, scripts and AI agents.
- Time the restore and do the full-pool math. An RTO you have never measured is just a guess, not a plan.
Our DR Drill Checklist puts these steps on one page. If you would rather have our engineers run it with you, the Quarterly ZFS Health Check includes a timed restore. Contact us at klarasystems.com.
.
