OpenZFS and storage in general is a complex and important part of any project’s architecture. It should be planned thoughtfully and ideally, ahead of time! In this article, we’ll talk about how to understand, measure, and plan for your storage performance needs.
Advanced ZFS Snapshots
Earlier in this series, we discussed the Basics of ZFS Snapshot Management: how to create OpenZFS snapshots, restore files from a snapshot, and delete snapshots. Today’s article dives a bit deeper into OpenZFS snapshot management with snapshot holds, clone creation and promotion, and assigning permissions to snapshot-related operations.
On a FreeBSD 13.0 test system, I created a series of snapshots of the home directory in the default zroot pool. Here is a listing of the available snapshots:
zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT zroot/usr/home@snapshot1 80K - 96K - zroot/usr/home@snapshot2 72K - 1.99M - zroot/usr/home@snapshot3 64K - 2.03M - zroot/usr/home@snapshot4 64K - 2.00M - zroot/usr/home@snapshot5 512K - 260M - zroot/usr/home@snapshot6 0B - 259M -
By default, as the root user, I can destroy any of these snapshots. Here I delete and verify the deletion of snapshot1:
zfs destroy zroot/usr/home@snapshot1 zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT zroot/usr/home@snapshot2 72K - 1.99M - zroot/usr/home@snapshot3 64K - 2.03M - zroot/usr/home@snapshot4 64K - 2.00M - zroot/usr/home@snapshot5 512K - 260M - zroot/usr/home@snapshot6 0B - 259M -
And here goes snapshot4:
zfs destroy zroot/usr/home@snapshot4 zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT zroot/usr/home@snapshot2 72K - 1.99M - zroot/usr/home@snapshot3 72K - 2.03M - zroot/usr/home@snapshot5 520K - 260M - zroot/usr/home@snapshot6 0B - 259M -
But what if I have a snapshot that I don’t want even the root user or an errant deletion script to destroy? Perhaps that snapshot represents a baseline point-in-time that I might wish to return to. Enter zfs-hold(8): Hold a snapshot to prevent it being removed with the zfs destroy command.
To create a hold, specify a name for the hold (known as a tag) and the name of the snapshot. Here I’ll create a hold tag of keepme on snapshot3:
zfs hold keepme zroot/usr/home@snapshot3
You can use any value you want for the tag that reminds you why you created the hold, as long as the snapshot doesn’t already have a tag using that same value.
To see the list of holds on a snapshot, specify the snapshot name:
zfs holds zroot/usr/home@snapshot3 NAME TAG TIMESTAMP zroot/usr/home@snapshot3 keepme Thu Aug 19 11:04 2021
Now that this snapshot has a hold, I am unable to delete the snapshot:
zfs destroy zroot/usr/home@snapshot3 cannot destroy snapshot zroot/usr/home@snapshot3: dataset is busy
If I no longer need to hold this snapshot and actually want to delete it, I’ll need to first release its hold. To do so, specify the name of the hold tag and the name of the snapshot:
zfs release keepme zroot/usr/home@snapshot3 zfs holds zroot/usr/home@snapshot3 NAME TAG TIMESTAMP
Now the snapshot deletion will succeed:
zfs destroy zroot/usr/home@snapshot3 zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT zroot/usr/home@snapshot2 140K - 1.99M - zroot/usr/home@snapshot5 528K - 260M - zroot/usr/home@snapshot6 0B - 259M -
A snapshot, by definition, is a read-only copy of a file system. This is a good thing for file restoration purposes as it allows you to access a file’s version from a known point in time. However, there are times when it is useful to have a read-write copy of a file system. zfsconcepts(8) defines a clone as: a writable volume or file system whose initial contents are the same as another dataset. As with snapshots, creating a clone is nearly instantaneous, and initially consumes no additional space.
As an example use case, consider a dataset containing a directory of virtual images that are each 2 GB in size and which are used for testing by multiple users. If each tester copies an image to their testing directory whenever they need to perform a test, the amount of consumed storage will increase by 2 GB for every test. Contrast that with instead creating a clone for each user. Since clones are created from snapshots, there is a known point in time, a definite benefit in testing. Better yet, the only storage consumed will be any edits made to the clones. This more efficient use of storage can apply to any use case where multiple users are working on large files (another example would be a directory containing videos that need to be edited for production). And, any edits to the clone are independent of the snapshot (your original baseline) as well as the “production” file system the snapshot is based on.
It’s important to note that clones can only be created from a snapshot and that a snapshot cannot be destroyed as long as it has any clones. Interestingly, clones do not inherit the properties of the snapshot, meaning that different properties can be specified when creating the clone. The clone can be located anywhere in the ZFS hierarchy as long as it is in the same pool.
To create a clone from a snapshot, specify the snapshot name followed by the clone location and name. In this example, I’ll verify which snapshots still exist before creating a clone of snapshot5 named clone1 and locate it on the tmp file system in the zroot pool:
zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT zroot/usr/home@snapshot2 140K - 1.99M - zroot/usr/home@snapshot5 528K - 260M - zroot/usr/home@snapshot6 0B - 259M - zfs clone zroot/usr/home@snapshot5 zroot/tmp/clone1
Since a clone is a readfully writable file system, zfs list doesn’t distinguish between file systems and a cloned snapshot:
zfs list NAME USED AVAIL REFER MOUNTPOINT zroot 1.17G 11.9G 96K /zroot zroot/ROOT 932M 11.9G 96K none zroot/ROOT/default 932M 11.9G 932M / zroot/tmp 96K 11.9G 96K /tmp zroot/tmp/clone1 0B 11.9G 260M /tmp/clone1 zroot/usr 260M 11.9G 96K /usr zroot/usr/home 260M 11.9G 259M /usr/home zroot/usr/ports 96K 11.9G 96K /usr/ports zroot/usr/src 96K 11.9G 96K /usr/src zroot/var 632K 11.9G 96K /var zroot/var/audit 96K 11.9G 96K /var/audit zroot/var/crash 96K 11.9G 96K /var/crash zroot/var/log 152K 11.9G 152K /var/log zroot/var/mail 96K 11.9G 96K /var/mail zroot/var/tmp 96K 11.9G 96K /var/tmp
However, the origin property will indicate that it is a clone and give the value of the parent snapshot:
zfs get origin zroot/tmp/clone1 NAME PROPERTY VALUE SOURCE zroot/tmp/clone1 origin zroot/usr/home@snapshot5 -
Now that I have a clone, let’s see what happens if I try to delete the parent snapshot, snapshot5:
zfs destroy zroot/usr/home@snapshot5 cannot destroy 'zroot/usr/home@snapshot5': snapshot has dependent clones use '-R' to destroy the following datasets: zroot/tmp/clone1
The error message clearly indicates that the dataset (the clone) needs to first be destroyed. Once I am finished with the clone, I can delete the clone and then successfully delete the parent snapshot:
zfs destroy zroot/tmp/clone1 zfs destroy zroot/usr/home@snapshot5 zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT zroot/usr/home@snapshot2 140K - 1.99M - zroot/usr/home@snapshot6 0B - 259M -
But what if you like the changes in the clone more than the data in the original snapshot? A common use case for creating clones is to test changes, and it is desirable to push them into production once testing is complete.
In this case you can promote the clone which reverses the parent-child dependency relationship. In other words, the clone is now a “real” file system and the original snapshot is now a clone with all the old snapshots intact.
Let’s see this in action by creating a clone on one of the remaining snapshots:
zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT zroot/usr/home@snapshot2 140K - 1.99M - zroot/usr/home@snapshot6 0B - 259M - zfs clone zroot/usr/home@snapshot2 zroot/tmp/promote-me zfs destroy zroot/usr/home@snapshot2 cannot destroy 'zroot/usr/home@snapshot2': snapshot has dependent clones use '-R' to destroy the following datasets: zroot/tmp/promote-me
So far, so good. Let’s see what happens when I promote the promote-me clone:
zfs promote zroot/tmp/promote-me zfs get origin zroot/tmp/promote-me NAME PROPERTY VALUE SOURCE zroot/tmp/promote-me origin - -
As expected, promote-me is no longer dependent on snapshot2. However, this might surprise you:
zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT zroot/tmp/promote-me@snapshot2 0B - 1.99M - zroot/usr/home@snapshot6 0B - 259M -
snapshot2 now resides on the zroot/tmp/promote-me file system. Is there a remaining dependency? Let’s check:
zfs get origin NAME PROPERTY VALUE SOURCE zroot origin - - zroot/ROOT origin - - zroot/ROOT/default origin - - zroot/tmp origin - - zroot/tmp/promote-me origin - - zroot/tmp/promote-me@snapshot2 origin - - zroot/usr origin - - zroot/usr/home origin zroot/tmp/promote-me@snapshot2 - zroot/usr/home@snapshot6 origin - - <snip>
Now things look much better, don’t they? Enabling compression is good in most cases. The same test can be done changing the recordsize. Let’s try with a small one just to see how one single setting can affect space efficiency in a dramatic way. I will disable compression to better understand the numbers:
root@geroda:/testpool # zfs set compression=off testpool root@geroda:/testpool # zfs set recordsize=4K testpool root@geroda:/testpool # /root/test_recordsize.sh 1 K size -> 3 K alloc 2 K size -> 3 K alloc 3 K size -> 5 K alloc 4 K size -> 5 K alloc 5 K size -> 13 K alloc 6 K size -> 13 K alloc 7 K size -> 13 K alloc 8 K size -> 13 K alloc 9 K size -> 17 K alloc 23 K size -> 29 K alloc 24 K size -> 29 K alloc 25 K size -> 33 K alloc 31 K size -> 37 K alloc 129 K size -> 137 K alloc 254 K size -> 264 K alloc 255 K size -> 264 K alloc 256 K size -> 264 K alloc
And what happens if I try to delete snapshot2?
zfs destroy zroot/tmp/promote-me@snapshot2 cannot destroy 'zroot/tmp/promote-me@snapshot2': snapshot has dependent clones use '-R' to destroy the following datasets: zroot/usr/home@snapshot6 zroot/usr/home
Well, I probably don’t want to destroy zroot/usr/home… This is what zfs-promote(8) means when it says: the origin file system becomes a clone of the specified file system.
Before creating a clone that you eventually plan on promoting to be a “real” file system, you want to give careful thought to where that file system should live. Example 10 in zfs(8) gives a real-world example of replacing a production file system with a testing clone that is now ready for running in production. Note that the example uses zfs rename to change file system names between beta, legacy, and production after performing the promote. You will want to follow a similar workflow in order to differentiate which file system is which when changing to a new production file system and eventually destroying the legacy file system.
Basics of ZFS Snapshot Management gives several examples of assigning a particular user specific permissions to ZFS operations on a specified file system. To see if any non-root users or groups have been assigned permissions to ZFS operations, specify the file system to check. Here I’m checking ZFS operations permissions on the zroot pool of a default FreeBSD installation:
zfs allow zroot
If I only get the prompt back, no permissions have been assigned and most ZFS operations can only be performed by the root user. The following permissions are related to snapshot operations:
Permissions can be granted (with allow) and revoked (with unallow) to users and groups. Keep in mind that ZFS operations are powerful and act on data, one of your most valuable assets. Be judicious when allowing operational access and periodically review permissions to determine if they are still appropriate. Refer to zfs-allow(8) for syntax guidance.
This article covered some advanced OpenZFS features that can help you get even more out of your snapshots. However, if your needs are more complicated and demand qualified services, the experts at Klara are available to help you with your OpenZFS support.
Like this article? Share it!
Data security is about preventing data from being disclosed, ensuring that only the correct people can access it. Data integrity ensures the data is correct, that it has not become corrupt due to hardware failure or other issues. With ZFS, you can get both.
If you’re getting ready to close the year, we’ve got you covered with some of the best content that we put out in the past year. Check out our top ZFS and FreeBSD content from 2022 and go down the open source rabbit hole for the holidays!