OpenZFS Horror Stories

Join the Klara team including co-founder Allan Jude and solutions engineer JT Pennington as we tell scary Halloween stories of close calls with data loss. Learn what not to do, and just how important having good, tested backups can be. 

As ZFS data recovery experts, Klara is often called in when things have gone horribly wrong. In the spirit of Halloween, the team shared stories of some of the close calls with the data grim reaper. 

The stories we told:

• The Shrinking (zvol) 

• A Nightmare on PXE Street 

• Multi the Mount Slayer 

• The Crossed Wires Massacre 

Top Burning Questions from the Session—Answered!

We handpicked two of the most thought-provoking questions from the session, with expert answers from Allan and JT:

🗨️  Would a zpool checkpoint have prevented data loss in the first story where the zvol was shrunk?  How?
      The zpool checkpoint feature saves the state of the pool when it is created, as well as deferring the reuse of all sectors on the disks. This means that rewinding the pool to the zpool checkpoint can undo almost any operation, including zpool upgrade, destroying a dataset, resizing a zvol, or almost anything else.

It is best practice to create a checkpoint before performing any pool-level maintenance. However, checkpoints have some downsides. Firstly, you can only have a single checkpoint, and while that checkpoint exists, no space is ever reclaimed or reused, so a pool will quickly run out of space. Checkpoints are not meant for long-term use but for protection during maintenance. 

The second downside is that if you do have to resort to rewinding to a checkpoint, all changes of any kind made to the pool after the checkpoint are undone. There is no way to choose which changes to roll back, as you can at a per-dataset or even per-file level with ZFS snapshots.

🗨️  How many of your customers are using openzfs on Linux vs. FreeBSD?
      Klara sees about an even mix of customer using OpenZFS on FreeBSD and Linux. The team has extensive experience supporting ZFS on all platforms, including Redhat distros (RHEL, Rocky, and Alma Linux), Ubuntu and Debian, FreeBSD, TrueNAS Core and SCALE, and even OmniOS and other illumos distros.

Date: October 31, 2024

Time: 1:00 PM EDT 

Duration: 45 minutes.

Know someone else who might be interested? Share this entry with them!