RAIDZ Expansion in Practice: What It Solves and What It Doesn’t

For most of ZFS's life, RAIDZ had one answer to "can I add a disk?" and that answer was no. You sized the vdev on day one, and growing the pool later meant buying a whole second set of disks for a new vdev or destroying and rebuilding the pool. RAIDZ expansion, developed by Matt Ahrens and Don Brady with sponsorship from the FreeBSD Foundation and TrueNAS, shipped in OpenZFS 2.3. It finally changed that answer. It is a genuinely important feature, and as happens with every other major feature, it is already being asked to do things it was never designed to do. 

Let's walk through how it works, where it excels, where it doesn’t, and what to do in each of those cases. 

How Expansion Actually Works

Expansion attaches one new disk to an existing RAIDZ vdev: 

$ zpool status tank 
        raidz1-0    ONLINE 
          da0       ONLINE 
          da1       ONLINE 
          da2       ONLINE 
          da3       ONLINE 

$ zpool attach tank raidz1-0 da4     # 4-wide raidz1 becomes 5-wide 
$ zpool status tank                  # watch the reflow progress 

Behind the scenes, ZFS reads every allocated block in the vdev and rewrites it across the new, wider set of disks. This "reflow" runs while the pool stays online and serving I/O; it resumes across reboots, and redundancy is maintained throughout: a RAIDZ2 can still lose two disks mid-expansion. When it completes, new writes use the new full width, offering a larger data-to-parity ratio. You can repeat the process later, one disk at a time, to grow a pool over and over. 

One prerequisite: the pool must have the feature flag available, so pools created before 2.3 need an upgrade first: 

$ zpool upgrade tank        # enables new feature flags, one-way as always 
$ zpool get feature@raidz_expansion tank 

That's the whole interface. The subtleties are in what the reflow does and does not change during the reflow. 

What It Solves

Expansion is built for pools that need to grow gradually. The classic case is the small deployment: the home lab, the office server, the edge box that started as 4 disks because that's what the budget allowed. Before 2.3, that pool's owner faced the list of undesirable options: buy another full set of disks for a second vdev, or backup, destroy, and rebuild. Now the answer is one disk, one command, no downtime. Now you can stop paying up front for capacity you won't need for two years, and disk prices hopefully move in your favor while you wait. 

It also removes a planning trap. "How wide should I build?" used to be a permanent decision made at the very start, when you have the least information you'll ever have. With expansion, starting at 5-wide and growing to 8-wide over three years is a legitimate strategy rather than a dreaded rebuild. 

What It Doesn't Solve

Old data keeps its old parity ratio. The reflow redistributes existing blocks across the wider stripe, but it does not recompute them at the new data-to-parity ratio. A 5-wide RAIDZ2 stores blocks as 3 data + 2 parity; expand the vdev to 6-wide, and those blocks are still 3+2. Only newly written blocks use 4+2. So while you will gain most of the additional space from the newly added drive, the pool won’t have the same capacity as a freshly made 6-wide RAIDZ2. The capacity gain from expansion will arrive gradually, as old data gets rewritten during its normal lifecycle, and the pool's space accounting stays conservative in the meantime. If you expanded specifically because you're low on space, know that you get the new disk's worth of raw capacity, not a retroactive efficiency upgrade on everything you've already stored. 

If you want the full ratio now, rewrite the data yourself. Until 2.3.4 and 2.4.0 there was no rebalance command, so you had to do this manually: 

$ zfs snapshot -r tank/data@reflow 
$ zfs send -R tank/data@reflow | zfs receive tank/data2 
# verify, swap mountpoints, destroy the original 

Do this dataset by dataset, and of course this assumes you have enough free space to store that dataset twice during the transfer. It only makes sense to do this if the reclaimed space will actually matter. 

With newer versions of ZFS, you can use the rewrite command to walk through a directory of files and rewrite them in-place. This can be done on files one by one, or on an entire directory. This has the advantage of not requiring enough free space toduplicate the dataset during the rewrite. The downside is that this rewrite, similar to doing it with rsync, will explode your snapshot size. All the rewritten data is “new”, even though it is identical to the existing data. It will have the new more efficient stripe width, but you’ll have to sacrifice your older snapshots to avoid using double the space.  

The RAIDZ level is fixed. Expansion changes the stripe width, but never the parity. A RAIDZ1 stays a RAIDZ1 at 4-wide, at 7-wide, at 10-wide, and that is important to keep in mind. A 10-wide single-parity vdev of large modern disks is a long resilver standing between you and data loss. The failure tolerance you chose on day one is the one you keep. The solution is to choose parity for the width you intend to reach in the future, not the width you start at: if the plan is to grow past 6 or 7 disks, start with RAIDZ2 even though it starts at 4 disks. 

Your IOPS don't improve. As the vdev gets wider, the data is distributed across the full stripe width. That means random I/O performance is still roughly one disk's worth per RAIDZ vdev, exactly as it was before expansion. If the problem you're solving is a sluggish VM store or database, expansion will hand you more terabytes of the same disappointment. As discussed in Part 1 of our ZFS misconfiguration series, that workload REALLY wanted mirrors, and the fix is rebuilding the pool with mirror vdevs, not widening the existing vdevs. 

It only goes one direction. There is no shrink, no disk removal from a RAIDZ vdev, and no undo. Attach the wrong disk (wrong size, wrong pool, mistyped vdev) and your options narrow immediately. Read the command twice; zpool attach on a RAIDZ vdev is one of those commands that deserves the same respect as ashift. 

The reflow is real work. Reading and rewriting every allocated block takes time proportional to how much you've stored, and it competes with your production I/O the entire time. On a mostly full pool, it can take days to complete. Two rules to follow: expand before you're desperate (a pool at 95% is a miserable place to run a reflow, and the fill-level warnings from the ZFS Misconfigurations and How to Fix Them: Part 1 article apply double), and schedule the start for a quiet period even though the pool stays online. 

The Decision Table

Expansion is the right tool when growth is incremental; the workload is capacity-oriented (media, backups, archives), and you're adding headroom before it's urgent. 

If any of the following apply, RAIDZ expansion is probably not the right tool: 

Need more random IOPS        -> add mirror vdevs, or rebuild the pool as mirrors 
Need to change RAIDZ level   -> zfs send to a new pool, then destroy the old one 
Need to shrink or remove     -> send/receive migration; there is no remove/shrink 
Pool already at 90%+         -> add a whole new vdev instead, then expand later 
Fleet-scale, planned growth  -> size vdevs up front; expansion is for gradual drift 

In larger fleets with procurement cycles and planned refreshes, adding whole vdevs remains the better pattern: it adds IOPS along with capacity and involves no reflow. Expansion shines exactly where planning can't: the pool that grew past its forecast, or the budget that arrives one disk at a time. 

Verdict 

RAIDZ expansion does one thing, does it safely, and does it while the pool stays online, which is more than a decade of forum threads dared to hope. Treat it as what it is: a width adjustment for healthy, capacity-oriented pools, with a well-documented asterisk about old data's parity ratio. Also treat it as what it isn't: a performance fix, a parity upgrade, or an undo button, and it will never disappoint you. If you're creating a new pool today, the old advice still applies: choose a pool layout based on the usage demands and let expansion handle the required capacity drift over time.  

If you need help deciding how to design a pool for future expansion, consider reaching out to us about our Storage Design service. We have the experience to help you make the right trade-offs.