Biggest ZFS Misconfigurations and How to Fix Them: Part 1

ZFS has a well-earned reputation for protecting your data. It checksums everything, it heals silent corruption, and it survives hardware that has no business still spinning. What it cannot do is protect you from your own pool design. Most of the ZFS problems we get called in to diagnose aren’t bugs. They are configuration decisions, made years earlier in about four seconds, that have been quietly taxing every read and write since. 

This is Part 1 of a two-part tour of the misconfigurations we see most often, why they hurt, how to detect them, and what fixing them actually takes. Fair warning: some of these can be fixed with one command, and some can only be fixed with a rebuild. That difference is the best argument for reading this before you create your next pool. 

1. The Wrong ashift: A Mistake You Keep Forever

ashift tells ZFS the physical sector size of a vdev, as a power of two: ashift=9 means 512-byte sectors, ashift=12 means 4K. It is set when the vdev is created, and it can never be changed. That would be fine if drives were honest, but a generation of 512e disks reported 512-byte logical sectors while writing 4K physically, and every pool that believed them got ashift=9. 

The cost is write amplification at the drive level: every 4K physical write serviced through 512-byte emulation, forever. Check what you actually have: 

$ zdb -C tank | grep ashift 
            ashift: 9        # on 4K media, this pool is hurting 

On FreeBSD, force the floor before creating anything: 

$ sysctl vfs.zfs.min_auto_ashift=12 
$ zpool create tank mirror da0 da1 

This can also be an issue when replacing disks. If you replace a failed 512e drive with a true 4K native drive that does not support emulation of 512- byte sectors, all of the unaligned writes will fail, and now you are either rebuilding your pool during a disk failure or shopping for old-stock drives that can deal with your misconfigured pool. 

The fix for an existing pool is the painful part: there isn’t one. You cannot reshape ashift in place. You build a new pool with correct ashift and migrate with zfs send/receive. If you’re buying disks today, assume 4K or larger, and if a vendor tells you their SSD prefers 8K pages, believe them and use ashift=13. 

2. recordsize That Fights Your Workload

Recordsize sets the maximum block size for a dataset; by default, it is 128K. For general file storage, the default is fine, because small files get small blocks automatically. The trouble starts when an application does small random I/O inside large files, which is exactly what databases do. 

MySQL’s InnoDB works in 16K pages. Put that datafile on a 128K-recordsize dataset, and every 16K read drags 128K off disk, and every 16K page update turns into a 128K read-modify-write. That is 8x amplification, per direction, you caused by not configuring anything. 

$ zfs create -o recordsize=16K tank/mysql     # InnoDB: 16K pages 
$ zfs create -o recordsize=8K  tank/pgsql     # PostgreSQL: 8K pages 
$ zfs create -o recordsize=1M  tank/media     # big sequential files 

For more on what the right recordsize is for different workloads, see our article on ZFS recordsize tuning. 

There are two things people miss. First, recordsize only applies to newly created files, so changing it on a live dataset does nothing for existing files until that file is recreated; a one-time copy or a dump-and-restore completes the fix. Second, this is a per-dataset property. The whole point of datasets is that the database, the VM images, and the media library each get their own tuning. One dataset per workload is the habit that prevents this class of problem entirely. 

The same logic applies to zvols, where the property is volblocksize, and it is fixed at creation time. Match it to the guest filesystem’s block size when you carve the zvol, because the only way to change your mind later is to create a new zvol and copy the data across. 

3. One Wide RAIDZ vdev Where Mirrors Belonged

ZFS distributes writes across top-level vdevs, and random IOPS scale with the number of vdevs, not the number of disks. A single 12-disk RAIDZ2 has a lot of capacity and roughly the random-I/O profile of one disk. Twelve disks as six mirror pairs have six vdevs pulling I/O. Same shelf, radically different behavior under load. 

We see the wide-RAIDZ pool most often where someone optimized for the capacity number in the purchase order and then wondered why the VM cluster on top of it crawls. Even within RAIDZ there is room to be smart: Klara’s ZFS pool layout article works the example of 12 disks as four 3-disk RAIDZ1 vdevs versus two 6-disk RAIDZ2 vdevs; similar parity overhead, twice the vdevs, meaningfully better IOPS. 

The rule of thumb: mirrors for random I/O (databases, VMs), RAIDZ for capacity-oriented sequential work (backups, media, archives), and never one giant vdev because the spreadsheet liked it. Fixing a layout means rebuilding the pool or adding correctly shaped vdevs and letting new writes rebalance, so this is another decision to get right on day one. 

4. Dedup, Enabled Optimistically 

For years, zfs set dedup=on was the most expensive single command in storage. Every unique block gets an entry in the deduplication table, and that table is consulted on every single write. While the DDT fits in RAM, you lose some performance. When it stops fitting, every write turns into random reads of table entries from disk, and pool performance falls off a cliff. We have seen pools where the only realistic recovery was migration to new hardware. 

$ zpool status -D tank      # shows DDT size and where it lives 
$ zfs get compressratio,dedup tank 

Three rules. First, dedup is per-dataset, so if you must use it, scope it to the dataset that actually contains duplicate data, never the whole pool. Second, disabling dedup only affects new writes; existing blocks stay in the DDT until rewritten, so the penalty remains even after you attempt to correct it. Third, do the math before, not after: if your data doesn’t contain substantial duplication, you’re paying the overhead for nothing, and compression (which is nearly free with LZ4 or zstd) already captures most easy savings. 

The story is improving: OpenZFS 2.3 shipped Fast Dedup, which Klara helped build, with a smaller table and the ability to prune single-copy entries. It moves dedup from “almost never” to “viable when the data justifies it.” That last clause still does all the work. 

5. Running the Pool Full 

ZFS is a copy-on-write filesystem. Every write, including a modification of existing data, needs fresh free space to land in. As the pool fills, finding contiguous free space gets harder, allocations get slower, and fragmentation climbs; a pool in the high-90s can slow to a crawl and stay slow, because freed space comes back fragmented. 

$ zpool list -o name,size,alloc,cap,frag,health 
NAME   SIZE  ALLOC  CAP   FRAG  HEALTH 
tank   43.5T 39.7T  91%   61%   ONLINE   # this pool needs help today 

Guidance varies by workload, but treating roughly 80% as the planning threshold keeps you out of the failure mode. Enforce it, don’t just intend it: 

$ zfs create -o refreservation=10T tank/reserved   # reserve the last 10T of space 
$ zfs list -o space -s used -r tank | tail         # find what's eating the pool 

And check snapshots before blaming users: a dataset showing modest USED can be pinning terabytes in snapshots nobody remembers taking. 

What’s in Part 2 

Five more, including the SLOG that people buy as a “write cache” and the sync=disabled setting that makes benchmarks fast and auditors nervous, plus L2ARC sizing, special vdev redundancy, and snapshot hygiene at fleet scale. Until then: check your ashift before your next migration window, not after.