Announcement

New LTS for FreeBSD is now available in an initial release  Learn More

Klara

Key Article Takeaways 

  • HDDs remain the economical choice for bulk storage, but ZFS pools should use CMR drives, not drive-managed SMR.

  • Enterprise SSDs are well suited for databases and VMs. Prioritize power-loss protection (PLP), endurance, and sustained write performance over peak benchmarks.

  • NVMe delivers low latency and high concurrency, but sector format, thermal limits, and PCIe lanes can constrain performance.

  • Each ZFS device role has different requirements. Pool storage, SLOG, L2ARC, and special vdevs call for different choices in performance, reliability, and redundancy.

Every storage performance engagement eventually arrives at the same question: what should we buy? It is usually asked first, and it should be asked last, because in a ZFS pool the right answer depends entirely on the job the device will do. ZFS can give a disk one of four possible jobs: pool storage, log device, cache device, or special vdev. The same drive can be the best choice for one of these roles and a liability in another. This article works through them role by role: what each job demands, which class of hardware fits it, and the specific traps in each class that cost people their weekends. 

HDDs for Capacity Storage 

For bulk capacity, spinning disks still win on price per terabyte, and a RAIDZ2 of 7200 RPM drives remains the right answer for backup targets, media, archives, and anything measured in "how many terabytes" rather than "how many IOPS." 

The hard rule: CMR only, never drive-managed SMR or host-managed zoned drives. Shingled recording overlaps tracks to gain density, which makes sustained random writes catastrophically slow, and a ZFS resilver is exactly that workload. This stopped being theoretical back in 2020, when SMR drives quietly shipped inside some NAS-branded drives: in ServeTheHome's testing, a RAIDZ resilver that takes under a day on CMR drives took over nine days on the DM-SMR model, and drives were dropping out of arrays badly enough that TrueNAS published a formal notice against them. Check the datasheet for CMR/SMR designation before buying, especially in the 2-8TB consumer range where SMR hides. ZFS is growing support for Host Managed SMR (HM-SMR) to properly cooperate with the drive and leverage these new 30+ TB drives, but support won’t land until OpenZFS 2.6 or later. 

Two smaller checks. First, sector size: modern drives are 4K-native or 512-emulated, and ZFS must know: 

$ zpool get ashift tank        # 12 = 4K sectors; set at vdev creation, permanent 

An ashift of 9 on 4K drives silently amplifies every write, and it cannot be fixed without rebuilding the vdev. We covered the damage this can cause in Misconfigurations Part 1. Second, there are the interfaces: SAS buys you dual-porting and better behavior in large backplanes, and one rule from the OpenZFS hardware guide is worth quoting: never place SATA disks into a SAS expander without an interposer.

Capacity also changes the redundancy math. A 20TB drive that takes a day or more to resilver is a day or more of degraded resilience, every time, and that window is why big-disk vdevs should be RAIDZ2 or Z3 rather than RAIDZ1: single parity plus modern drive sizes leaves you one unrecoverable read error away from a restore-from-backup afternoon. 

What spinning disks should never be used for are databases. A 7200 RPM drive delivers a couple hundred random IOPS, which can mean TPS (Transaction Per Second) rates that barely break three digits. The queue-depth metric covered in our article on ZFS storage metrics goes vertical and stays there. 

SATA/SAS SSDs and Power-Loss Protection 

Flash pricing has made SSDs the default for everything that isn't bulk capacity. The market further splits into two segments that look identical on the shelf: consumer and enterprise. The dividing line that matters for ZFS is power-loss protection, capacitors or another mechanism that guarantees the drive can finish writing what it acknowledged when the power goes away. PLP is effectively an enterprise-only feature, and it is the difference between a drive that can be trusted with sync writes and one that lies about flushes to look good on benchmarks. 

The other two datasheet lines worth watching for. The first is endurance, expressed as DWPD (Drive Writes Per Day) or TBW (Total Bytes Written), because a database or VM workload will chew through a consumer drive's write endurance budget years early. The second is sustained write behavior, because consumer QLC drives write quickly into an SLC cache and then fall off a cliff after only a few minutes, or even seconds, of sustained writes, sometimes below hard-drive speeds. This exact failure mode is what can turn a long resilver or bulk load into a week-long battle. Enterprise SATA in mirrors remains a perfectly reasonable design for mid-size databases and VM stores; it is slower than NVMe but dramatically better than its reputation suggests.

An important housekeeping note for any flash pool: set autotrim=on (or run zpool trim on a schedule) so the drives know which blocks are free. All SSDs will run slower as they near being full, and without TRIM, the drive doesn’t know which data you have deleted, so it will always think it is full. Enterprise drives lean on overprovisioned spare area for steady-state performance, and TRIM is how ZFS keeps things performant.

NVMe Performance and Hardware Considerations 

NVMe removes the protocol bottlenecks and adds concurrency that flash needs to meet its full performance potential: microsecond latencies, deep parallelism, and the best fit for the mirrors-of-fast-media layout databases want. There is some fine print, though, in descending order of pain: PLP is still required. An NVMe drive without power-loss protection is still the wrong device for a SLOG, no matter what the benchmarks say.

Format matters too: we recommend formatting NVMe media to 4096-byte sectors before giving the drives to ZFS. Form factor matters too: M.2 sticks thermal throttle under sustained load in ways U.2/U.3 drives with real heatsinks do not, so a datasheet benchmark achieved for thirty seconds is not what you end up seeing in production. Also, PCIe lanes are a limited resource: eight NVMe drives may want more lanes than many boards can give them, so check before you buy.

Once deployed, watch wear levels like any consumable: smartctl -a /dev/nvme0 reports percentage-used against the drive's rated endurance, and a mirror ageing evenly is a schedule rather than a surprise.

One expectation-setting note: ZFS's checksumming, copy-on-write, and caching were designed when disks were slow, and on very fast NVMe some of that overhead becomes visible. That is exactly the case OpenZFS 2.3's Direct IO sets out to resolve, letting applications that manage their own caching bypass the ARC. We'll cover when to use it, and when not to, in an upcoming article on OpenZFS 2.3 features.

Matching the Drive to the Job 

The answer key, role by role: 

Pool, capacity     CMR HDD 7200rpm, RAIDZ2       $/TB is the metric 
Pool, performance  Enterprise SSD/NVMe, mirrors  $/IOPS is the metric
SLOG               Small PLP NVMe                Latency is the metric;
                                                 rarely >16GB in use, buy small
L2ARC              Any decent read-optimized     Safe to lose, no PLP needed;
                   flash                         only after RAM is maxed
Special vdev       Enterprise NVMe, MIRRORED     Holds the only copy of your
                                                 metadata; match pool redundancy 

The SLOG needs low latency and PLP and almost no capacity, which is why a small enterprise drive beats a large consumer one at twice the price. If you are using metadata vdevs, starting with OpenZFS 2.4, and there is no dedicated SLOG device, ZFS will reserve a portion of your metadata vdevs to use as an “embedded special log”. This can avoid the need to source separate high-endurance, low-capacity NVMe devices, and simplify the server BoM. 

The L2ARC is the one role where cheap is fine, because it is only a cache. Losing it costs nothing but warmth of the cache. By contrast, the special vdev is the one role where cheap is fatal: it is not a cache, it holds the only copy of pool metadata, and it must be mirrored to the pool's redundancy standard, and use devices with PLP, a trap we covered in Misconfigurations Part 2. 

For the database systems we've spent the last three articles on, a proper reading of the metrics can tell you when the media has genuinely hit its limits. When they do, the answer is to buy mirrors of enterprise flash for the data, a small PLP NVMe for the log, and as much RAM as you can justify before either. Storage money spent in that order reduces the bottlenecks in the order they are most likely to appear. 

Planning Your ZFS Storage Hardware 

Hardware selection is where theory meets a purchase order, and it generates more questions than any article can answer. Which specific models, how wide to mirror, when Optane-class devices still make sense on the second-hand market, etc. Let Klara's engineers guide you through these decisions with our ZFS Storage Design service to select the right hardware and pool architecture for your workload. 

For existing ZFS deployments, our Quarterly ZFS Health Check evaluates pool layout, device wear, and performance against your current workloads. You'll receive a written report with recommendations ranked by risk and a quarter-over-quarter record to track changes before they become production issues. 

 

Back to Articles