Designing OpenZFS Storage for Independence: Pool Architecture, Failure Domains, and Migration Paths

Designing a storage system that is truly independent of vendor lock-in at every level and provides real data sovereignty without exposure to foreign lawful intercept and extraterritorial access risks requires considering the four key facets of storage independence: 

• Lifecycle and Operational Control 

• Component Choice 

• Interoperability 

• Exit and Portability 

Lifecycle and Operational Control 

Open source software is a fundamental building block of any independent system. Having access to the source code of the critical components of the storage stack provides much greater control over the lifecycle of the system, and how you operate it. With proprietary vendor solutions, all your long-term plans can be invalidated at the whim of the vendor, based on their business decisions rather than your own. With an open source license, designed to guarantee your rights, rather than an expensive vendor license designed to limit them, you have great control over when and how often to upgrade your storage stack. Some vendor licenses are so restrictive, they mandate how you operate your system, and restrict who is allowed to access or service them. If you vendor is blocking you from getting a second opinion, how much should you trust their advice? 

Component Choice 

Choosing OpenZFS as the core of your storage system keeps your options open. Starting with CPU architecture, ZFS was designed to make hardware transitions as seamless as possible, with an original design goal to support easy migration from SPARC to Intel based CPUs. In today’s landscape, that means a choice between x86_64, ARM64, PowerPC, and the upcoming RISC-V. Once you’ve settled on a CPU architecture, ZFS is also available across a plethora of operating systems, including FreeBSD, Linux, Solaris derivatives including OmniOS, and preproduction support for MacOS and Windows. This hardware and operating system flexibility is central to OpenZFS storage independence, preventing architectural decisions from becoming long-term constraints. 

Lastly, ZFS can build your storage infrastructure atop whatever storage hardware you choose, including whatever mix provides the right trade-offs for your workload and budget. ZFS contains decades of optimizations to get the most out of spinning drives of all types, from low power archival drives to the highest performance SATA and SAS drives on the market. ZFS has long supported using solid state flash media to accelerate primarily HDD based storage, and fully supports pure flash arrays. The value of being able to mix and match HDDs, SSDs, and NVMe devices to maximize benefit while minimizing cost should not be underestimated. 

The value in having the flexibility to mix and match HDDs, SSDs, and NVMe devices maximizes benefits while minimizing OpenZFS storage dependence on any specific hardware vendor or media type should not be underestimated. 

ZFS replication also opens up options for linear tape. Serializing the filesystem into a stream that can be written to tape and reconstituted later, including incremental updates without requiring access to the previously written tapes, adds yet another tool to the system architect’s toolbox. 

Interoperability 

Having a fast, reliable, affordable, and sovereign storage system is great, but your applications need to be able to access it for it to be of any value. Being more than just a filesystem, ZFS can be combined with other open source software to interface with almost every application and system you might encounter. In addition to the typical file sharing protocols like SMB/CIFS, NFS v3/v4, pNFS, and even (S)FTP, ZFS can expose block volume via iSCSI, fibre channel, and various network block device protocols. Object storage is also available, with multiple S3-compatible options to choose from. 

Using standard interfaces and protocols ensures that your storage infrastructure can interface with whatever different workloads and appliances you have in your infrastructure now, or that you might choose to deploy in the future. 

Supporting standards doesn’t have to mean lowest common denominator either. With SMB Multi-channel and NFS-over-RDMA, achieving saturation of 100 gbps links is well within reach. 

Exit and Portability 

The biggest question you have to ask when designing a truly independent system is “What if I change my mind and want to use something else?” Can you picture the look on your vendor’s face when you ask this? Can you guess which details they might leave out when answering? 

With a ZFS based storage infrastructure, we can swap out components easily. Want to replace your x86 servers with newer ARM based ones? Connect your existing JBODs to the new server and import the ZFS pool. Job done. 

Switching operating systems? ZFS is ZFS, so all that will change are system-specific configurations, such as the NFS server settings. You are not locked into any one specific OS and distribution. Whatever flavour of Linux your team prefers, be it Ubuntu, RHEL, Rocky, Debian, or something else entirely like FreeBSD.  

Running out of storage, but the previous model of disks is not available anymore? You have two options: 

• add additional VDEVs with no requirement that the disks be the same model, brand, interface (SATA/SAS/NVMe), or media (rust vs flash) 

• Gradually replace all of the old disks with newer ones, unlocking additional space as each VDEV completes the upgrade 

You are not locked into any specific vendor or ecosystem, deploy whatever storage media you choose. 

And if in the future, ZFS is no longer the best fit, because it is a POSIX filesystem, a wide selection of migration tools such as rsync, rclone, robocopy, and restic are available to migrate your data to whatever replacement you choose, via SSH, SMB, NFS, or S3 API. 

Exit strategy is the ultimate test of storage independence: if you cannot leave, you are not truly independent. 

Pool Architecture 

Now that we have established why you should choose ZFS to build your storage infrastructure, we need to examine how to architect the pool to maintain our hard won independence. 

The initial design decisions about pool geometry are fairly set in stone. Klara is working to improve that with the new more flexible AnyRaid, but that is still under development. If the pool is constructed of one or more relatively wide RAID-Z groups, then the primary means of expansion are to add additional groups (generally with a similar number of drives), or to replace all members of an existing group with larger drives incrementally. 

If your pool is likely to require significantly more flexibility, or more modest growth increments, choosing a pool layout with much smaller VDEVs, and specifically choosing mirrors as the basic building block can provide that flexibility, including the ability to shrink the pool by removing VDEVs. 

It is best to design the pool based on your preferred growth model, either vertical growth (replace disks with larger ones in the future), or horizontal growth (add VDEVs with more disks or entire storage shelves). 

Performance can be a factor in this decision, especially with HDDs. More spindles and specifically more VDEVs, will provide more IOPS, so a larger number of modestly sized drives will likely outperform fewer larger drives. 

Failure Domains 

Storage needs to be architected around what can fail, and how those failures can be mitigated. 

Starting from the highest level first, jurisdictional failure. Where data is located governs the majority of the legal and sovereignty risks. Will you have standing to defend your data from government interference? 

Zooming in a level, we have the data center failure. Does your design include multi-site replication and resilience against the loss of a data center? Are all of the critical infrastructure accounts run through one person, or a single credit card? Eliminating as many single-points of failure (SPoF) as possible with good design will make for a stronger, independent storage system. 

Dealing with node failure is mostly a question of trade-offs. Do you need a tighter RPO and RTO that requires a more complex HA system, or can you trade a few minutes of downtime for a less technically risky replication driven approach? Can you solve the requirements for availability at the application level, and not need to provide HA at the node level? 

For individual disk shelves and backplanes, controllers and cables, the name of the game is fault partitioning. How to ensure that the number of disks impacted by a single failure remains within the fault tolerance of your pool layout. Multipath can ensure that a single failure doesn’t degrade the storage at all. Allocating VDEV members from different shelves to ensure the loss of a shelf, controller, or backplane doesn’t fault the pool, and allows it to keep operating while the issue is resolved. 

And lastly we have the individual device or sector failure. ZFS provides strong checksums to ensure that the data retrieved in the future exactly matches what was originally written. The right design for each use case will provide a level of redundancy that both ensures the durability and availability of the data, without wasting space on excess parity. 

The other factor to consider when designing for durability, is the time to repair. How long will it take to source, install, and resilver to the replacement media. Many of the design decisions we have discussed and specifically media type and pool geometry will be important factors in how long the system will take to return to nominal. Ensuring that subsequent failures that might occur before this process completes do not exceed the fault tolerance of the system may be the single most important factor in storage design. 

Migration Paths 

Much like described earlier in the exit strategy section, getting your existing data onto your new ZFS storage infrastructure is mostly a matter of choosing the protocol and mechanism most compatible with your existing storage infrastructure. After the initial pass to copy data to ZFS, additional incremental passes will reduce the delta and allow for a minimal cutover window to ensure all data is migrated safely and consistently. Klara offers detailed migration planning as part of its Storage Design services. 

During the initial ingestion of data to your new ZFS array is the best time to create your plan for migrating the data from the new array to the next storage system. That may well be another ZFS array after the next hardware refresh, but the decisions you make now can make that process infinitely easier, to significantly harder. 

One of the biggest factors is how the data on your pool is broken up into datasets. ZFS datasets provide the policy boundaries between different data, allowing you to apply different record sizes, compression, deduplication settings, and have different snapshot and replication policies, which are pivotal to achieving your desired RPO and RTO. 

Data should be broken up based on purpose, access pattern, workload profile, and size. Making sure that no single dataset dominates the pool will ensure that come migration time, it is easier to move multiple datasets in parallel, and that each dataset is not so large that you might struggle to find a destination to back it up, or to migrate it to. 

When performing a ZFS-to-ZFS migration, the steps are very straightforward. Create a snapshot of each target dataset and start an initial replication to the destination. During this time, the dataset is still in active use on the primary and will continue to accumulate changes. The initial transfer may take significant time as it must transfer all of the data. Once this process is complete, create an additional snapshot of each dataset, and run an incremental replication, which will copy only the data that has changed since the initial snapshot. This will take less time as any unmodified data will not need to be copied. Repeat this process until the snapshots stop getting smaller, you have achieved sync with a delay of only the amount of time the data takes to transfer. Coordinate a cut-over time, where the source system will be read-only for the duration it takes to replicate one last snapshot with the final set of changes. Once this final transfer completes, applications can switch to using the destination as the primary for the data. The migration is complete, with minimal downtime and no risk to the consistency of the data. 

If you have correctly broken the data up into reasonable datasets, each of these migrations can be done at a different time, or they can all be done concurrently, whatever best fits your requirements. 

Independence is Designed, Not Achieved 

The way vendors trap you is not with the support contract but with architectural lock-in. OpenZFS storage independence is rooted in the deliberate design choices that preserve optionality at every layer. When your storage depends on specific hardware, software, or replacement parts, it takes away control and leaves you facing difficult migrations. The vendors want to make the process difficult enough that most won’t bother.  

That is why migration capability is the most important durability metric for independence. If you had to move, how much work would it be? How long would it take? 

A healthy storage deployment is one you can replace at any time. If the hardware has a flaw, if there is a firmware bug that the manufacturer just can’t seem to fix, or if the OS vendor pulls the rug out from under your long-term support, you should be able to replace the problematic component and get your data back on a stable footing easily. This can only be achieved with thoughtful, open design. 

Maintaining true storage independence requires more than just choosing the right storage stack — it requires aligning architecture, failure domains, and migration strategy with long-term operational goals. 

Organizations looking to build a sovereign storage infrastructure can work with Klara Systems through its OpenZFS Storage Design Solution to ensure their deployment is resilient, portable, and built for long-term independence.