How Klara and TrueNAS collaborated to fix one of ZFS's longest standing limitations​

In the world of enterprise storage, few technologies have reshaped architectures and capabilities as profoundly as ZFS. When ZFS was introduced over two decades ago, it rewrote the rulebook for data integrity, scalability, and storage pool management. Yet, for all its innovations, one capability consistently lagged expectations: deduplication at scale. 

Deduplication, the process of identifying and eliminating extraneous copies of data, is not novel. It has been a staple of backup appliances and storage systems since the 90s. The original ZFS implementation, while functionally correct, was often impractical in real-world enterprise environments due to the performance and memory overheads. The amount of memory required was unpredictable, and if it exceeded the available memory, most write operations would be bottlenecked waiting for random reads from the dedup table, dropping performance off an unexpected cliff. That all changed through a focused collaboration between Klara and TrueNAS, resulting in what the community now calls ZFS Fast Dedup. 

The Problem: Deduplication Without Compromise 

To understand the breakthrough, we must first understand the constraint that plagued ZFS deduplication since its inception. 

ZFS’s legacy deduplication capability worked by keeping a Deduplication Table (DDT) that tracks the checksums of every block in the pool. When writing new data, ZFS would calculate the checksum to ensure data integrity, then consult the DDT to determine if that data already existed somewhere in the pool. 

This seems straight forward enough, but in practice it scaled poorly for several reasons: 

• Unbounded size: the DDT tracked every block written to the pool to detect future duplicates. This meant that as the volume of data grew, the size of the DDT also grew, and there was no way to constrain it. Until the introduction of Fast Dedup, there was also no way for the DDT to shrink, once it had grown, even as data was removed, the table itself would remain at its peak size even if it was nearly empty. Worse yet, the size of the DDT was difficult to predict, as it depended on the number of records created, which can vary by workload. 

• Memory Consumption: Effective deduplication on large pools required caching the entirety of the DDT in RAM in order to maintain performance. If the DDT was too large to fit in the ZFS RAM cache, any attempt to write a new block was likely to have to read in parts of the DDT from disk, slowing those write operations significantly. 

• Performance Impact: Any changes made to the DDT, such as inserting the hashes of newly written blocks or updating the reference count of existing blocks, required writing out new versions of each section of the DDT that changed, and all of the related indirect blocks and metadata. The strong hashes used for dedup mean that changes will be randomly distributed throughout the DDT, resulting in worst case performance. 

• Predictability: As the DDT grew beyond the amount of available cache memory, it causes a “performance cliff”, performance would drop sharply as sections of the DDT must then be read in from disk, increasing write latency and disrupting workloads. Administrators could not reliably forecast dedup ratios or the memory footprint required — making planning and budgeting difficult. 

For many IT organizations, deduplication became a checkbox feature: attractive on the spec comparison, and in the documentation, but too risky in production. 

The Market Context: Why Deduplication Still Matters 

It is important to understand why deduplication is worth the added complexity and unpredictability it introduces, and why it remains a priority for many organizations. 

Storage Cost Optimization 

Enterprises face relentless data growth. In regulated industries, data retention policies mandate keeping years or even decades worth of history. Many workloads, including containers, CI/CD, and VMs, run multiple instances of identical or near identical checkouts and images, in these cases, deduplication can reduce storage footprints by 10x or more, dramatically lowering costs: 

CapEx:

• Storage servers and disk slots 
• Procurement of disks and other media 

• Backup and DR capacity 

OpEx: 

• Floor and rack space in data centers 

• Power and cooling 

• Lifecycle planning and execution 

Bandwidth and Cache Efficiency 

Every block that ZFS deduplicates not only saves storage capacity, but also eliminates the majority of the I/O that would have been used to write that data to disk, freeing that throughput for other workloads. The ZFS ARC (Adaptive Replacement Cache) is also able to use a single copy of the data to serve all referenced blocks from a hot cache, improving hit ratios and efficiency. 

Additionally, replication, snapshots, and backups, all leverage deduplication to reduce the volume of data transmitted over the network, which can be a direct advantage in WAN-connected environments or cloud DR setups. 

Cloud Economics 

For hybrid and cloud-backed storage tiers, deduplication translates directly into lower egress, storage, and snapshot costs. When every gigabyte stored comes with a monthly (or even hourly) fee, every reduction directly translates into immediate and future savings. 

Yet, until recently, ZFS users weren’t able to reliably achieve these benefits due to the limitations detailed above. That’s where Klara and TrueNAS stepped in. 

The Collaboration: Klara and TrueNAS 

Who Are the Players? 

Klara Inc: A specialized engineering form with deep expertise in filesystems, data storage, operating systems, and performance engineering. 

TrueNAS (formerly iXsystems): The steward of TrueNAS, providing enterprise-grade NAS and SAN platforms trusted by Fortune 500 companies. 

Both organizations share a commitment to open source and the sustainability of ZFS as a practical, enterprise-ready, high-performance storage system. Their collaboration on Fast Dedup was strategic: blending deep ZFS internals knowledge with real-world product requirements and customer use cases. Directly contributing the results to the OpenZFS project to ensure its future compatibility and sustainability. 

What This Means for Your Storage 

While the engineering behind Fast Dedup is significant, the more important question for leadership is not how it works internally, but what it changes operationally and economically. 

Storage architecture decisions are rarely made purely on technical merit. They are driven by cost models, operational risk, lifecycle planning, and the ability to scale without continually revisiting foundational design choices.  

For years, ZFS offered exceptional data integrity and flexibility, but deduplication remained a feature many organizations deliberately avoided because its resource requirements were difficult to forecast and its operational impact was uncertain. 

Fast Dedup materially changes that calculus.

By reducing the memory footprint, improving performance predictability, and introducing mechanisms that allow administrators to model the impact of deduplication before enabling it, the collaboration between Klara and TrueNAS turns deduplication from a niche optimization into a practical design option. Instead of being a feature reserved for specialized environments with large memory budgets, it becomes something storage architects can evaluate as part of normal capacity planning. 

For IT leaders, the key takeaways are: 

1. 1. Deduplication Is Now Enterprise-Viable 

No longer simply a checkbox in the feature matrix, fast dedup is now part of strategic storage efficiency plans. 

1. 2. Predictability Enables Planning

Forecasting ratios and resource impact means storage investments can be justified with data and aligned to business outcomes. Leveraging the dedup quota setting ensures that your system maximizes the benefits of dedup, while avoiding the performance cliff caused by exceeding that planned capacity. 

1. 3. Operational Simplicity Scales

Administrators can enable and monitor dedup without constant firefighting, freeing teams to focus on higher-value work. Fast dedup is able to scale down, returning resources when it is switched off for specific datasets. 

1. 4. Cost-to-Value Improvements

Fast dedup avoids the need to overbuild systems to handle unpredictable DDT RAM requirements. Fast dedup provides predictable performance, maximizing the benefits of data reduction while lowering total cost of ownership and improving storage ROI. 

What Changed: Fast Dedup Explained 

Fast Dedup was not just a patch on top of the existing ZFS deduplication, but a complete re-architecture of the deduplication pipeline to make it: 

• Log based to reduce performance impact under heavy load 

• Configurable and predictable 

• Scalable in all directions, including downwards 

• More Resource-efficient  

• Generate less IOPS amplification 

Here are the key innovations and their implications: 

1. Dynamic DDT Management 

Rather than updating the entire DDT as part of each transaction group (every few seconds), Fast Dedup keeps a sorted in-memory tree of the changes and writes an append-only log to be able to recreate the in-memory tree if the system is restarted. 

Once the in-memory tree reaches a configured upper bound in size, the tree and log are rotated together, and a new pair receive any future changes. The tree is then applied to the DDT in bulk batches, sorted by the hash, so all of the keys that belong to the same block of the DDT are written at once, so that the indirect blocks and metadata only need to be rewritten a single time. Once the entire tree is processed, it is freed and the log is deleted. 

2. Configurable Limits 

Fast Dedup also introduced the dedup quota, allowing the system to be configured to temporarily disable deduplication if the size of the DDT reaches the quota. This avoids the “performance cliff” caused by the DDT not fitting comfortably in the ZFS cache and having to be read from disk during write operations. The quota can be used to constrain the size of the DDT to fit in memory, or to remain within the capacity of a dedicated dedup vdev, backed by NVMe devices. 

3. Pruning Excess Entries 

In order to be able to scale down, Fast Dedup introduced a new maintenance command that can remove older entries from the dedup table that have only been seen a single time. If a block has been in the storage pool for months, and a duplicate has never been written, it is less likely that one will be written soon, compared to the chance that a newly written block will have a second copy created. With the dedup prune feature, these older unique blocks are removed from the DDT, treated as if they were written while dedup was disabled, in order to make room for new blocks to be added to the DDT in their place. Working in coordination with the dedup quota feature, this ensures that dedup continues to operate once the quota is hit, by removing older entries to free up space in the DDT. 

To make this effective, Klara also authored patches to allow ZAP (ZFS Attribute Processor) data structures to shrink when they are sparely populated. 

4. Improved On-disk Structure 

Legacy dedup stored 4 versions of each unique block, one for each number of copies the block was assigned. Fast Dedup eliminates this, and instead stores just the maximum requested level of redundancy, upgrading an existing entry with additional copies if required. This reduced the size of each entry significantly, even when accounting for the additional timestamp field added by Fast Dedup. 

The indirect size of the DDT was also increased, to allow the table itself to benefit from compression, and to reduce the number of IOPS required to update an entry. 

ZFS behavior was also changed to not store three copies of every block in the DDT. Since the pool itself already provided redundancy in the form of the RAID transforms, this added redundancy is needless, as if RAID-Z cannot recover the data, it is unlikely additional copies will help. The loss of a small chunk of the DDT is not unrecoverable, it just means that any blocks that were covered by that section of the DDT can never be freed, as ZFS will not be able to ensure there are no remaining references, so it will merely leak a small amount of disk space. 

Final Thoughts

In enterprise IT, the value of a technology isn’t judged by its elegance, or its feature set, but by the problems it solves, the risks it mitigates, and the outcomes it enables. In that regard, Fast Dedup is a substantive leap forward for ZFS, closing a gap that held back wider adoption for more than a dozen years. 

The collaboration between Klara and TrueNAS started with identifying a real pain point, following by rigorous engineering to deliver a practical solution, exemplifying how open systems can evolve to meet enterprise demands. 

For organizations wrestling with data growth, cost pressures, and performance expectations, this development turns ZFS deduplication from “nice to have” to “practical and valuable”. The era of unpredictable deduplication is over. With that limitation addressed, ZFS removes one of the last meaningful advantages claimed by traditional enterprise storage systems and delivers the simplicity and openness modern storage architectures require.