ZFS Fast Dedup for Proxmox VE 9.x: Technical Implementation Guide

Introduction 

Fast Dedup, introduced in OpenZFS 2.3, transforms ZFS deduplication from a resource‑constrained feature into a viable option for selected Proxmox VE 9.x workloads, provided pool design and memory budgets are established through measurement rather than speculation.i Proxmox 9.0 and 9.1 ship OpenZFS 2.3.x, enabling feature@fast_dedup on pools created or upgraded under VE 9.x.ii  This document provides technical guidance on Fast Dedup architecture, enabling procedures, sizing methodology, and workload patterns for Proxmox environments. 

Fast Dedup Architecture 

Classic ZFS deduplication performs inline lookups in a pool‑wide Deduplication Table (DDT) for every block write, storing a new block only on table  misses.iii The DDT can grow without bounds until it no longer fits in ARC, at which point dedup performance degrades catastrophically due to random I/O and memory pressure.

Fast Dedup introduces structural modifications to the deduplication subsystem:

• A new on‑disk feature@fast_dedup structure optimized for locality and predictable access patterns. 

• A deduplication table quota mechanism that enforces a hard limit on DDT size; writes to dedup‑enabled datasets bypass deduplication once the quota is reached, preventing unbounded DDT growth. 

• A deduplication log that batches updates and reduces random write operations into the DDT. 

• Prefetch support, allowing the DDT to be loaded into ARC on demand via zpool prefetch, reducing cold‑start latency after node reboots. 

• Prune operations that selectively evict older UNIQUE entries from the DDT, freeing space under the quota to allow future writes to dedup. 

The result is a bounded, operator‑controlled deduplication system where memory consumption and I/O patterns are predictable and measurable. 

Availability on Proxmox VE 9.x 

Proxmox VE 9.0 and 9.1 include OpenZFS 2.3.x in the bundled ZFS packages. Fast Dedup support is therefore available on all Proxmox 9.x nodes: 

• Pools created or upgraded under Proxmox 9.x can enable feature@fast_dedup once the ZFS package version is verified as 2.3.x or later via zpool upgrade -v. 

• Enabling feature@fast_dedup is a one‑way pool upgrade operation; pools with this feature cannot be imported by ZFS 2.2.x or earlier implementations. 

• There is no upgrade path from legacy dedup to fast dedup. If there is an existing legacy DDT, fast dedup will not be used. 

• Proxmox 8.x and earlier releases ship ZFS 2.2.x or older and do not support Fast Dedup; forum discussions reporting "unsupported by kernel" errors reflect using the new ZFS CLI with the pre‑2.3 kerne module and are not applicable to VE 9.x deployments. 

Any Fast Dedup implementation on Proxmox assumes VE 9.x nodes with consistent ZFS 2.3.x kernel and userspace versions across all cluster members that share the storage pool. 

Enabling Fast Dedup on Proxmox Pools 

The procedure for enabling Fast Dedup on a Proxmox pool follows this sequence: 

1. Verify feature availability on all cluster nodes that will import the pool: 

   zpool upgrade -v | grep -A 1 fast_dedup.

2. Ensure all nodes run Proxmox VE 9.x with identical ZFS 2.3.x package versions to avoid feature incompatibility during pool import.
3. Enable the feature@fast_dedup pool feature: 

zpool set feature@fast_dedup=enabled tank 

4. Set a deduplication table quota at the pool level. The quota value is determined by the RAM and SSD budget allocated to deduplication: 

zpool set dedup_table_quota=64G tank 

5. Enable deduplication on specific datasets where high block‑level duplication is expected: 

zfs set dedup=on tank/vmstore 

The dedup property remains dataset‑specific;dedup_table_quota is pool‑wide and constrains total DDT capacity under Fast Dedup. Once the quota is reached, new writes for dedup‑enabled datasets bypass deduplication and are written as unique blocks, preventing unbounded DDT growth. 

Sizing: DDT Quota, ARC, and SSD Allocation 

The critical sizing constraint is aligning dedup_table_quota with available ARC capacity and, optionally, dedicated flash storage for DDT metadata. The size of the DDT should generally not exceed 50% of the maximum ARC size, or if the pool has dedicated dedup vdevs, 85% of their available capacity. Remember that dedup vdevs are critical, so they should use mirroring to guard against device failure that would cause catastrophic damage to the pool. 

Memory and Storage Budget 

DDT entries consume ARC memory. Fast Dedup improves locality and enforces quota‑based limits, but does not eliminate in‑memory metadata overhead. The entire DDT must fit within the ARC, or be supplemented where necessary by fast special vdevs or dedicated flash-based dedup vdevs. 

For Proxmox nodes, RAM must also be reserved for virtual machine guests and base ARC capacity for non‑dedup workloads. Deduplication should receive a bounded memory allocation rather than competing for all available cache. 

Workload Selection on Proxmox 

Fast Dedup should be applied selectively within Proxmox clusters, targeting datasets where block‑level duplication is empirically high and stable.

Workloads with High Deduplication Potential 

• VM template and clone stores: multiple virtual machines instantiated from golden images or thin‑cloned root disks exhibit very high block‑level duplication. 

• VDI and lab/test environments: many nearly identical OS installations or containerized workloads with shared base images. 

• Backup and replica datasets: full or near‑full logical copies stored on the same pool, particularly image‑level backups with high internal duplication. 

Workloads Where Dedup Is Not Recommended 

• Databases, logging subsystems, and high‑entropy workloads with low dedup ratios; metadata overhead is unlikely to be offset by space savings. 

• Application encrypted data, since duplicate data will be encrypted before it is written to ZFS, it will not deduplicate. 

• Mixed "catch‑all" datasets hosting both high‑dedup and low‑dedup workloads; isolation into separate correctly configured dataset is strongly recommended. 

Proxmox Storage Design Pattern 

A recommended pattern for Proxmox 9.x clusters separates deduplication from general‑purpose storage: 

• A dedup‑enabled dataset for VM images and templates where duplication is empirically high (measured dedup ratios exceeding 3:1). 

• A non‑dedup dataset for general VM disks and write‑heavy applications, relying on compression (zstdorlz4) as the primary space‑efficiency mechanism. 

This separation allows precise control over memory and I/O budgets for each workload class. 

Operational Tools: Prefetch and Prune 

Fast Dedup provides operational controls relevant to Proxmox environments where nodes reboot for kernel upgrades, Proxmox updates, or maintenance.

DDT Prefetch 

After a reboot, deduplication performance can be degraded until the DDT is fully read back into ARC through normal access patterns. Fast Dedup supports explicit prefetch operations to load the DDT proactively: 

zpool prefetch -t ddt tank 

This command pulls DDT metadata into ARC, shortening the "cold start" window during which dedup lookups suffer elevated latency. Prefetch operations should be integrated into post‑reboot procedures for Proxmox nodes with dedup‑heavy pools. 

DDT Prune 

To adapt to changing workloads or memory budget constraints, the zpool ddtprune command selectively removes older UNIQUE entries from the DDT, freeing space under the quota. To remove the oldest 10% of entries: 

zpool ddtprune -p 10 tank 

Or to prune any entries older than 90 days: 

zpool ddtprune -d 90 tank 

Prune operations allow administrators to reclaim DDT resources without deleting the associated data. The oldest cohort of blocks that have never deduplicated are removed from the DDT, making them ineligible for dedup in the future, but making room for new blocks that are more likely to deduplicate. For Proxmox clusters, periodic prune operations should be scheduled during maintenance windows and integrated into capacity management procedures. 

Monitoring and Validation 

Effective Fast Dedup deployments require continuous monitoring of DDT size, ARC hit ratios, and I/O performance metrics. 

Key Metrics 

• DDT size relative to quota: monitored via zpool status -D or equivalent. 

• ARC hit ratio: target greater than 85% for well‑tuned deployments; measured via arcstat or arc_summary. 

• Dedup ratio: measured via zpool get dedupratio tank; ratios below 2:1 typically do not justify the metadata overhead. 

• I/O latency and IOPS: validate that dedup does not introduce unacceptable latency for VM workloads. 

Conclusion 

Fast Dedup in OpenZFS 2.3 transforms deduplication into a bounded, measurable subsystem suitable for selected Proxmox VE 9.x workloads where block‑level duplication is empirically high and memory budgets are explicitly allocated. Proxmox 9.x deployments can enable feature@fast_dedup, configure dedup_table_quota, and apply deduplication selectively to VM template stores, VDI environments, and backup datasets while leaving general‑purpose VM storage on non‑dedup datasets with compression enabled. Operational discipline—prefetch after reboots, periodic prune operations, continuous monitoring of DDT size and ARC hit ratios—ensures that Fast Dedup delivers space efficiency without unpredictable performance degradation. 

Klara’s ZFS Storage Design solution provides the expertise to design your hypervisor environment to maximize the benefits of all ZFS features, from dedup and caching, to compression and flash acceleration.