About Fastmail
Fastmail is an independent email provider recognized for its engineering-led approach to privacy, reliability, and open standards. The company operates a globally used platform built on self-managed, bare-metal infrastructure—an architecture that prioritizes performance consistency, long hardware lifecycles, and full operational control. Fastmail’s commitment to transparency and open technologies continues to shape its role as a trusted, standards-driven alternative within the modern email ecosystem.
Location: Australia
Founded: 1999
Industry: Internet Software and Services
The Challenge
Fastmail has more than two decades of experience running its own hardware, optimizing bare-metal servers for reliability, longevity, and predictable performance.
When the company upgraded its email storage platform to NVMe SSDs backed by OpenZFS, the shift brought major operational advantages—simpler encryption, strong compression, and the ability to tune datasets for different workloads.
At Fastmail’s scale—tens of millions of small files across production systems—unexpected edge-case behaviour began appearing on a subset of machines. Some machines showed higher than expected CPU usage, and OpenZFS did not appear to use memory as effectively as anticipated. These symptoms pointed to an issue on how OpenZFS interacted with the Linux memory subsystem under Fastmail’s production workload.
Fastmail needed to understand what was causing the issue and how to correct it without interrupting a globally active email platform. The challenge was to diagnose the production-only behavior and restore predictable, efficient performance across their fleet.
Because the anomalies only appeared under real production load and only on a subset of machines, isolating the underlying cause required deep understanding of both OpenZFS internals and Linux memory subsystem.
The Solution
Klara’s OpenZFS experts collaborated with Fastmail to investigate the CPU and memory anomalies impacting their NVMe-backed storage clusters. The work led to focused patches that resolved the issue and improved system behavior.
Root Cause Analysis
Fastmail asked Klara’s OpenZFS engineers to investigate the unexpected CPU and memory behavior appearing across parts of their NVMe-backed email storage fleet. Klara reviewed how OpenZFS interacted with the memory subsystem of the particular version of Linux being used on the target systems, while under Fastmail’s workload of tens of millions of small files.
Memory Subsystem Investigation
Through deeper analysis of production data, Klara identified conditions where OpenZFS memory usage diverged from expected behavior. These conditions led to higher-than-expected CPU usage and memory behavior that did not align with Fastmail’s expectations on the affected systems.
Targeted Patch Development
Once the underlying cause was clear, Klara engineers created and tested a set of focused patches designed to address the issues Fastmail observed. The fixes were validated in Fastmail’s production environment, where they returned the affected systems to expected CPU and memory behaviour.
Upstream Contribution
After verification, the patches were contributed upstream to OpenZFS. This ensured Fastmail could rely on long-term stability while also strengthening the platform for the broader OpenZFS community. For Fastmail, upstreaming wasn’tjust a technical decision, a way to avoid future technical debt; it aligned with their commitment to being “a good internet citizen”.
Business Impact
For an email provider operating a globally active platform, consistency is everything. The fixes immediately translated into smoother operations and more predictable behaviour across the fleet. The technical improvements from Klara’s OpenZFS work translated directly into operational value for Fastmail. With the fixes in production, Fastmail saw several key benefits:
Improved Resource Efficiency
Klara’s patches fixed the CPU and memory issues Fastmail was seeing in production, helping them “get the absolute most out of our hardware” (Rob Mueller, CTO).
Storage Efficiency Fully Realized
OpenZFS features like built-in encryption and transparent compression, which already give Fastmail almost double the effective storage capacity across their NVMe servers, can now be used without the trade-offs of the performance anomalies that prompted the investigation.
Predictable Performance at Scale
The OpenZFS clusters handling small file workload returned to expected CPU and memory behavior, instead of the irregular usage patterns that previously affected part of the fleet.
Support for Their Hardware Strategy
With the issues resolved, Fastmail continues operating confidently within their 5–10-year hardware lifecycle strategy while retaining the cost and operational advantages of running on bare metal.
Lasting Impact for the Community
Because the fixes were upstreamed to mainline OpenZFS, Fastmail gains long-term stability through future releases, and other OpenZFS users benefit from the same improvements.
