Modern web and mobile applications are increasingly dependent on software defined storage. Most commonly, this means Amazon Web Services’ S3 storage buckets. What you may not realize is that you don’t actually need Amazon for Amazon-compatible cloud storage! In this article, we’ll discuss how and why to avoid vendor lock-in by providing your apps fully S3-compatible storage using free and open source software.
Demystifying OpenZFS 2.0
You may know that FreeBSD 13.0 shipped with OpenZFS 2.0. You might be wondering: “I thought OpenZFS did feature flags, not versions, so what is up with the 2.0?” You may have heard rumblings that it is based on (gasp!) Linux and wonder what that is all about. We wrote some more about ZFS 2.0 in the past in the following article: Getting Started with OpenZFS 2.0
To understand the need for a 2.0, we need to look at a bit of ZFS history.
ZFS was originally developed at Sun Microsystems and released under an open source license as part of OpenSolaris. When Oracle bought Sun and ended development of OpenSolaris, some developers, including many of the original ZFS developers, created Illumos, a fork of the last open source version of Sun’s operating system that included ZFS, so that open source development could continue.
Over time, the Illumos ZFS code was ported to many operating systems, including FreeBSD in 2007 and Linux in 2008. In 2013, the OpenZFS project was created to coordinate cross-platform development with a goal of providing a single common repository for OS-independent code. Pull requests for bug fixes and new features would be managed by the Illumos team. Version numbers were replaced by feature flags to differentiate OpenZFS from Oracle ZFS and to allow developers to decide which features to create and integrate into their OS-specific port. That same year, OpenZFS held its first annual Developer Summit to give developers of the various open source ZFS ports the opportunity to meet in person and participate in planning and coordination efforts.
As communication between developers increased and the actual work of maintaining OS-specific ports continued, it became clearer over time that the development workflow envisioned by the OpenZFS Project needed to be overhauled in order to better meet the needs of both the developers and OpenZFS users. Features were beginning to diverge across the various operating systems. Not all of the bug fixes fixed in one OS were upstreamed back, meaning other operating systems had to implement their own versions of bug fixes. Since each OS had their own community of developers, other communities weren’t always aware of new development efforts, resulting in duplication of effort for the same desired features.
It also became clear that new feature development had shifted from Illumos to Linux. Since FreeBSD tracked Illumos commits, this meant that any new features and fixes created on Linux had to first be backported to Illumos before they could be reported to FreeBSD.
After much discussion and planning it was agreed that it made sense for everyone to switch from Illumos to Linux as the upstream repo. And, it was agreed that future changes would be discussed across platforms before being implemented and that there would be appropriate porting layers to prevent GPL’d or Linux-KPI shim code from being introduced to other operating systems. Continuous integration (CI) for the repo would ensure that all proposed changes would have to pass CI on both Linux and FreeBSD before they could be merged. Thus, the design of OpenZFS 2.0 was born.
While the decisions about OpenZFS 2.0 were made in 2018, it took a lot of work from a lot of developers to do the necessary work to change the OpenZFS framework from its original inception to the proposed “2.0” framework. The necessary work to change the Linux-specific ZFS on Linux port to something that could replace Illumos as the upstream “source of truth” that other operating systems could safely port from was completed in November 2020 with the announcement of OpenZFS 2.0.0: “the former ZFS on Linux project has been renamed OpenZFS! Both Linux and FreeBSD are now supported from the same repository making all of the OpenZFS features available on both platforms.”
Continued coordination and collaboration with OpenZFS developers ensures that the same code will continue to work on both Linux and FreeBSD and that new features will be immediately available to developers.
OpenZFS 2.0 is more than just an overhauled development process. It provides a slew of new features, including:
- Sequential resilver: dramatically decreases the resilvering time to rebuild a failed mirror. Full redundancy is restored as quickly as possible and then the pool is automatically scrubbed to verify all of the data checksums.
- Persistent L2ARC: makes the L2ARC cache device persistent across reboots which eliminates the cache warmup time normally needed after importing a pool.
- ZStandard compression: – a modern, high performance, compression algorithm which provides compression levels similar to GZIP and performance load similar to lz4.
- Redacted zfs send/receive: allows sending subsets to a target system which can save space by not replicating unnecessary data. It can also exclude sending sensitive information.
- Reorganized man pages: each subcommand of zfs and zpool has its own man page.
- Support for inheriting and setting user properties in channel programs.
- zfs-wait(8) and zpool-wait(8): wait for long running background operations to complete, such as resilver or scrub.
- zfs jail(8), zfs unjail(8): attach/detach ZFS filesystems from FreeBSD jails.
- zfs rename -u: rename a filesystem without remounting it.
- Increased performance: faster clone deletion, faster send/receive performance for small record sizes, more efficient ARC and memory management., improved write performance for heavily fragmented pools, and optimized AES-GCM encryption for encrypted pools.
Being in-sync with Linux also has its benefits:
- Command line standardization makes system administration easier.
- Improved pool portability as OpenZFS pools created on one operating system can be used by another.
Check out an even more detailed description of the newest features of OpenZFS 2.0, in our article Getting Started with OpenZFS 2.0.
While the original development workflow for OpenZFS needed changes, one of the things the OpenZFS Project has always done right is transparent communications. It strives to create an active and welcoming community of OpenZFS developers, users, and enthusiasts. Design decisions are collaborative and openly available. Anyone is welcome to attend the annual DevSummit—I’m not a developer and I’ve been able to attend most of the DevSummits in person and enjoyed all of them. Anyone is welcome to attend the monthly OpenZFS Leadership Meeting video conference or watch the recordings of past meetings. Learn more at: https://openzfs.org/wiki/Participate.
Like this article? Share it!
Did you know?
Improve the way you make use of ZFS in your company
Did you know you can rely on Klara engineers for anything from a ZFS performance audit, to developing new ZFS features to ultimately deploying an entire storage system on ZFS?
Although easy to overlook, storage is the most fundamental part of any computing project—without storage, there is neither code nor data! The right storage solution should be accessible, reliable, easy to maintain, and free from vendor lock-in. In this article, we examine some of the reasons that open source software is a natural fit for this crucial component.
Today, we’ll concentrate on exposing the data on your NAS to the network using NFS, Samba, and iSCSI shares. Network File System (NFS) is a commonly used protocol for accessing NAS systems, especially with Linux or FreeBSD clients. We’ll provide an overview of each type of share to help guide you in deciding which is most suited to the clients that will be accessing the NAS.
Let’s examine how non-developer contributors enhance user experience, improve bug reporting, and influence feature requests, all while becoming advocates and evangelists for your open source project.