Demystifying OpenZFS 2.0

Demystifying OpenZFS 2.0

To understand the need for a 2.0, we need to look at a bit of ZFS history.

You may know that FreeBSD 13.0 shipped with OpenZFS 2.0. You might be wondering: “I thought OpenZFS did feature flags, not versions, so what is up with the 2.0?” You may have heard rumblings that it is based on (gasp!) Linux and wonder what that is all about. We wrote some more about ZFS 2.0 in the past in the following article: Getting Started with OpenZFS 2.0


Why 2.0?

To understand the need for a 2.0, we need to look at a bit of ZFS history.

ZFS was originally developed at Sun Microsystems and released under an open source license as part of OpenSolaris. When Oracle bought Sun and ended development of OpenSolaris, some developers, including many of the original ZFS developers, created Illumos, a fork of the last open source version of Sun’s operating system that included ZFS, so that open source development could continue.

Over time, the Illumos ZFS code was ported to many operating systems, including FreeBSD in 2007 and Linux in 2008. In 2013, the OpenZFS project was created to coordinate cross-platform development with a goal of providing a single common repository for OS-independent code. Pull requests for bug fixes and new features would be managed by the Illumos team. Version numbers were replaced by feature flags to differentiate OpenZFS from Oracle ZFS and to allow developers to decide which features to create and integrate into their OS-specific port. That same year, OpenZFS held its first annual Developer Summit to give developers of the various open source ZFS ports the opportunity to meet in person and participate in planning and coordination efforts.

As communication between developers increased and the actual work of maintaining OS-specific ports continued, it became clearer over time that the development workflow envisioned by the OpenZFS Project needed to be overhauled in order to better meet the needs of both the developers and OpenZFS users.  Features were beginning to diverge across the various operating systems. Not all of the bug fixes fixed in one OS were upstreamed back, meaning other operating systems had to implement their own versions of bug fixes. Since each OS had their own community of developers, other communities weren’t always aware of new development efforts, resulting in duplication of effort for the same desired features.

It also became clear that new feature development had shifted from Illumos to Linux. Since FreeBSD tracked Illumos commits, this meant that any new features and fixes created on Linux had to first be backported to Illumos before they could be reported to FreeBSD.

After much discussion and planning it was agreed that it made sense for everyone to switch from Illumos to Linux as the upstream repo. And, it was agreed that future changes would be discussed across platforms before being implemented and that there would be appropriate porting layers to prevent GPL’d or Linux-KPI shim code from being introduced to other operating systems. Continuous integration (CI) for the repo would ensure that all proposed changes would have to pass CI on both Linux and FreeBSD before they could be merged. Thus, the design of OpenZFS 2.0 was born.

Matt Ahrens provides a good visual of the workflow difference between the original OpenZFS and OpenZFS 2.0 in his 2019 OpenZFS DevSummit keynote presentation (slides 11-13).

While the decisions about OpenZFS 2.0 were made in 2018, it took a lot of work from a lot of developers to do the necessary work to change the OpenZFS framework from its original inception to the proposed “2.0” framework. The necessary work to change the Linux-specific ZFS on Linux port to something that could replace Illumos as the upstream “source of truth” that other operating systems could safely port from was completed in November 2020 with the announcement of OpenZFS 2.0.0: “the former ZFS on Linux project has been renamed OpenZFS! Both Linux and FreeBSD are now supported from the same repository making all of the OpenZFS features available on both platforms.

Continued coordination and collaboration with OpenZFS developers ensures that the same code will continue to work on both Linux and FreeBSD and that new features will be immediately available to developers.

You might also be interested in

Improve the way you make use of ZFS in your company

ZFS is crucial to many companies. We guide companies and teams towards safe, whitepaper implementations of ZFS that enhance and improve the way the infrastructure is enabling your business.

New Features!

OpenZFS 2.0 is more than just an overhauled development process. It provides a slew of new features, including:

  • Sequential resilver: dramatically decreases the resilvering time to rebuild a failed mirror. Full redundancy is restored as quickly as possible and then the pool is automatically scrubbed to verify all of the data checksums.
  • Persistent L2ARC: makes the L2ARC cache device persistent across reboots which eliminates the cache warmup time normally needed after importing a pool.
  • ZStandard compression: – a modern, high performance, compression algorithm which provides compression levels similar to GZIP and performance load similar to lz4.
  • Redacted zfs send/receive: allows sending subsets to a target system which can save space by not replicating unnecessary data. It can also exclude sending sensitive information.
  • Reorganized man pages: each subcommand of zfs and zpool has its own man page.
  • Support for inheriting and setting user properties in channel programs.
  • zfs-wait(8) and zpool-wait(8): wait for long running background operations to complete, such as resilver or scrub.
  • zfs jail(8)zfs unjail(8): attach/detach ZFS filesystems from FreeBSD jails.
  • zfs rename -u: rename a filesystem without remounting it.
  • Increased performance: faster clone deletion, faster send/receive performance for small record sizes, more efficient ARC and memory management., improved write performance for heavily fragmented pools, and optimized AES-GCM encryption for encrypted pools.

Being in-sync with Linux also has its benefits:

  • Command line standardization makes system administration easier.
  • Improved pool portability as OpenZFS pools created on one operating system can be used by another.

Check out an even more detailed description of the newest features of OpenZFS 2.0, in our article Getting Started with OpenZFS 2.0.

Participate

While the original development workflow for OpenZFS needed changes, one of the things the OpenZFS Project has always done right is transparent communications. It strives to create an active and welcoming community of OpenZFS developers, users, and enthusiasts. Design decisions are collaborative and openly available. Anyone is welcome to attend the annual DevSummit—I’m not a developer and I’ve been able to attend most of the DevSummits in person and enjoyed all of them. Anyone is welcome to attend the monthly OpenZFS Leadership Meeting video conference or watch the recordings of past meetings. Learn more at: https://openzfs.org/wiki/Participate.


Like this article? Share it!

You might also be interested in

Improve the way you make use of ZFS in your company

ZFS is crucial to many companies. We guide companies and teams towards safe, whitepaper implementations of ZFS that enhance and improve the way the infrastructure is enabling your business.

More on this topic

Advanced ZFS Snapshots 

In our previous articles, we introduced you to the basics of ZFS snapshot management, and explained concepts such as creating OpenZFS snapshots, restoring files from a snapshot, and deleting snapshots.
With this article, we dive a bit deeper into OpenZFS snapshot management with snapshot holds, clone creation and promotion, and assigning permissions to snapshot-related operations.

Manipulating a Pool from the Rescue System

We’ve all been there: that moment of panic when a system fails to boot back up. Perhaps there was a glitch with an upgrade. Maybe you’re wondering if you fumble-fingered a typo when you made that last change to loader.conf.
Fortunately, with FreeBSD and its built-in rescue mechanisms it is possible to quickly recover from most scenarios that prevent a system from booting into normal operation. And if you’re using OpenZFS, you can rest assured that your data is intact.
With this article, let’s take a look at some common recovery scenarios.

Tell us what you think!