From its birth at Sun, ZFS grew exponentially in popularity. Many were impressed by its revolutionary features, and ported it to run on their systems. They were able to do this thanks to Sun open sourcing the code. As it was expanding onto other operating systems, its mother company was swallowed up by Oracle, who shut the door on access to the code. From the ashes grew the OpenZFS project, which continues to make ZFS available to those outside of Oracle’s walled garden.
The release of ZFS to the public had a big impact on the business world. “ZFS was revolutionary for completely decoupling the filesystem from specialized storage hardware and even a specific computer platform.” ZFS allowed anyone with any budget to have access to enterprise-level storage solutions.
These advancements were soon available to those outside of the Solaris customer base. During its early development, the code for ZFS had been proprietary. However, Sun was a big supporter of open-source ideals, unlike most tech companies at the time. So, it wasn’t long before they open-sourced most of ZFS’ code.
In 2005, Sun launched their OpenSolaris initiative. This meant that the Solaris 10 code base (renamed OpenSolaris) was now licensed as open source. It was licensed under the “Sun’s Common Development and Distribution License (CDDL) — a modified Mozilla open-source license recently approved by the Open Source Initiative (OSI).” Sun CEO, Scott McNealy, stated that “the OpenSolaris project would include more than 1,600 of Sun’s patents associated with the Solaris OS”. These included the ZFS code base.
With ZFS free from an oppressive license, it didn’t take long for it to be integrated into other operating systems. FreeBSD added “experimental support for Sun’s ZFS filesystem” in FreeBSD 7.0, in 2007. This was all thanks to the efforts of Pawel Dawidek starting in 2006. ZFS support was declared production ready in FreeBSD 8.0 in late 2009.
In 2008, Brian Behlendorf launched a project named ZFSOnLinux to make ZFS available on Linux systems. Originally, this did not include the ZFS POSIX Layer (ZPL), the part of ZFS that presents a dataset as a normal filesystem, but as the popularity grew, this feature was ported as well. Unfortunately, ZFS cannot be included in the Linux kernel because the two licenses used (CDDL and GPL respectively) are not compatible. “However, it can be distributed as a separate DKMS package for the core package.” (In 2019, Canonical integrated ZFS into their 19.10 release allowing ZFS on root.)
The attempts to port ZFS to Mac OS have a more storied history. In April 2006, a message appeared on the OpenSolaris mailing list revealing that Chris Emura, Apple’s Filesystem Development Manager, was interested in porting ZFS to MacOS. This was followed by an announcement by Sun CEO, Jonathan Schwartz, in early 2007 stating that “this week you’ll see that Apple is announcing at their Worldwide Developer Conference that ZFS has become the filesystem in Mac OS 10.”
After initially denying it, Apple announced in 2007 that they would be including a “read-only version (of) ZFS” in Mac OS X 10.5 Leopard. Apple representatives made sure to stress that “ZFS is not the default filesystem for Leopard”.
Apple’s official port of ZFS floundered for a while before being dropped in favor of one of Apple’s many attempts to rewrite their default filesystem, HFS+. There are multiple reasons floated for Apple managements’ ultimate decision to drop ZFS. These include Steve Jobs being angry with Schwartz for letting the cat out of the bag early, being afflicted with the not invented here syndrome, and uncertainty about the future of Sun.
Just because Apple decided to drop support for ZFS, that didn’t mean that ZFS was no longer available on Mac OS. In 2008, Dustin Sallings created the MacZFS project with the goal of “continuing where Apple left off with ZFS”. The MacZFS project ceased in 2013. In 2014 the effort was resumed, partly based on the ZFSOnLinux repo, under the name OpenZFS on OSX, which continues to this day.
As ZFS was growing in popularity, a dark cloud was building on the horizon. Sun was in trouble. ZDNet noted that “the emergence of commodity Linux x86-based servers…and pressure from other high-end enterprise systems vendors such as IBM and HP” signaled problems for Sun. There were rumors that IBM, Fujitsu, or HP were planning to purchase Sun. In fact, Sun rejected a deal from IBM because they considered the offer too low.
In a surprising move, database company Oracle purchased Sun Microsystems for $7.4 billion in 2010. This move shocked many because Oracle had never been in the hardware business. Oracle was known for being very protective of their intellectual property and preferred to keep its software proprietary. So, it wasn’t surprising that shortly after the acquisition, Oracle “ceased public development of both ZFS and OpenSolaris“. OpenSolaris was just one of many open source projects shut down by Oracle. In the wake of the acquisition, many executives and innovators left to find better opportunities elsewhere.
According to Matt Ahrens, “This raised big concerns about the future of open source ZFS”. Later that year, former users and developers of OpenSolaris “forked the last public release of OpenSolaris as the Illumos project”.
With ZFS code once again being open sourced, many switched to the new code base. “FreeBSD and Linux treated Illumos as their upstream for ZFS code.” However, the vast majority of ZFS users were not Illumos users “thanks significantly in part to FreeNAS which uses the FreeBSD operating system”. ZFS may have been available to the community once again, but there was very little cooperation between the different platforms and there was a lot of duplication of effort.
In a way this was not too different from the Sun days. In 2014, Matthew Ahrens spoke at AsiaBSDcon about the history of ZFS. He noted that “the vast majority of ZFS development happened behind closed doors at Sun”. At the time, very few people from outside of Sun made contributions to ZFS. This was because “it did not have an open development model”.
In an effort to prevent fragmentation, it was decided to create a separate project. In 2013, Matthew Ahrens and other ZFS developers announced the creation of a fork of ZFS named OpenZFS.
This new project was created with three goals in mind: to manage communication between all OpenZFS developers, to ensure that users have a consistent experience when using OpenZFS, and to expand public awareness of OpenZFS. One of the biggest ways that OpenZFS engages the community is by hosting a yearly OpenZFS Developer Summit conference to promote the filesystem and the companies who use it.
OpenZFS has inspired a couple of new ports. OpenZFS on OS X (also known as O3X) was created to make OpenZFS available on Mac OS. Currently, it supports OS X 10.8 through macOS 10.15. Support for macOS 11 is under development. There is also an effort underway to port OpenZFS to Windows via OpenZFS on Windows. This is a fork of O3X and is considered to be alpha quality.
In the meantime, Oracle has continued to develop their own internal version of ZFS for their own use. As a result, “there are, sadly, two politically and technologically-incompatible branches of “ZFS”. OpenZFS is by far the more popular of the two, and widely available because of the open-source license the project uses. It is doubtful that the two code bases will ever reach parity or share code because of Oracle dislike of open-source.
The story of ZFS is still being written, but we have not gotten to current day yet. Check back later, or better yet subscribe, to find out what happened next and how the future of OpenZFS looks like.
Stay tuned for Part 3.
What first looked like some CPU oddity in a monitoring graph became a full-fledged hunt for a bitcoin miner. Figuring out what happened and how they were operating is part of the investigation. Collecting evidence to make a case soon became akin to a detective story.
To expand your OpenZFS storage infrastructure to meet constantly evolving infrastructure demands, you must understand where the limits of your current system lie. Only then can you plan to expand to provide the capacity, durability, and performance that modern applications demand.
OpenZFS and storage in general is a complex and important part of any project’s architecture. It should be planned thoughtfully and ideally, ahead of time! In this article, we’ll talk about how to understand, measure, and plan for your storage performance needs.
Pingback: New top story on Hacker News: History of ZFS and how OpenZFS was born - Welcome to world wide tech news
There are a lot of details peculiarly left out here. There is no mention whatsoever of the openindiana (openindana.org) that was established after the takedown of OpenSolaris. It was initially based on OpenSolaris but over the years they transitioned the upstream to that of IllumOS. OpenIndiana is now officially the open-source distribution of the Solaris operating system. They don’t call it OpenSolaris in order to avoid being sued by Oracle. While it doesn’t have as wide hardware support as Linux/FreeBSD, it works well on selected PC hardware. It was developed along with another distribution named NexentaOS.
If you want a stable and well-maintained implementation of ZFS I recommend running it on OpenIndiana. Nexenta is good too but I don’t think it is free of charge.
Thank you for the article. I wonder if you have any recollection or pointer of how some of the zfs function/features come along. Especially promote, clone, snapshot, send and receive. Are they available since Sun Microsystem time or added later on?