A Quick Look at the History of Package Management on FreeBSD

A Quick Look at the History of Package Management on FreeBSD

How FreeBSD's package management system evolved over time

Pkgng became FreeBSD's official package manager in FreeBSD 10 in 2014. A new era in package management had begun for FreeBSD.

A Quick Look at the History of Package Management on FreeBSD

There are a lot of great stories about how computing tools come about—for example, Jeff Bonwick's proposal to create the ZFS file system—but unfortunately, not all tools are as well documented or storied. Today, we'll take a look at how FreeBSD's package management system evolved over time.

Every operating system needs a way to install third party applications. These tools are commonly called package managers. In FreeBSD, applications can be easily installed from either pkg—a system managing precompiled binary packages—or the ports tree, which automates building and installation of packages directly from their source code.

The Ports System

In the early days, BSD didn't have a package manager. In Andrew Pantyukhin's essay "Third-party software management under BSD" he noted that BSD was originally designed "with rich userlands so that users might have a chance to never think about anything third-party." This is a fancy way of saying that BSD (and other systems) were designed to come preinstalled with all the software that the user would need.

Those that wanted to install software often had an uphill battle. Pantyukhin commented further, "Before package management existed, as we know it now, developers preferred spending their time troubleshooting installation issues to thinking about deinstallation. Back then, a user usually had to get a file archive, extract it, optionally hack it and compile it, and install it."

Into that breach came Jordan K. Hubbard, one of the founders of the FreeBSD project and creator of the FreeBSD port system that we have today. According to Michel Talon's essay on the FreeBSD ports system, it is "a complex collection of tools and data files allowing to retrieve, compile and install software packages on a computer in a coherent way".

The NetBSD team interviewed Hubbard on the tenth anniversary of their package manager pkgsrc, which is based on the FreeBSD port system. When asked why he created the ports system, Hubbard said, "As much as I'd like to say that it all suddenly came to me in a blinding flash of inspiration one day, the truth is somewhat more prosaic. I was simply getting more and more frustrated with the fact that every new FreeBSD system I built and installed (and I was building quite a few PCs in those days) needed to be customized with pretty much the same "extra bits" every time (bash, emacs, MH, and so on) and every time was also basically the same old drill - remember where to find the bits, fetch and unpack the bits, configure/patch the bits as necessary, build and install."

Hubbard decided to automate the problem. As he said in the interview, he used his knowledge of make "to quickly whip up what became bsd.port.mk and create a couple of dozen ports to test and refine the concept. In a couple of weeks I had all my favorite bits of software under the new system". In August 1994, he felt the code was ready and made it available to the world. Hubbard pushed bsd.port.mk to the FreeBSD CVS repo with the following comment: "Commit my new ports make macros. Still not 100% complete yet by any means, but fairly usable at this stage."

He followed this up by uploading ports for a couple of his favorite tools: emacs, jove, and bash. In a short period of time, the ports collection grew to over 200. Once he saw that the ports were growing in popularity, Hubbard decided to simplify it. As he told the NetBSD interviewers, "it was also becoming clear to me that we were going to need some sort of package management system which allowed end-users to have all the benefits of make install in the ports collection without having to actually go through all the intervening steps, so I dragooned Satoshi Asami...into the role of Ports Meister and went off to write pkg_install(1) as the logical counterpart to ports".

And thus, the ports system we FreeBSD users know and love was born.

PKGNG - A New Generation of Package Management

The ports system served FreeBSD well over the years, but it was more of a stopgap than a permanent solution. The ports system worked well for advanced users and server administrators, but beginners tend to have a harder time with it.

The ports system had other shortcomings. In fact, the creator of the port system left a comment in the code stating it was ugly and badly written:

This is seriously ugly code following. Written very fast! [And subsequently made even worse.. Sigh! This code was just born to be hacked, I guess.. 🙂 -jkh

Baptiste Daroussin gave a talk at the 2012 BSDCan entitled "Modernising FreeBSD package management". He underlined some problems with the port system.

First, ports offer no support for upgrading packages—to upgrade a package, you need to uninstall it and then install a new version of the port. Second, ports are missing important metadata, such as the license of the package and the website for the package. Third, the ports system doesn't do a great job of dependency tracking and could easily break installations by removing dependencies needed by other applications. Finally, the ports system does not work with third party repositories.

To combat these issues and give FreeBSD an updated package manager, Daroussin and his team worked on a new system. They started work in 2010 and released the first version of the pkgng program on August 30th, 2012. The release notes state that pkgng is "a tool that is designed to replace pkg_install and provide modern features and advanced package management for FreeBSD".

Pkgng uses a SQLite database, making it very easy to search for metadata and other required information. In Joe Maloney's article on pkgng, he says "storing the information about packages in a database makes searching for information about installed packages fast. It also helps make installation and removal fast, and it allows for easier upgrades."

Pkgng also simplified everything by using a single command (pkg) instead of the multiple commands used previously, such as pkg_add, pkg_create, pkg_delete, pkg_info, pkg_updating, and pkg_version. Pkgng didn't do away with the ports tree, but works with it. Daroussin said that pkgng was designed to use the existing ports without their maintainers needing to modify their packages.

Pkgng became FreeBSD's official package manager in FreeBSD 10 in 2014. The release notes state "The pkg_add, pkg_create, pkg_delete, pkg_info, pkg_updating, and pkg_version utilities have been removed. pkg(7) must now be used to install binary packages. pkg(7) is the next generation FreeBSD package manager, also referred to as "pkgng". If upgrading from a previous FreeBSD release, be sure to run pkg2ng# to convert the old package database to the new format."

Thus, a new era in package management had begun for FreeBSD.