FreeBSD Support: Greatest Hits

Klara provides development and support solutions around FreeBSD and OpenZFS to a wide range of customers, from the largest appliance vendors and infrastructure providers in the industry, the most well-respectedresearch and academic institutions, to the humblest e-commerce website. Each presents novel problems that require innovative solutions and allow us to build on our wealth of expertise. 

In the long and storied tradition of such things, let's take a look at some of the most interesting support cases that have come across our desks. 

Disappearing File Handles 

An institution that provides R&D infrastructure for public sector clients reported a rather odd issue. Clients connected to one specific NFS share would experience issues, and “stale file handle” errors are not-quite-regular intervals. The problems would most often happen around 6 am or noon, but never at exactly the same time; there was always a bit of variance that suggested the problem was not starting at exactly noon, but something like 12:00:46 one day, and 12:01:12 the next. 

After investigating, it was determined that the NFS shares were bulk research data that was effectively read-only and was being incrementally replicated via ZFS from another system. So once every 6 hours, the primary server would reach out and connect to the NFS server and push a set of incremental changes, usually only a few megabytes of metadata. However, because the dataset on the NFS server was not mounted strictly read-only, ZFS replication was run with the force flag and had to perform a `zfs rollback` to undo the local modifications (generally just updating the last-modified data of the root directory when it was mounted). This rollback requires ZFS to unmount the filesystem, swap the object set underneath to a version matching the unmodified snapshot, and then re-mount it. Thus, any attempt to use a file handle from the now unmounted filesystem causes an error, to ensure the applications accessing the data over NFS know that the file may have changed due to the rollback. 

Changing the replication schedule to avoid peak working hours, and applying the `readonly` property to prevent local modifications that would need to be rolled back before the next incremental receive, resolved the issue while maintaining the ability to offload the majority of the “read” workloads to a separate mirror server, preserving capacity on the primary server. 

A similar technique is used by another client to ensure large research datasets are available at multiple locations, to provide high-speed access to the data, while still allowing changes to be coordinated via the primary site. 

Expired Routes 

Modern technology provides a lot of flexibility, but with that flexibility comes some complexity. When everyone can configure their system differently, they can end up encountering edge cases that no one else ever has. Add to this that as technology evolves, we don’t always want to entirely give up the legacy technology, and in no place is that more prevalent than networking. 

When an IPv6 address is configured, it can have a “lifetime”, an amount of time it is good for before it expires, to either be renewed or replaced with a different address. Tied to that, the “prefix” (network route) has a “lifetime” as well, usually tied to that of the address. 

It was discovered that clients using DHCPv6 with a short lease time would apply that lease time as the lifetime of the IPv6 address, as it should, but when the address was updated with a new lifetime when the lease was renewed, the prefix was not being updated as well and would expire, causing the route entry to be removed. 

The fix was fairly obvious: update the lifetime of the prefix when the lifetime of the address is updated. However, it is never that simple. What if there are multiple addresses on the interface within the same prefix? What if the newest address to be added has a shorter lifetime than the existing prefix? 

All of these edge cases had to be handled, so the code went from: 

1. When adding a new address: if the prefix does not exist, add it with the same lifetime 
2. When updating an address, do not adjust the prefix 

To: 

1. When adding a new address, if the prefix does not exist, add it with the same lifetime 
2. When adding or updating an address, if the prefix lifetime is sooner than the new address, update the prefix lifetime to that of the new lifetime 

Slowest Boot Ever 

Sometimes architectural decisions have unintended consequences, such as the case of this academic institution that often finds itself the target of nation-state actors trying to compromise it systems. To harden its systems against that, they all run from a read-only NFS root, so a compromised machine cannot modify the operating system it is running. The architecture also features a robust audit and logging system to ensure there is a trusted audit trail for every action. 

The problem started when a specific infrastructure server seemed to hang during the early boot process. As part of the NFS boot setup, these servers would PXE boot to load the FreeBSD loader, which would then load the kernel to mount the NFS root. When this was taking excruciatingly long on these specific servers, going hours without completing, the client’s first thought was to blame the network adapter, as the specific brand had a reputation for being problematic. However, even after switching the NIC to a known good model from a different vendor, the problem persisted. 

Inspection of the network traffic stream revealed an interesting detail: the problematic server was the logging server, so while it was running only the boot loader and attempting to load the kernel over the network, it was being constantly barraged with syslog packets. A bug in the loader code meant that if the loader needed to transmit a packet (requesting the next block of the kernel it was loading), it would interrupt the in-progress receive of a packet, resulting in packets being dropped. 

A small patch to the PXE loader was able to improve the situation, but to reduce the amount of traffic that the loader has to examine, the configuration was also changed so that the syslog packets were sent to a different IP address, bound to the log server, but not the assigned DHCP address used for PXE boot. This reduced the amount of traffic the PXE loader needed to sort through, and further improved the boot speed of this critical server, while also allowing log traffic to be redirected to another machine pre-emptively to ensure no logs were missed during scheduled maintenance. 

There Should Be a Way 

Sometimes a support case just comes down to “wouldn’t it be nice if there was a way to accomplish these two things at the same time”. 

Let’s start with a bit of background. FreeBSD has a Mandatory Access Control (MAC) subsystem that provides for customized security rules. It also comes with a number of default modules and mechanisms. One of those is “See Other UIDs”, which went enabled, restricts the ability to see the processes and sockets owned by other UIDs. It has some configurable options, including “allow visibility of processes and sockets owned by the same primary group” and “allow this one specific group ID to see all processes”. The latter is designed to allow monitoring systems visibility across the system, so they can ensure certain daemons and services are running, or that the number of web server processes is not too high, etc. 

However, that option is quite inflexible; there can only be one group ID. So if you have more than one service account that needs to be able to view other processes, you must either add the additional services to the group of the first, or try to force all of these services to run as a specific GID. 

Instead, Klara extended the “specific GID” configuration knob to take a space-separated list of GIDs and allow each on the list access to view all processes. Special care was taken to normalize and sort the list using the same mechanism used in user credentials by the kernel, to ensure the comparison is fast and linear rather than quadratic. 

Support 

These are just a few of the more memorable FreeBSD support cases we have dealt with recently. Sometimes it is just a simple patch to add a small capability, getting to the bottom of a really odd behaviour, or finding a clever solution to a problem no one else has. When you run into a problem in production, having access to experienced resources that can find the issue, fix it, and upstream the fix so that it will be in every future version of FreeBSD can be invaluable. 

If you have a stubborn problem or would like to have access to the best FreeBSD Support, please get in touch with us.