A Simple File Monitoring Problem
Some time ago a customer presented us with a seemingly simple problem. They had a workflow where files were uploaded to a server with scp, and they wanted to run an action on the server after the upload had finished. Their initial solution was to use the inotifywait utility to watch the upload directory and wait for a CLOSE notification, upon which they would run their action. `inotifywait` can monitor a file hierarchy for accesses or modifications, such as file creation, writes, deletes, and so on. This solution worked perhaps 99% of the time, but once in a while inotifywait would fail to emit the CLOSE notification after an upload.
The root cause of the missing notifications had to do with how inotifywait is implemented under the hood. It used a FreeBSD-specific library, `libinotify.so`, which implements Linux's inotify interface in userspace using the `EVFILT_VNODE` kqueue filter. Linux's inotify and FreeBSD's EVFILT_VNODE are both system call interfaces which let an application passively monitor a file or directory for changes.
The interfaces themselves are quite different from each other; below is a sample function which monitors all files in a given directory for file close events and prints events when they occur, effectively the same functionality that our customer wants. First, the inotify-based version:
void
watchdir_inotify(const char *dirname)
{
struct inotify_event *ev;
size_t evsz;
int ifd, wd;
ifd = inotify_init();
if (ifd < 0)
err(1, "inotify_init");
wd = inotify_add_watch(ifd,dirname, IN_CLOSE);
if (wd < 0)
err(1, "inotify_add_watch(%s)", dirname);
evsz = sizeof(*ev) + NAME_MAX + 1;
ev = malloc(evsz);
if (ev == NULL)
err(1, "malloc");
/* Main loop. */
while (true) {
ssize_t n;
n = read(ifd, ev, evsz);
if (n < 0)
err(1, "read");
if (ev->len > 0)
printf("CLOSE: %s\n", ev->name);
}
}
This is straightforward: we create an inotify descriptor, ask it to watch the specified directory for IN_CLOSE events, and then loop, reading event descriptions from the descriptor. This can easily be integrated into a larger event loop by using kevent() or epoll() to wait for data to arrive on the inotify descriptor.
Here is a version which uses kqueue() and EVFILT_VNODE to try and implement the same functionality:
struct entry {
LIST_ENTRY(entry) link;
char name[NAME_MAX + 1];
int fd;
bool seen;
};
LIST_HEAD(entry_list, entry);
static void
scan(int kq, DIR *d, struct entry_list *entries)
{
struct dirent *dp;
struct kevent kev;
struct entry *e, *tmp;
int error;
LIST_FOREACH(e, entries, link)
e->seen = false;
rewinddir(d);
while ((dp = readdir(d)) != NULL) {
int flags;
if (strcmp(dp->d_name, ".") == 0 ||
strcmp(dp->d_name, "..") == 0)
continue;
/* Search for an existing entry. */
LIST_FOREACH(e, entries, link) {
if (strcmp(e->name, dp->d_name) == 0) {
e->seen = true;
break;
}
}
if (e == NULL) {
int fd;
/* Not found, create a new entry. */
fd = openat(dirfd(d), dp->d_name, O_RDONLY);
if (fd < 0) {
if (errno == EOPNOTSUPP)
continue;
err(1, "openat(%s)", dp->d_name);
}
e = malloc(sizeof(*e));
if (e == NULL)
err(1, "malloc");
(void)strlcpy(e->name, dp->d_name, sizeof(e->name));
e->fd = fd;
e->seen = true;
LIST_INSERT_HEAD(entries, e, link);
flags = EV_ADD | EV_CLEAR;
} else {
flags = EV_CLEAR;
}
/* Watch the file for close events. */
EV_SET(&kev, e->fd, EVFILT_VNODE, flags,
NOTE_CLOSE | NOTE_CLOSE_WRITE |
NOTE_RENAME | NOTE_DELETE,
0, e);
error = kevent(kq, &kev, 1, NULL, 0, NULL);
if (error < 0)
err(1, "kevent");
}
/* Remove entries that were not seen. */
LIST_FOREACH_SAFE(e, entries, link, tmp) {
if (e->seen)
continue;
LIST_REMOVE(e, link);
close(e->fd);
free(e);
}
/* Add the directory itself to the queue. */
EV_SET(&kev, dirfd(d), EVFILT_VNODE,
EV_ADD | EV_CLEAR,
NOTE_WRITE,
0, 0);
error = kevent(kq, &kev, 1, NULL, 0, NULL);
if (error < 0)
err(1, "kevent");
}
void
watchdir_kqueue(const char *dirname)
{
struct kevent kev;
struct entry_list entries;
DIR *d;
int error, kq;
LIST_INIT(&entries);
kq = kqueue();
if (kq < 0)
err(1, "kqueue");
d = opendir(dirname);
if (d == NULL)
err(1, "opendir(%s)", dirname);
/* Monitor all files already in the directory. */
scan(kq, d, &entries);
/* Main loop. */
while (true) {
error = kevent(kq, NULL, 0, &kev, 1, NULL);
if (error < 0)
err(1, "kevent");
if (kev.udata == NULL) {
/* Directory itself changed, rescan. */
scan(kq, d, &entries);
} else {
struct entry *e;
e = (struct entry *)kev.udata;
if ((kev.fflags &
(NOTE_RENAME | NOTE_DELETE)) != 0) {
/* File was renamed, scan. */
scan(kq, d, &entries);
}
if ((kev.fflags &
(NOTE_CLOSE | NOTE_CLOSE_WRITE)) != 0)
printf("CLOSE: %s\n", e->name);
}
}
}We can immediately see that this is way more complex. EVFILT_VNODE operates on file descriptors and can only watch the file referenced by the file descriptor. This means that to watch a whole directory, we have to:
- traverse the directory and open each file, allocating a tracking structure for it and registering it with our kqueue;
- monitor the directory itself for changes, so we can rescan when files are added or removed;
- keep track of each file's name in the directory, including monitoring each file for renames.
All of this is quite a lot of work. Moreover, for large directories the performance is poor due to the need to scan the directory whenever entries are added or removed. Furthermore, since each file in the directory needs to be opened, the application may need to use many file descriptors, increasing kernel memory usage and potentially running up against descriptor limits. Finally, there is at least one race condition in the kqueue version: if scp(or any other utility) is adding a new file to the watched directory, the underlying operations will look something like this:
- create the file and open it for writing;
- write the data to the file using the file descriptor obtained in step 1;
- close the file descriptor.
Do you see the problem? Step 1 will cause the watchdir_kqueue() function to rescan the directory, open the new file, and register it with the kqueue. However, at the same time, the upload service is executing steps 2 and 3. If it finishes step 3 before the new file is registered, watchdir_kqueue() will not report the CLOSE event, as it happened before the file was registered. This is exactly the problem our customer was running into.
This can be seen in practice by running the following commands to create a new file in a watched directory:
$ echo a > fooversus
$ dd if=/dev/random of=foo bs=1M count=128
The former will run very quickly and has a good chance of triggering the race, while the latter should be more reliable at generating `CLOSE` notifications from the sample program. In any case, of the two solutions, the one using inotify is clearly preferable: it's simpler and more efficient, and at least on Linux, there is no such race condition. There's only one catch: until recently FreeBSD had no native inotify implementation. The `libinotify.so` mentioned above suffers from the same race condition by design, so while we can implement the inotify interface in software that use it, these bugs are unavoidable.
Towards A Solution
Confronted with these limitations of EVFILT_VNODE, it seemed clear that FreeBSD's file monitoring interfaces were inadequate, but the path forward was not immediately obvious. We looked at other operating systems to see how they handle file monitoring.
Other BSDs (OpenBSD, NetBSD and DragonflyBSD) also implement EVFILT_VNODE, but they also necessarily suffer from the same problems since EVFILT_VNODE,` requires each monitored file to be opened. NetBSD implements the inotify interface in the kernel as part of a Linux system call compatibility layer, but there it is also implemented on top of EVFILT_VNODE, rather than being directly integrated into the VFS, and thus inherits much of the same limitations and complexity.
Linux on the other hand has the inotify interface. inotify introduces a new type of file descriptor and lets one use it to monitor files and directories for filesystem events. Crucially, monitoring a directory will automatically cause all files in the directory to be monitored, as well as any any files added to the directory after monitoring starts. In contrast with EVFILT_VNODE,, there is no need for the application to open each file or keep track of file names.
Aside from inotify, Linux also has the fanotify interface, which provides similar but even richer functionality, such as the ability to intercept file accesses. In practice, inotify seems to be more commonly used in OSS, perhaps because it is older and simpler to use.
On MacOS, file monitoring can be done using the `FSEvents` API. This consists of a kernel component and a system daemon which runs in userspace and uses /dev/fsevents to obtain filesystem access event info, which is published to consumers. In particular, consumers do not obtain the event stream directly from the kernel. This design allows the system to implement event stream policies (e.g., how to handle dropped events) and state management in userspace, providing flexibility while keeping the kernel side relatively simple, but it is also rather complex and general for our modest use-case.
Inspired by FSEvents, we spent some time implementing a new kqueue filter, EVFILT_FSWATCH, which could provide some of the same functionality without a userspace daemon. The idea was to provide a way to monitor all events on all files in a mount point, since that is fairly easy to implement in the kernel. When registering the filter, the application also provides a pointer to an array of structures, each of which describes one or more events on a particular path. The kernel knows the layout of these structures and can fill them out when the application polls for new events by calling kevent(). It avoids the disadvantages of EVFILT_VNODE, but we only went as far as implementing a proof-of-concept and had not addressed various practical problems, such as handling dropped events, efficient path filtering, or access restrictions.
Why inotify?
Ultimately, we decided to implement a native inotify interface in FreeBSD. While inotify has shortcomings of its own--the inability to monitor file hierarchies recursively, poor control over back-pressure and event drops, among others--it is widely used in software run on FreeBSD (hence the need for a libinotify.so compatibility shim), and it is relatively straightforward to implement. Furthermore, a native, compatible implementation lets us use inotify in the Linuxulator "for free," which helps quite a few Linux binaries run directly on FreeBSD.
In fact, it turned out that the lack of a native inotify implementation was a sore point for quite a few FreeBSD users. There is a great deal of OSS which uses it, including foundational libraries such as GLib and Qt (which both provide higher-level interfaces on top of it); some software supports EVFILT_VNODE as an alternative selected at compile-time, but quite a few applications do not, and indeed, until recently, over 100 ports in the FreeBSD ports tree referenced the userspace libinotify directly.
While libinotify.so is very useful and is a key part of enabling standard desktop software on FreeBSD, the above-mentioned issues with EVFILT_VNODE caused a lot of frustration. Aside from the race conditions which motivated this investigation in the first place, the resource overhead of EVFILT_VNODE was prohibitive in some cases. For instance, syncthing, a popular file synchronization service, will log a warning when using kqueue to monitor a large directory due to frequent reports of high CPU and memory usage in that scenario. With our native inotify implementation, these problems will go away, though further work is needed to make FreeBSD's inotify symbols available in the golang.org/x/sys/unix package and then to make use of them in syncthing's inotify-based file event notification package.
Rather than having two competing interfaces in FreeBSD, it would have been nice to somehow extend EVFILT_VNODE to support the functionality we needed, but to do that we need a way for kevent() to export file names out of the kernel--removing the need for userspace to scan directories and keep track of file names--and that means that it is simply not a good fit for our problem. So we set ourselves the goal of implementing inotify natively, and in particular, staying source-compatible with Linux to ensure that existing code just works on FreeBSD.
The inotify Interface
The earlier example of watchdir_inotify() already shows much of the interface, which consists of three system calls: inotify_init(), inotify_add_watch() and inotify_rm_watch(). They respectively create an inotify descriptor, start monitoring a file or directory, and stop monitoring a file or directory. Calling close(2) on an inotify descriptor implicitly stops all monitoring associated with it.
The inotify_add_watch() system call takes an inotify descriptor, a path and an event mask, and returns a watch descriptor, which is effectively a pseudo-file descriptor that can be passed to inotify_rm_watch(). In particular, each monitored object gets a unique watch descriptor associated with the inotify descriptor. These watch descriptors are not subject to file descriptor resource limits, which allows many directories to be monitored simultaneously. However, they are subject to a global limit defined by the vfs.inotify.max_user_watches sysctl.
In FreeBSD we provide an extra system call, inotify_add_watch_at(2`, akin to openat(2), which takes a directory descriptor (or the AT_FDCWD constant) and a relative path. This is to enable the use of inotify in Capsicum sandboxes, as sandboxed programs do not permit filesystem accesses without a directory descriptor.
The example program in `/usr/share/examples/inotify/inotify.c` offers a more full-fledged example which demonstrates the different events that inotify can report. All it does is add a watch for each path listed on the command line and report all events that it receives. The inotify-tools package provides similar utilities; in FreeBSD 15.0 these and other third-party packages will make use of our native inotify implementation out of the box.
inotify in FreeBSD
After several months of development, testing and code review, our inotify implementation for FreeBSD has landed in the development branch and will be available in FreeBSD 15.0 and 14.4. It aims to be completely compatible with Linux, but due to design differences between the two kernels, there are some small differences in behaviour. We expect that the vast majority of applications will not notice them.
The main difference is in the way that files with multiple hard links are handled. Recall that a given file can have multiple names since files themselves do not contain a name, but rather are referred to by directory entries, which consist of a name and an inode number. The inode number uniquely identifies the file within its filesystem. Thus, different entries in a directory might refer to the same underlying file, and a file can have names in multiple directories. The following examples illustrate this:
Here we have a file with two hard links named link1 and link2 out of an abundance of creativity. Suppose an application uses inotify to monitor dir1 for events. What happens if a process opens the file via link2? On Linux, the documented behavior is that inotify will not report the open since only watches for accesses via the watched directory. On FreeBSD however, an access via either hard link will generate an inotify event, and the name recorded in the event will be that of the link in the watched directory.
Using the example program we can see this in action.
$ mkdir /tmp/dir1 /tmp/dir2
$ echo a > /tmp/dir1/link1
$ ln /tmp/dir1/link1 /tmp/dir2/link2
$ inotify --libxo json,pretty /tmp/dir1
Now, suppose we access `/tmp/dir2/link2` from a different shell:
$ cat /tmp/dir2/link2
a
The inotify program will report these accesses from `cat(1)` even though they did not come via `/tmp/dir1`:
{
"__version": "1",
"events": [
{
"wd": 1,
"event": "IN_OPEN",
"name": "link1"
},
{
"wd": 1,
"event": "IN_ACCESS",
"name": "link1"
},
{
"wd": 1,
"event": "IN_CLOSE_NOWRITE",
"name": "link1"
}
]
}
In the second example, we have only a single directory:
Here, if the directory `dir` is watched, any operation on the file via either hard link will generate two events, one for each link:
$ mkdir /tmp/dir
$ touch /tmp/dir/link1
$ ln /tmp/dir/link1 /tmp/dir/link2
$ inotify --libxo json,pretty /tmp/dir
Then, accessing one link with `cat(1)` again yields the following output:
{
"__version": "1",
"events": [
{
"wd": 1,
"event": "IN_OPEN",
"name": "link2"
},
{
"wd": 1,
"event": "IN_OPEN",
"name": "link1"
},
{
"wd": 1,
"event": "IN_ACCESS",
"name": "link2"
},
{
"wd": 1,
"event": "IN_ACCESS",
"name": "link1"
},
{
"wd": 1,
"event": "IN_CLOSE_NOWRITE",
"name": "link2"
},
{
"wd": 1,
"event": "IN_CLOSE_NOWRITE",
"name": "link1"
}
]
}
This behaviour is due to the low-level details of how the FreeBSD kernel keeps track of file names, which are quite different from Linux.
Specifically, in Linux, inotify events are tied to a particular open file handle, which contains the name of the link used to access the underlying inode. Thus, when a file descriptor is used to access a file somehow, e.g., calling read(2), Linux can generate an event using the name associated with the descriptor. FreeBSD, by contrast, does not store file names with file descriptions, so our inotify implementation needs to fall back to searching the name cache to find names for a given inode. This architectural difference is the cause of the divergence described above.
In practice, hard links are not very commonly used, and we expect that users will not notice this difference in behaviour. During development, we ran a number of third-party test suites against the our implementation, and none of them reported any problems due to this difference.
The FreeBSD inotify implementation also neatly handles file accesses via the NFS server. That is, if a watched directory is exported via NFS and accessed by a client, inotify events will be generated by the server, as can be seen with DTrace:
# dtrace -n 'vfs::vop_inotify:entry {stack();}'
dtrace: description 'vfs::vop_inotify:entry ' matched 1 probe
CPU ID FUNCTION:NAME
0 79459 vop_inotify:entry
kernel`VOP_INOTIFY_APV+0x129
kernel`vop_read_post+0x1b4
kernel`VOP_READ_APV+0xcd
kernel`nfsvno_read+0x313
kernel`nfsrvd_read+0xf04
kernel`nfsrvd_dorpc+0x549
kernel`nfssvc_program+0x1054
kernel`svc_run_internal+0x144f
kernel`svc_thread_start+0xb
kernel`fork_exit+0xa3
kernel`0xffffffff81b176ee
Similarly, inotify is interoperable with FreeBSD's nullfs filesystem, which allows arbitrary directories to be mounted elsewhere in the file hierarchy. That is, one can monitor a directory in a nullfs mountpoint and receive events for accesses via the "lower" layer. Conversely, monitoring a directory in the lower layer will also report accesses via the nullfs mount.
Wrapping Up
Finally, our inotify implementation aims to be compatible with Linux as much as possible, so it was trivial to add Linuxulator support. That is, Linux binaries which use inotify will now work on FreeBSD (assuming of course there are no other missing features). Quite a few applications make use of it in some way, so this should be a significant step forward for the FreeBSD emulation layer.
Klara helps organizations deploy, optimize, and support FreeBSD infrastructure at every scale, from storage platforms and virtualization environments to kernel development and performance engineering. Organizations looking to modernize and maintain FreeBSD infrastructure can rely on Klara Inc. for FreeBSD development, support, and focused solutions backed by deep kernel and systems expertise.