httm – The Hot Tub Time Machine is Your ZFS Turn-Back-Time Method

If you're anything like me, when you accidentally delete something you shouldn't have, you feel a sudden stab of awfulness about it Digging through backups to find the latest copy of something you lost can make you feel, however irrationally, like a bad person. 

Don't worry. You don't need therapy. Whenever this happens to me—and it does happen to me—I remember that I wrote a little tool called httm.  Using httm helps me find my lost files much more quickly, whichmakes the associated I'm a moron feeling pass more quickly as well. 

httm does lots of other cool things too, a few of which I hope I will have time to touch upon, but let's consider The Case of The Infamous Fat Fingered Sysadmin first. 

The Case of The Infamous Fat-Fingered Sysadmin 

As a system administrator, you may occasionally try a new software package which leaves some traces of itself behind when you remove it. That cruft might be essentially harmless—but if you're like me, cruft in your /etc or even your /usr/share/man folder will drive you nuts. 

If you’re not known for your skinny fingers or light touch, you might also accidentally remove the occasional entire folder when you only meant to remove a file: 

% cd /usr/share/man 
# When you obviously meant: sudo rm -rf ./man1/ubuntu-advantage.1.gz! 
% sudo rm -rf ./man1/ ubuntu-advantage.1.gz 

Oops, you've managed to delete your man1 folder. The folder that contains the documentation for the most fundamental tools on your system (ls, rm, even man).  

Before those misty tears form, you remember you set your system up with automated ZFS snapshots—so you’re sure you’ve got a copy of the lost data, but now you’ve got to trawl through your snapshots trying to find it. 

Happily, this is exactly where httm can help! 

Like any good tool httm lets you ask questions, so let's ask a few questions: 

1. Do you have any copies of man1 readily available in snapshots of the ZFS dataset which should contain it? 

% httm /usr/share/man/man1

─────────────────────────────────────────────────────────────────────────────────────────────────────────── 

... 
Mon May 23 00:12:39 2022  1.3 KiB  "/.zfs/snapshot/autosnap_2022-05-23_10:00:59_hourly/usr/share/man/man1" 
Mon May 23 09:55:08 2022  1.3 KiB  "/.zfs/snapshot/snap_5d7f8f8d_prepApt/usr/share/man/man1" 
Mon May 23 13:03:37 2022  1.3 KiB  "/.zfs/snapshot/snap_560b8ba2_prepApt/usr/share/man/man1" 

─────────────────────────────────────────────────────────────────────────────────────────────────────────── 

Mon May 23 13:03:37 2022  1.3 KiB  "/usr/share/man/man1" 

─────────────────────────────────────────────────────────────────────────────────────────────────────────── 

2. Do you have any copies of man1 available on any locally available ZFS dataset? 

% httm -a /usr/share/man/man1 

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 

... 
Thu May 19 17:50:09 2022  1.3 KiB  "/data/rpool/ROOT/ubuntu_wi01vc/.zfs/snapshot/autosnap_2022-05-20_00:00:42_daily/usr/share/man/man1" 
Sat May 21 18:14:26 2022  1.3 KiB  "/data/rpool/ROOT/ubuntu_wi01vc/.zfs/snapshot/autosnap_2022-05-22_14:01:04_hourly/usr/share/man/man1" 
Sun May 22 11:29:08 2022  1.3 KiB  "/data/rpool/ROOT/ubuntu_wi01vc/.zfs/snapshot/autosnap_2022-05-23_01:00:37_hourly/usr/share/man/man1" 
... 

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 

Mon May 23 13:03:37 2022  1.3 KiB  "/usr/share/man/man1" 

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 

3. Obviously, this is great! You don't have to search through your .zfs/snapshot directory. You don't even have to remember upon which ZFS dataset /usr/share/man/man1 is located. 

Now, you want to restore. Since the parent directory man once contained man1 we start there, and follow the dialogs to select a snapshot from which to restore man1, like so: 

% httm -r /usr/share/man/ 

This gets you an interactive menu, showing all snapshots which contain the file you’re looking for. You can arrow through the menu and hit enter to select a snapshot, after which httm asks if you’re certain you want to overwrite the current file with a copy of it from the snapshot you chose. 

The good times don't stop here. Consider Our Complex Feelings about Apple's Time Machine. 

Our Complex Feelings about Apple's Time Machine 

Speaking as a more than occasional Mac user, Apple may have taken a few wrong turns, but the Time Machine concept wasn't one of them. It introduced backups and data recovery to average users, and made backing up your data a relatively easy and (sometimes) painless process. 

The actual Time Machine implementation on the other hand... well, it is very slow, and can sometimes be very flaky. It could have been done much better, with greater integrity, if the underlying storage substrate was ZFS instead of HFS+, or now APFS. 

Of course, for dataset-level recovery, ZFS has you covered (zfs send/recv, zfs rollback, zfs clone). However, ZFS itself provides no automation of this file-level/Time Machine-style/personal computing use case.  

ZFS gave us the abstractions to work those file-level problems out, but, for one reason or another, never provided an end-user-friendly mechanism addressing the "I need a copy of that file from 2 days ago, and that 2 day old version is on a ZFS snapshot which is located on a remote server" use case.  

Just as Time Machine solved that problem for many Mac users, ZFS and httm can do something similar for ZFS users. 

Let's mount a remote share and view our snapshots there: 

# mount a remote share from your Mac 
% open smb://<your name>@<your remote share>.local/Home 
# set the location of you snapshot share point and local relative directory 
# these variables can also be set at the command line via --snap-point and --local-dir 
# more simply -- your local-dir is what you backup to your snap-point is where you backup 
% export HTTM_SNAP_POINT="/Volumes/Home" 
% export HTTM_LOCAL_DIR="/Users/<your name>" 
# execute httm to browse snapshots on the remote system from your Mac for your 2 day old file 
% httm -i -R ~ 

But you may ask: "How does this work? MacOS doesn't have native ZFS snapshots for my home directory? At least not for me." Happily, httm also works with rsync-ed datasets, like so: 

# transfer your home directory 
% rsync --inplace -avhe "ssh -T" --owner=<your name> --group=<your name> --progress \ 
    --delete "/Users/<your name>/" <your name>@<your remote share>.local:"/data/Home/" 
# then snapshot your home directory (you may need to add yourself to /etc/sudoers) 
% ssh <your name>@<your remote share>.local -t \ 
"/bin/sync && /usr/bin/sudo /usr/sbin/zfs snapshot data/Home@snap_"$( /bin/date +%F-%T )"_HomeSync" 

Again, pretty neat, right? Well -- get ready to have your mind blown by all the fun we can have at the command line... 

The Shell Gun Slinger 

ZFS was born on UNIX, and it was my goal to make a utility for the real UNIX and ZFS lover, because I am one. But, as we all know, we can be a somewhat skeptical and somewhat traditional bunch. We like what we like, and something new... it better work with the things we like! 

You might be thinking "httm and its interactive modes are fine for average users, but a TUI is practically a GUI—and let's face it, GUIs are still suspect! What does httm offer a simple UNIX and ZFS lover, surviving by wits and the light of a pseudo-tty alone?" 

I will try to answer. Imagine you want to just get the raw file locations of unique file versions of your .zshrc residing on snapshots (and none of that additional pretty printed nonsense). You, sir or madam, are going to love httm's -n (print raw output newline-delimited) and -0 (print raw output null-delimited) and --no-live (toggles whether you want to see "live" file versions included in the output) switches which will discard any additional formatting, so you can actually work with the data. 

% httm -n --no-live ~/.zshrc 

/home/rswinford/.zfs/snapshot/snap_35fcc113_prepApt/.zshrc 
/home/rswinford/.zfs/snapshot/autosnap_2022-05-23_09:00:56_hourly/.zshrc

Unix Lovers Trivia Q: Why might we want to sometimes want to NUL delimit file names? 
A: Because Unix allows almost any character in a filename, including whitespace, newlines, commas, pipe symbols, but not NUL. 

You  could, of course, produce something similar with grep and cut. But the point is that httm works well with the other tools CLI users use every day. 

Perhaps you want to view the differences between each unique snapshot versions of your .zshrc and the live file? httm doesn't break a sweat: 

% filename="$HOME/.zshrc" 
for version in $(httm -n $filename); do 
    # check whether files differ (e.g. snapshot version is identical to live file) 
    if [[ ! -z "$( diff -q  "$version" "$filename" )" ]]; then 
        # print that version and file that differ 
        diff -q  "$version" "$filename" 
        # print the difference between that version and file 
        diff "$version" "$filename" 
    fi 
done 

Files /home/rswinford/.zfs/snapshot/autosnap_2022-05-19_00:00:42_daily/.zshrc and /home/rswinford/.zshrc differ 
118c118 
< alias fzf="sk" 
--- 
> #alias fzf="sk" 
155,156c155,158 
< source ~/.zsh/skim-key-bindings.zsh 
< source ~/.zsh/skim-completion.zsh 
--- 
> #source ~/.zsh/skim-key-bindings.zsh 
> #source ~/.zsh/skim-completion.zsh 
> source /usr/share/doc/fzf/examples/completion.zsh 
> source /usr/share/doc/fzf/examples/key-bindings.zsh

Or you want to send all the unique versions of your .zshrc in a compressed package to a friend in an email? httm has you covered: 

% httm -n ~/.zshrc | tar -zcvf all-versions-zshrc.tar.gz -T - 

So far, you might still feel like a few sticks of software bubble gum and a few rolls of software duct tape (find, awk and a few pipes) could produce something similar enough to httm to work as well in practice. 

But httm allows me to chase my intuitions about certain things. For instance, I might know a file has been deleted from a folder, but I don't actually know which file was deleted, which folder it was in, or when it was deleted. 

httm can list all the deleted file versions from a folder and redirect that list to a text file: 

# httm search deleted file in recursive mode, print raw file locations with no live versions 
% httm -d -n -R --no-live ~ > deleted-file-versions.txt 

... 
/home/rswinford/.zfs/snapshot/autosnap_2022-05-01_00:00:22_monthly/.oh-my-zsh/plugins/boot2docker/README.md 
/home/rswinford/.zfs/snapshot/autosnap_2022-05-01_00:00:22_monthly/.oh-my-zsh/plugins/boot2docker/_boot2docker 
/home/rswinford/.zfs/snapshot/autosnap_2022-05-01_00:00:22_monthly/.oh-my-zsh/plugins/boot2docker 
...

Of course, those are the deleted file versions, and there may be dozens of them for each deleted file. What if you want to know what names did the files have before they were deleted? 

Remember httm works with your other tools, including grep: 

# httm search deleted file in recursive mode, print raw file locations 
%  ~ httm -d -n -R ~ | grep -v '.zfs/snapshot' > pseudo-live-file-versions.txt 

... 
/home/rswinford/.cargo/bin/httm 
/home/rswinford/.cargo/bin/bat 
...

Now I can see that I removed the httm and bat commands from my .cargo/bin path and installed native packages for each. 

I've had fun imagining fun ways to use httm, but I'm interested in what you might come up with. Pull requests are always welcome to the "Example Usage" section of the httm project README for especially creative and UNIX-y uses of httm. 

fzf and zsh hot keys for the kids 

Just as httm is built to be used with the tools you enjoy, httm is inspired by other tools I enjoy. I'd be remiss if I didn't mention fzf as an influence. fzf, and its Rust clone 'skim' or sk, are so-called fuzzy finders—meaning that if you feed them an input, you can search and ultimately select an item for output. For httm, a fuzzy finder is your fundamental user interface. 

Fuzzy finders can be used in combination with your shell to make common tasks much quicker. If you're like me, you search your command history often. Once upon a time, one would just alias history | grep to histg and type histg failed when one wanted to discovered the commands one had previously used which contained the word failed. But these days, with a fuzzy finder and shell hot keys, who has the time? You haven't lived until you've used fzf for a history search. 

httm also has first class support for zsh hot keys of its own, which you can install by typing --install-zsh-hot-keys, and, of course, you are free to write your own. One fun use of the httm hot keys is immediately finding yourself in a restore session, and being able to quickly restore files when you make that dreaded fat finger flub (ctrl + m). Another fun use of zsh hot keys is to dump snapshot version locations to your command line, like so: 

# Type `less` and then type `esc + s` 
% less  
# This will dump you to a `select` mode where you can select a snapshot to dump to the terminal buffer 
% less /home/rswinford/.zfs/snapshot/autosnap_2022-05-19_00:00:42_daily/.zshrc 

httm and file level snapshots 

If you've never used inotifywait (available on both Linux and FreeBSD), it's pretty handy for when you want to snapshot a directory after a file is accessed, written to, or moved to a watched directory.  For example: 

% inotifywait -r -m --exclude "/srv/downloads/incomplete/" -e moved_to "/srv/downloads/" | while read -r line; do 
        sudo /usr/sbin/zfs snapshot rpool/downloads@snap_"$( /bin/date +%F-%T )"_completionSnap 
done 

However, imagine you're watching a large directory tree where there may be many ZFS datasets, how would you know which dataset to snapshot? 

httm has your back! 

If you're comfortable with httm making a few choices for you, httm can also take a snapshot of the containing dataset using just the filename as input: 

% inotifywait -r -m --exclude "/srv/downloads/incomplete/" -e moved_to "/srv/downloads/" | while read -r line; do 
        sudo httm --snap "$line" 
done 

Or, more simply: 

% sudo httm --snap /var/log/syslog 
httm took a snapshot named: rpool/ROOT/ubuntu_wi01vc/var/log@snap_Jun-25-2022-23:11:50_httmSnapFileMount

httm is the file level snapshot tool you never knew you wanted! 

I hope you've enjoyed this extended introduction to httm. I really hope you use and enjoy httm, and perhaps even contribute something back. If you're a little like me, I think you'll like it.