Automation and Hacking Your FreeBSD CLI

As much as natural language systems have advanced, computers are still bad at understanding output designed for humans. Components that can output structured, machine readable information provide a way for that data to be simply and reliably filtered, modified, formatted, and reused—improving the capabilities and functionality of the entire IT stack.

Consider the server operating system, a collection of small tools and
massive subsystems that work together for the benefit of the users. While many things are built to work together, countless tools require the data be reformatted before it is useful.

Managing all these systems manually can be time consuming, but with automation—which structured output greatly simplifies and empowers—we can reduce human error, boost productivity, and enable greater application interoperability.

The more we automate, the more we can save time and boost our productivity to higher levels.

The Need For Readable Outputs in Operating
Systems like FreeBSD

Many of the tools and applications on a unix system were developed with humans in mind, so their output is
meant for humans to read. This can mean aligning columns of output with a variable number of padding
characters, outputting scaled units, or just the addition of connective text and extraneous characters to
make it easier for a human to pick out the import data.

Take the not-only-FreeBSD `arp` command as an example:

# arp -na
? (192.168.64.13) at 52:88:80:9b:bb:00 on vtnet0 permanent [ethernet]
? (192.168.64.1) at d6:57:63:1f:93:64 on vtnet0 expires in 394 seconds [ethernet]
? (192.168.64.49) at a0:36:9f:42:60:fe on vtnet0 permanent published [ethernet]

Outputs like these are good for human consumption, but they’re very difficult for machines to parse.

Although sysadmin tools like awk, sed, and grep can be used to parse this sort of human-friendly output for
machine use, doing so is complex and can be fragile. For example, it might not be obvious that in some
cases, the eighth column of arp’s output is the keyword “published” instead of that entry’s
interface type!

Even if the admin gets every edge case right, there's no guarantee that a purely “cosmetic” upgrade to the
source’s human-readable output formatting won’t break the admin’s parsing routine in the future.

Debugging an ad-hoc human-to-machine parser can also be difficult, time consuming and painful. Awk, sed, and
grep are amazing tools—but admins have been complaining about the readability of complex regular expressions
for decades.Instead of spending tens, hundreds, or even thousands of hours building complex parsers to
convert human-readable output from our tools to reuse it in other programs, it’s better to offer
machine-readable output options (such as JSON or XML) in the first place.

FreeBSD libxo to the rescue!

libxo(3) is a library for emitting text, XML, JSON or HTML output. It was originally developed at Juniper by
Phil Shafer, and made its FreeBSD debut in 11.0-RELEASE.

libxo is made of two parts: the libxo(3) library that programs can use
to format their output, and the xo(1) utility that can be used in scripts.

Many programs support libxo(3) output, including but not limited to:

• arp
• df
• iscsictl
• jls
• last
• lastlogin
• netstat
• nfsstat
• procstat
• ps
• savecore
• sesutil
• vmstat
• w
• wc

Let's take arp as an example again—have you ever noticed that the man page has libxo
options?

SYNOPSIS

arp [--libxo options] [-n] [-i interface] hostname

Let’s see the output, shall we?

# arp -na --libxo json
{"__version": "1", "arp": {"arp-cache": [{"hostname":"?","ip-address":"192.168.64.13","mac-address":"52:88:80:9b:bb:00","interface":"vtnet0","permanent":true,"type":"ethernet"}, {"hostname":"?","ip-address":"192.168.64.1","mac-address":"d6:57:63:1f:93:64","interface":"vtnet0","expires":366,"type":"ethernet"}, {"hostname":"?","ip-address":"192.168.64.49","mac-address":"a0:36:9f:42:60:fe","interface":"vtnet0","permanent":true,"published":true,"type":"ethernet"}]}}

For direct human consumption, JSON is a
terrible format, and this output would be a clear step back from the default formatting. But it’s an
enormous improvement for machines!

One of JSON’s big advantages for machine reusability is that each
parameter is split out, and optional parameters can be excluded entirely. Each arp-cache entry only reports
the fields that are relevant—and extraneous things like the words “at” and “on”, and assumed unit values
like “seconds” are elided. If you need XML instead of JSON, libxo can do that too! It can even make it
pretty printed:

# arp -na --libxo xml,pretty
<arp version="1">
  <arp-cache>
    <hostname>?</hostname>
    <ip-address>192.168.64.13</ip-address>
    <mac-address>52:88:80:9b:bb:00</mac-address>
    <interface>vtnet0</interface>
    <permanent>true</permanent>
    <type>ethernet</type>
  </arp-cache>
  <arp-cache>
    <hostname>?</hostname>
    <ip-address>192.168.64.1</ip-address>
    <mac-address>d6:57:63:1f:93:64</mac-address>
    <interface>vtnet0</interface>
    <expires>290</expires>
    <type>ethernet</type>
  </arp-cache>
  <arp-cache>
    <hostname>?</hostname>
    <ip-address>192.168.64.49</ip-address>
    <mac-address> a0:36:9f:42:60:fe</mac-address>
    <interface>vtnet0</interface>
    <permanent>true</permanent>
    <published>true</published>
    <type>ethernet</type>
  </arp-cache>
</arp>

Have you noticed that version at the top? We use that to detect any changes to the schema in the
output. One example of this is the jls command, which is currently at version 2. Tracking the version
of the output application allows the tool parsing the output to change how it parses the output based on the
version—or to warn the admin that it might not understand the output of newer versions than it is aware of.

# jls —libxo=json,pretty
{
  "__version": "2", 
  "jail-information": {
    "jail": [
    ]
  }
}

How to Parse JSON outputs using jq in
FreeBSD

The command-line JSON processor jq is another amazing piece of software. It’s available on
FreeBSD as a package.

It can do a range of transformations, from basic filtering, to advanced map/reduce. For many people, it is a
tool that gets used daily, alongside grep and other common unix text processing tools.

The Unix inventors realized this important aspect of information security early on and integrated it into the
filesystem layer. Since everything is a file in Unix, protecting that file is one part of the
filesystem. Note that other protection mechanisms exist in operating systems like FreeBSD. We will focus
on the protections that ZFS adds on top of those.

The first filter is the simplest:, `.` takes in an input and produces a structurally unchanged, but
beautified and colorized output:

# arp -na --libxo json | jq .
{
  "__version": "1",
  "arp": {
    "arp-cache": [
      {
        "hostname": "?",
        "ip-address": "192.168.64.13",
        "mac-address": "52:88:80:9b:bb:00",
        "interface": "vtnet0",
        "permanent": true,
        "type": "ethernet"
      },
      {
        "hostname": "?",
        "ip-address": "192.168.64.1",
        "mac-address": "d6:57:63:1f:93:64",
        "interface": "vtnet0",
        "expires": 693,
        "type": "ethernet"
      }
    ]
  }
}

Next, we can use the object identifier, which takes in a key, .foo. It can be nested, like .foo.bar

If we apply that to the output of arp, we can select just a subset of the output, the cache
contents:

# arp -na --libxo json | jq '.arp."arp-cache"'
[
  {
    "hostname": "?",
    "ip-address": "192.168.64.13",
    "mac-address": "52:88:80:9b:bb:00",
    "interface": "vtnet0",
    "permanent": true,
    "type": "ethernet"
  },
  {
    "hostname": "?",
    "ip-address": "192.168.64.1",
    "mac-address": "d6:57:63:1f:93:64",
    "interface": "vtnet0",
    "expires": 485,
    "type": "ethernet"
  }
]

Have you noticed that we enclosed `arp-cache` in quotes? That’s because if we didn’t, `jq` would parse it as
a subtraction.

We can also get an array index using [index]. In this example, we only return the first matching entry
(numbered zero, because that’s how computers count):

# arp -na --libxo json | jq '.arp."arp-cache"[0]'
{
  "hostname": "?",
  "ip-address": "192.168.64.13",
  "mac-address": "52:88:80:9b:bb:00",
  "interface": "vtnet0",
  "permanent": true,
  "type": "ethernet"
}

We can also iterate over the array using []. In this output, we see each of the objects (arc cache entries)
returned individually, rather than as a JSON array.

# arp -na --libxo json | jq '.arp."arp-cache"[]'
{
  "hostname": "?",
  "ip-address": "192.168.64.13",
  "mac-address": "52:88:80:9b:bb:00",
  "interface": "vtnet0",
  "permanent": true,
  "type": "ethernet"
}
{
  "hostname": "?",
  "ip-address": "192.168.64.1",
  "mac-address": "d6:57:63:1f:93:64",
  "interface": "vtnet0",
  "expires": 323,
  "type": "ethernet"
}

And lastly, we can use a pipe character, just like the Unix shell does! The pipe will need to be within
single quotes, or escaped, to keep the shell from interpreting it before it gets to jq:.

# arp -na --libxo json | jq '.arp."arp-cache"[] | ."ip-address"'
"192.168.64.13"
"192.168.64.1"

In the command above, jq filtered arp’s output and extracted the arp-cache array, iterated over the objects
and passed them to another filter, `.”ip-address”`, giving us just the list of IPs.

We can also create new arrays and objects by wrapping them as needed: Here we create an array of new objects,
each containing an IP address and MAC address pair, discarded all of the other information.

# arp -na --libxo json | jq '[.arp."arp-cache"[] | {ipaddr: ."ip-address", macaddr: ."mac-address"}]'
[
  {
    "ipaddr": "192.168.64.13",
    "macaddr": "52:88:80:9b:bb:00"
  },
  {
    "ipaddr": "192.168.64.1",
    "macaddr": "d6:57:63:1f:93:64"
  }
]

Or:

# arp -na --libxo json | jq ' [.arp."arp-cache"[] | {(."ip-address"): ."mac-address"} ]’
[
  {
    "192.168.64.13": "52:88:80:9b:bb:00"
  },
  {
    "192.168.64.1": " d6:57:63:1f:93:64"
  }
]

Using xo in Shell Scripts!

If you, like us, have fallen in love with `jq` and `xo`, then it’s time to use them in shell scripts too! We
start by specifying our text output:

# xo 'Greetings {:name}, the time is {:date}\n' "$(whoami)" "$(date +%H:%M:%S)"
Greetings root, the time is 21:39:51

We can specify the style that we need using --style json:

# xo --style json 'Greetings {:name}, the time is {:date}\n' "$(whoami)" "$(date +%H:%M:%S)"
"name":"root","date":"21:41:13" 

We can also wrap the output in a path:

# xo --wrap hello/user --style json --pretty 'Greetings {:name}, the time is {:date}\n' "$(whoami)" "$(date +%H:%M:%S)”
"hello": {
  "user": {
    "name": "root",
    "date": "21:42:20"
  }
}

In a scripted environment, we can also use the --open and --close flags:

#!/bin/sh

xo --pretty --style json --top-wrap --open hello/user '{:name} {:date}' "$(whoami)" "$(date +%H:%M:%S)"
xo --pretty --style json --top-wrap --close hello/user


# sh hello.sh 
{
  "hello": {
    "user": {
      "name": "root",
      "date": "22:30:06"
    }
  }
}

And if we want the XML version, all we need to do is to change the flag:

#!/bin/sh

xo --pretty --style xml --top-wrap --open hello/user '{:name} {:date}' "$(whoami)" "$(date +%H:%M:%S)"
xo --pretty --style xml --top-wrap --close hello/user

# sh hello.sh 
<hello>
  <user>
    <name>root</name>
    <date>22:41:34</date>
  </user>
</hello>

And just like that, we can get structured, machine-readable output perfectly suited for automated tools.

Conclusion

FreeBSD’s libxo integration is one of many advantages it currently enjoys over Linux. This
integration allows simple, reliable automation based on the output of many system tools without the need for
complex and unreliable “output scraping.”

Using the library or scripting tool, you can extend your own applications to interoperate as well, without
having to undertake large redesign projects. Using this output can improve automation tools like ansible,
provide better output for APIs, and drive GUIs for appliances and applications.