Modern inetd in FreeBSD

The inetd ‘super-server’ is a special application which ties incoming network connections to locally-run commands.  Using a single `super-server` to handle all network requests conserves memory and CPU resources at the expense of increased application latency.  Although inetd has largely fallen out of fashion today, it was the most common method for handling network requests in the early days of the Internet.

When a peer connects to an inetd-managed port, inetd runs a command in a sub process to handle the incoming request. The sub process is given a socket file descriptor as standard input, standard output, and standard error. Once the sub process is finished—for example, after printing a requested web page to its STDOUT—it exits, returning control to inetd.

inetd is not a common part of deployments today, but it still has potential to be useful in production environments.

inetd Usage

inetd needs to be started by rc and enabled in rc.conf. It has its own configuration file /etc/inetd.conf which contains a lot of example configurations.

You can start inetd by adding an enable entry to rc.conf with sysrc and starting it with service:

# sysrc inet_enable=yes
# service inetd start

Inetd contains built-in versions of some services that don’t see much use anymore, but were common in the early days of the Internet. The inetd.conf configuration file has example entries for a number of services, including the internal services it implements. The four simple servers built into inetd on FreeBSD are:

• echo: Echo Protocol RFC 862 1983
  An echo server is just what it sounds like—it returns an identical copy of any traffic it receives. Echo may be operated on either TCP or UDP. Please note that echo over UDP does not authenticate the path, and can be used to perform UDP amplification attacks. Care should be taken that echo on UDP is not made available on the public internet!
• discard: Discard Protocol RFC 863 1983
  Discard is the original /dev/null as a service—all traffic sent to the server is simply discarded.
• chargen: Character Generator Protocol RFC 864 1983
  For TCP connections, the chargen server sends random characters to any peer that connects. For UDP, it sends a datagram containing random characters for any datagram it receives. Like echo, chargen over UDP can be abused to perform amplification attacks, and should never be exposed to the public internet.

• daytime: Daytime Protocol RFC 867 1983
  The daytime server returns an ASCII character string of the current date and time in an unspecified format.

• time: Time Protocol RFC 868 1983
  The time server returns to the client a 4-byte value representing the number of seconds since 00:00 (midnight) 1 January, 1900 GMT. This time will roll over in 2038.

These internal services are good to test your configuration to verify that inetd is working, before using inetd to act as a socket activator for other traffic.

Daytime returns a pleasant response (unlike the noise that chargen does) when we connect to it. For testing let’s enable the daytime internal services in /etc/inetd.conf:

...
# "Small servers" -- used to be standard on, but we're more conservative
# about things due to Internet security concerns.  Only turn on what you
# need.
#
daytime stream  tcp     nowait  root    internal
daytime stream  tcp6    nowait  root    internal
daytime dgram   udp     wait    root    internal
daytime dgram   udp6    wait    root    internal
#time    stream  tcp     nowait  root    internal
#time    stream  tcp6    nowait  root    internal
#time     dgram  udp     wait    root    internal
#time     dgram  udp6    wait    root    internal
#echo    stream  tcp     nowait  root    internal
#echo    stream  tcp6    nowait  root    internal
#echo    dgram   udp     wait    root    internal
#echo    dgram   udp6    wait    root    internal
#discard stream  tcp     nowait  root    internal
#discard stream  tcp6    nowait  root    internal
#discard dgram   udp     wait    root    internal
#discard dgram   udp6    wait    root    internal
#chargen stream  tcp     nowait  root    internal
#chargen stream  tcp6    nowait  root    internal
#chargen dgram   udp     wait    root    internal
#chargen dgram   udp6    wait    root    internal
...

A host’s services are described in the /etc/services file, there we can see that daytime for tcp, udp and sctp runs on port 13.

$ cat /etc/services | grep daytime
daytime          13/tcp
daytime          13/udp
daytime          13/sctp

inetd reconfigures in response to SIGHUP and the service reload command. After changing the configuration file and reloading, we will have inetd listening on port 13 and acting as a daytime server:

# service inetd reload
# sockstat  | grep inetd
root     inetd      1363  7  tcp4   *:13                  *:*
root     inetd      1363  8  tcp6   *:13                  *:*
root     inetd      1363  9  udp4   *:13                  *:*
root     inetd      1363  10 udp6   *:13                  *:*

We can test the services using netcat:

$ nc freebsd.example.com 13
Sun Dec 12 20:27:25 2021
^C

Using External Services

To inetd, external services are any other process that it might launch. inetd.conf also contains examples of how to launch common services including ftp, sshd, telnet and fingerd.

# Internet server configuration database                                  
#                                                                         
# Define *both* IPv4 and IPv6 entries for dual-stack support.             
# To disable a service, comment it out by prefixing the line with '#'.    
# To enable a service, remove the '#' at the beginning of the line.       
#                                                                         
#ftp    stream  tcp     nowait  root    /usr/libexec/ftpd       ftpd -l   
#ftp    stream  tcp6    nowait  root    /usr/libexec/ftpd       ftpd -l   
#ssh    stream  tcp     nowait  root    /usr/sbin/sshd          sshd -i -4
#ssh    stream  tcp6    nowait  root    /usr/sbin/sshd          sshd -i -6
#telnet stream  tcp     nowait  root    /usr/libexec/telnetd    telnetd   
#telnet stream  tcp6    nowait  root    /usr/libexec/telnetd    telnetd   

In the above examples, sshd needs to be given the -i flag to signify that it is being launched from inetd. The telnetd service, on the other hand, does not:

The telnetd command is a server which supports the DARPA standard TELNET
virtual terminal protocol.  Telnetd is normally invoked by the internet
server (see inetd(8)) for requests to connect to the TELNET port as
indicated by the /etc/services file (see services(5)). 

As we can see from its man page above, telnetd is intended to be launched from inetd and is not expected to be run standalone. Although telnetd is strongly deprecated, some commonly used software—such as tftpd—is still deployed this way today.

The power of inetd  lies in the simplicity of its interface. To demonstrate, let’s create a simple inetd-managed service. Since inetd handles the network connections, we can write our own daytime service shell script as simply as this:

$ cat daytime.sh 
#!/bin/sh

date
         

When our example script runs the date command, it prints the date to standard output. When invoked from inetd, our script’s STDOUT is passed to inetd’s own socket—and inetd then passes our shell script’s output to the network peer which requested it. This simple mechanism can also be used to read input from a host.

Each line in the inetd configuration file describes the service to be provided (daytime), the type of socket to create (stream) and protocol (tcp), wait or nowait (explained below), the user to run the local command as (user), the full path of the local command to run (/home/user/daytime.sh) and an example command line including ARGV[0] (daytime.sh). With this configuration, we can replace the internal daytime service with our shell script:

daytime stream tcp     nowait  user      /home/user/daytime.sh     daytime.sh
We can reload and test:
# service inetd reload

$ nc freebsd.example.com 13
Sun Dec 12 20:48:05 UTC 2021

This isn’t generally the best way to implement high performance, heavily trafficked services—but it can be a handy way to quickly expose simple commands to the network. For example, one might use inetd to provide read-only sysctl variables to a logging tool.

As with any network service, you should still be careful not to expose information—or the ability to run commands—to users who shouldn’t be able to access them. This is especially true for any commands exposed to the entire Internet!waitvs nowait

In the example configuration, each of the services is listed with either wait or nowait. The tiny internal servers, the TCP services are marked nowait, but the UDP ones are marked wait.

This indicator in the configuration file tells inetd what to do when it invokes the process to handle the connection. With nowait inetd is being told that the invoked process is going to fork and handle the connection and inetd can pass that responsibility to the invoked process.

The wait keyword tells inetd that it must wait until the invoked process terminates before handling new service requests.

Stream (think TCP) services that use the nowait key word are expected to handle single client requests forwarded from inetd. This is the expected mode of operation. wait stream services act as full servers and are instead given the listening socket and are expected to accept at least one connection from the socket before exiting.

For datagram services there is a dance required to handle connections for services and it is more common to use wait to handle those services. Datagram servers that use nowait should read at least one packet from the socket and connect(2) to the peer to create a binding in the kernel so they get packets correctly routed to them.

Further Configuration

inetd has evolved a lot since its first release in 4.3BSD. You can get a rough idea of how much functionality it grew by comparing the 4.3BSD man page to the FreeBSD 14.0-CURRENT man page.

inetd supports the ability to use the (now outdated) tcp mux protocol which enables multiplexing of services on a single port. This is similar to a feature in nginx that allows you to run non SSL services and multiplex them on a single application.

An interesting historical point in inetd’s life was the introduction of the tcp wrappers API. TCP wrappers were invented during one of the early hacker hunts, the full story of the hunt for this hacker is a great read, but shortly: The systems under attack were consistently being rooted and the attacker was running rm -rf destroying everything on the disk and making recovering logs very difficult. With minimal development a shim was put between the services on the host and the application, the shim logged each connection to the host.

The logging shim grew the ability to perform source access control for connections and was extended to perform a reverse finger (a look up of host/user on a remote machine). While it didn’t help them catch the hacker (they stopped when they figured out they were being hunted), it did help them figure out where they were coming from and track the incidents over a number of networks spread across the planet.

Since that incident tcp_wrappers grew to be more of an access control feature and are reminiscent of http application firewalls. An interesting idea for their use was to remove kernel involvement in responding to ICMP ECHO requests(i.e. those generated by the ping command).

For production uses today, it is worth exploring the rate limiting features that inetd has to offer. inetd can be configured to rate limit with four different controls:

-c maximum
    Specify the default maximum number of simultaneous invocations of
    each service; the default is unlimited.  May be overridden on a
    per-service basis with the "max-child" parameter.
-C rate
    Specify the default maximum number of times a service can be
    invoked from a single IP address in one minute; the default is
    unlimited.  May be overridden on a per-service basis with the
    "max-connections-per-ip-per-minute" parameter.
-R rate
    Specify the maximum number of times a service can be invoked in
    one minute; the default is 256.  A rate of 0 allows an unlimited
    number of invocations.
-s maximum
    Specify the default maximum number of simultaneous invocations of
    each service from a single IP address; the default is unlimited.
    May be overridden on a per-service basis with the "max-child-per-
    ip" parameter.

Conclusions

Although inetd may seem quaint in 2022, its core concepts live on in Linux’s systemd and macOS’s launchd.  In particular, limiting the number of idle processes with inetd or a similar superserver is useful in tiny images intended for containers and tiny embedded devices.

inetd also offers the ability to inject logic in the handling of connections to less often used services. There are accounts of inetd being used to create jails, enabling single request execution environments. While inetd might feel old, there are clear parallels in more modern software and it is an easily extendable building block.