Interacting with FreeBSD – Learning the Fundamentals of the FreeBSD Shell

In FreeBSD 14, the default root shell is changing. In this article, we will discuss the background and motivations for this change along with its implications and advantages.

Starting at the Beginning: The FreeBSD Shell

A shell is software that allows people to interact with their operating system. Before graphical displays with windows and mice existed, early shells were created where computer operators would type simple commands at the keyboard to launch applications.

Although some see the shell as “primitive,” this manner of interacting with a computer is extremely flexible and expressive. The command-line shell thus lives on as a quintessential element that defines Unix systems. There are many shells to choose from, but they have many things in common—these commonalities reflect features that go back to the early design of Unix and the fact that the shell has always been deeply embedded.

There are two shells included in the FreeBSD base system – sh and csh – corresponding to evolutions of the Bourne and C shells. Early on, the consensus was generally that the Bourne shell was the better shell for scripts and programming while the C shell was a better interactive shell. These strengths also influenced subsequent developments. Many innovations for interactive shell use were added first in tcsh, a fork of csh that can be considered its natural successor. Similarly, the Korn shell is built on the foundations of the Bourne shell, primarily adding features more commonly associated with programming languages.

Today, Bourne-style (sh) syntax has come to the fore. It is the one syntax covered by the POSIX standard. The interactive innovations of tcsh were adopted and further embellished by shells that used sh syntax such as Z shell and bash. Scripts included as part of a FreeBSD system for things like starting and stopping services all use sh. But until now, the C shell has remained the default shell for the root user on FreeBSD.

Using the same shell for both scripts and interactive use has the advantage that knowledge is transferable. So, changing the default root shell reduces how much a new user needs to learn. It also makes FreeBSD feel more familiar to new users coming from other Unix operating systems. For a long-time FreeBSD user who is unhappy with the change, it is easy to revert root's shell to csh.

Differences between Bourne and C Shells

For most common commands, which shell you launch them from makes no difference. The basic syntax for commands, arguments and variables does not vary.

The first difference you may notice between csh and sh is that where csh typically uses a % character to prompt you for input, sh uses $ by default.

Although the unprivileged user’s prompt is different, Bourne and C shells alike use a # prompt as a subtle indicator alerting you to take extra care when logged in as root. Once you dig below the surface, more differences emerge. Let’s highlight a few of the more notable differences.

Variables

When you want the shell to remember something so that it can be reused later, it can be assigned to a variable. This can be useful where you need to refer to a long file path more than once. For example, comparing assignments in the two shells, we have:

csh% set logs=/usr/jail/mail/var/log

and

sh$ logs=/usr/jail/mail/var/log

csh makes a clearer separation between shell variables such as this one and environment variables. Environment variables are passed on to other commands that are run while normal variables are private to the shell. Taking advantage of this, environment variables are often used for configuring external programs.

In sh, assigning an environment variable requires two steps—the first step assigns the variable, and the second exports it. Newer sh implementations allow this to be combined in a single export command, but the conceptual difference remains that exporting a variable involves setting a special flag for that variable. Looking at an example, the differences are only cosmetic.

Here we set the PAGER environment variable, which is configures your preferred tool for viewing files one page at a time:

csh% setenv PAGER less

and

sh$ export PAGER=less

When you want to use a variable, the forms $VAR and ${VAR} are common to both shells. The braces are useful where you want to follow a variable name with characters that would be valid as part of a variable name. Conveniently, a slash is not valid in a variable name so when using the logs variable we assigned earlier, we can type e.g. $logs/messages.

Bourne and C shells both expand variables inside double quotes but not inside single quotes—so if the variable VAR equals 3, “$VAR” is expanded to “3” but ‘$VAR’ is interpreted literally as ‘$VAR.’ Double quotes can be useful when you need to preserve literal space characters. Normally any space character starts a new argument to the command, even if it comes from a variable expansion.

Support for arrays is one convenient feature that csh supports but sh does not. In csh, it is common to assign to the path array to configure the list of directories that are searched for commands. For example:

csh% set path = ( /bin /usr/local/bin ~/bin )

path is a special case because it maps onto the PATH environment variable, which is a colon-separated list. The use of uppercase names for environment variables is only a convention. In sh, we have to operate on the environment variable directly:

sh$ PATH="/bin:/usr/local/bin:$HOME/bin"

Note that we don't need to use export here. PATH should already be marked for export.

Some more advanced sh variants do support arrays, along with a range of other variable flags. FreeBSD’s sh does not include this advanced functionality, but it does allow you variables to be declared readonly.  That property is private to the shell, however: if you declare an exported variable readonly, it will not be passed to other programs in a read-only form.

Understanding Redirection under the FreeBSD Shell

Redirection allows you to save the output from a command to a file or pass it on as the input for another command. The basic forms of redirection with | for pipes and > and < for input and output of files predate both shells and work much the same in either.

These operators only deal with redirecting the two main file descriptors – standard input and standard output. There is also a separate output file descriptor dedicated to error messages. This can be very helpful because you normally want to see the errors on the screen, not pass them on to another program or write them to a file. Csh has an extension that allows you to also redirect errors using a very concise syntax, for example:

csh% cmd >& file

By contrast, sh allows arbitrary reassignments of file descriptors. This flexibility is useful in scripts but can require a little thought to compose.  The equivalent of the above is:

sh$ cmd > file 2>&1

Each redirection is applied in turn going from left to right. Here, we want both the normal and error output to go to the file. In this case, standard output is first redirected to the file. Next file descriptor 2, which corresponds to standard error, is redirected to wherever file descriptor 1 points. The normal output is file descriptor 1 and we could have chosen to write the first redirection as 1> file.

Another example which is not uncommon is 3>&1 >&2 2>&3. This swaps the two file descriptors by first opening an additional third one as a copy of the first, then redirecting the first and finally using the temporary third descriptor to set standard error.

csh also has another redirection form with >! which checks if the target file already exists before it gets overwritten. This feature has made it into various sh variants—including FreeBSD's—but with the form >|.

Control Flow

Both shells provide ways to perform commands conditionally or iteratively.  The differences are more fundamental than they first appear on the surface.  Let's consider a simple example:

csh% if ( -f ~/.aliases ) then
if?    source ~/.aliases
if?  endif

sh$ if [ -f ~/.aliases ]; then
>     . ~/.aliases
>   fi

Perhaps the ugliest element of Bourne syntax is the need for that semi-colon before the then. But where csh has a special expression syntax, sh simply allows any command.

Given that commands are the basic element from which nearly everything in a shell script is composed, there's a certain simplicity and purity to the sh approach.

In the Bourne shell example, we are running a command with the unusual name of [.  There is no special parsing going on for its arguments, so it works in much the same manner as any other command. The closing ] bracket is just one of its arguments, which it happens to ignore. The shell is thus reliant on the semi-colon to know where arguments to the [ command end. A new-line would also serve that purpose, and it is common to see one on the following line.

The condition can be any arbitrarily complex command, or even a list of commands. The exit status of the final command determines the outcome. This means you can do things like the following:

sh$ if grep word file >/dev/null; then echo found; fi

In the above example, grep exits 1 if a match was found, or 0 if it was not—so rather than attempt to parse grep’s output, we redirect it to /dev/null and operate on grep’s exit code directly.

Since sh treats whole conditions and loops just like any other command, they can be included as steps in pipelines.

Aliases and Functions

An alias provides a simple shortcut where a single word is expanded to a complete command line. There is no need for a special character like the $ that precedes a variable name—but aliases only work when typed as the first word on the command line.

Most csh aliases can be trivially converted to sh by changing the definition to use an equals sign. For example:

csh% alias ll  ls -lFb

becomes

sh$ alias ll='ls -lFb'

When using aliases, any extra arguments you provide are added to the end of the command—so typing ll /tmp will expand to ll /tmp -lFb, and produce a long listing of files in /tmp.

csh also allows arguments to be inserted in an alias expansion via history references. These consist of abstruse sequences of characters starting with an exclamation mark. For anything more complicated than a simple command, it is best to use functions in sh.

Unlike aliases, functions can take parameters of their own. As a simple example, we can define a setenv function in sh that works a bit like csh's version of setenv:

sh$ setenv() {
  export "$1=$2"
}

In this example, $1 expands to the first parameter and $2 to the second. Also common is $@ which expands to the full list of parameters. Compared to the named parameters of most programming languages, this is rudimentary—but it is simple, and it’s much clearer than csh history references.

You may also see an alternate function syntax with the keyword function. Two forms exist because functions were added to the Korn and Bourne shells independently at around the same time. The syntax shown here is the one covered by the POSIX standard, and has the advantage of being shorter.

Interactive Enhancements

As preparation for making sh the default root shell on FreeBSD, it has seen many enhancements for interactive usage to bring it in line with other shells. If you type the beginning of a command or file name and press tab, the shell will now either complete the remaining characters or list possibilities. When exiting, the history of past commands is saved in a file determined by the HISTFILE variable. Support for incrementally searching the command history is also now available.

The command-line editing functionality is handled by a library named editline which is also used by other commands such as sftp and lldb. It is possible to configure what action occurs for particular keys or key combinations.

For example, you can configure the up arrow key to search for a history line that begins with the current contents of the command line, just as it does in csh.

This should go in the .shrc file if you only want it to apply to sh, or in .editrc if you want it to apply to all programs that use the editline library:

bind ^[[A ed-search-prev-history

Changing your FreeBSD Shell

On an Unix system, each user's preferred shell is maintained alongside details such as that user’s username, real name, home directory, password and group memberships. This is traditionally maintained in the file /etc/passwd.

For a system user like root, this file will still be applicable and you can edit the file manually if you wish. However, dedicated utilities for making such changes also exist.

FreeBSD supplies a pw utility for managing local users. So, if you don't want to wait for FreeBSD 14, you might change the root shell to /bin/sh with the following command:

pw usermod root -s /bin/sh

Users can change their own shell with the chsh command.  For example, the following selects zsh:

chsh -s /usr/local/bin/zsh

This may not work for non-local users if you use a central database with a protocol such as LDAP. When creating new users with the pw command, /bin/sh is already the shell that will be used by default. As described in the pw.conf(5) man page, this is configurable.

A common question that comes up in regard to changing the shell, and particularly the root user's shell, is whether doing so is likely to break anything. For the most part, it is quite safe to do so because it is only the interactive shell being changed, so the primary effect is on commands a user is typing. If you run a csh script from sh, the script will start with a line that looks something like #!/bin/csh -f which tells the system to run the script with csh. So existing shell scripts will continue to work.

Redirection is the most common  source of problems with automated tools subjected to an unexpected change in shells—for example, a script which uses 2>&1 to redirect all output to a logfile will break if executed in a csh environment, as will a script using >& if executed in sh. This issue tends to crop up when the same script may be called interactively, from cron, or over ssh—with what could be a different default shell in each scenario.

Another thing to be aware of if you change root's shell to one from ports is that you may complicate system recovery if an upgrade breaks the ports-installed shell. This can be where knowing how to manually change /etc/passwd can come in handy!

Another option is to make use of the toor user. On a FreeBSD system, toor is an additional username that shares a user ID of 0 with root. If you log in as toor, you have all the same permissions of root. All that is needed is to set the shell and a password for the toor account. Or, if you prefer, create some other account with a user ID of 0.

Sometimes the easiest approach is not to change the user’s login shell, but to simply run a different shell after login. You can simply use the default shell to type the name of your preferred shell. Or you can define aliases in your configuration files. For example, you might use the following shell alias:

alias toor='su -l root -c /bin/csh'

su on FreeBSD is somewhat fussy about the order of these options for that to work. Similar aliases can also work for remote machines via ssh. In that case, note that you may need the -t option to ssh to ensure a terminal device is allocated. Rather than a shell alias you might bind a desktop shortcut to run these commands. For example:

xterm -e ssh -t hostname exec /usr/local/bin/zsh -l

Terminal programs like xterm or tmux will look for the environment variable $SHELL before checking the user's shell so ensuring that gets set can be another way to arrange for a different shell to be invoked.

Additional Resources

Here are some interesting resources on the shell that you may also find useful:

• FreeBSD Handbook – section 3.9 – Shells
• FreeBSD Man Page – sh(1)
• FreeBSD Man Page – editrc(5)
• FreeBSD Man Page – tcsh(1)
• FreeBSD Man Page – pw.conf(5)
• Mailing list change proposal
• CSH Programming Considered Harmful
• Zsh