Discover the awesome powers of Dtrace and take a deep dive into FreeBSD observability
DTrace is an observability framework that enables dynamic tracing of programs, and the FreeBSD Kernel. DTrace has been ported to many different Operating Systems, including NetBSD, Mac OS, and Windows. DTrace was originally developed for Sun’s Solaris operating system with DTrace seeing its first release in 2005. DTrace landed in FreeBSD 7.1 in 2009, this work being documented by a BSDCan Talk that explains its usage and some of its internals. DTrace offers an incredible view into the operation of programs, and is an excellent tool for debugging and performing analysis of complex software.
DTrace was the first dynamic observability framework to see large scale deployment. If you have done performance analysis on Linux in recent years, you might be familiar with extended BPF (or just BPF now). BPF offers a similar set of tools and has similar interfaces to its command line tools, but the Linux kernel developers were able to learn from a decade of production DTrace use, and BPF offers even more power and more flexibility than DTrace.
The DTrace framework is a combination of probe points compiled into software, a C like programming language called D and a large collection of scripts built on top of DTrace programs. The main interface to DTrace that we will interact with is the dtrace command line tool.
An introduction to the D language is out of scope for this article, interested readers should look at the Illumos DTrace Guide or the 2011 DTrace Book by Gregg and Mauro for background on writing D programs and how to use its powerful features.
DTrace Probe points are specially marked sections of a compiled program that allow a smart tool (in our case dtrace) to hook in and inject additional code to run when the probe is activated, but also be able to remove the code when the probe is deactivated. Probe points are marked by the compiler, and their locations are stored in metadata that the DTrace tools access. On FreeBSD, this information is stored in Compact C Type Format (CTF) and included in ELF objects.
When DTrace probes are not enabled they have effectively 0 impact on the operation of software. DTrace has been used in production for 16 years and it has been demonstrated to be a solid tool for introspecting running systems. DTrace technologies enable tools that allow systems moving Gigabit/s of traffic to be debuggable with only modest impact on CPU usage.
When the kernel is being built, probe points are created for every function entry and exit, and for specific points in the kernel marked out by developers. The entry and exit probe points are called Function Boundary Traces or FBT (entry and exit being the edges of function calls), and static probe points are called Statically Defined Tracing or SDT. A third type of probe you might see is used for static trace points in userspace programs, and these are called USDT or User Statically Defined Tracing.
FBT probes allow us to instrument any function that ends up in the final program without having to make any special effort to declare places we might be interested. Not all source code functions end up in the final program; inlined functions do not have FBT probes.
SDT probes are added by developers and offer an API for inspecting the kernel at specific points. In this article, we are going to look at statically defined probe points for the IP, TCP, and UDP network protocols.
Statically Define Probes
SDT probes allow kernel developers to offer an external API to consumers using DTrace. SDT probes have the benefit of being specifically located in key parts of the kernel and enable the tracing of code in locations other than function entry and function return.
Did you know?
Improving your FreeBSD infrastructure has never been easier. Our teams are ready to consult with you on any FreeBSD topics ranging from development to regular support.
The downside to SDT probes is that they have to be implemented to be available, and they offer a static interface. In production releases you are stuck with the probes that kernel developers placed for you to use. Probes are limited to the arguments selected for the interface and you aren’t able to add access to other structures that you might want in the future.
Adding statically probes is straight forward once their subsystem has been added. This example from the network stack shows the receive probes for UDP and UDP-Lite:
UDPLITE and UDP SDT probes from sys/netinet/udp_usrreq.c:
if (proto == IPPROTO_UDPLITE)
UDPLITE_PROBE(receive, NULL, last, ip, last, uh);
UDP_PROBE(receive, NULL, last, ip, last, uh);
if (udp_append(last, ip, m, iphlen, udp_in) == 0)
There can be multiple points in the kernel that will call the same probe. This means that SDT probes are good for representing state transitions of logical points of execution rather than specific pieces of code.
Access to DTrace probes is available through a 4-level hierarchy. Probes are stored in libraries called providers, and each library can have multiple sub modules – there is an optional function namespace (used for FBT probes) and finally each probe has a name. On FBT probes the function is the function from the original source code, and the probe names represent the entry and return boundary points.
DTrace probes are written in the form provider:module:function:name. DTrace will do its best to match missed sections, and allows the use of wild cards.
We can list all of the probes on our system with dtrace -l:
This large number of probes represents all of the traced function calls that have ended up in the kernel. We can list the probes for a provider with the -P flag:
dtrace -l -P fbt | wc -l
This command shows that the vast majority of the available probes come from the fbt provider. DTrace on FreeBSD offers a large number of providers that (other than FBT) represent areas where developers think probe points would be interesting:
Some DTrace providers have manual pages that offer in-depth information on how to use the provider and its probes. For network probes, dtrace_tcp(4), dtrace_ip(4), and dtrace_udp(4) offer information about the arguments passed to the probes and explain what is reachable from DTrace.
In this article, we are interested in looking at network protocols with DTrace; the SDT probes for TCP are much richer than the static probes for other protocols:
The downside is that when using FDT probes you are at the whims of the compiler to generate the probe points, in the future those may change, therefore FBT offers an unstable API. We also have to have a good understanding of what the kernel implementation is trying to do to understand and best use the FBT probe points.
DTrace offers send and receive probes in the ip provider. With the data passed as arguments to the probes, we can use these to extract a lot of interesting information about the packets the system is sending and receiving.
Both of the probes in the ip provider have the same 6 arguments:
The arguments, pktinfo_t and csinfo_t, are unimplemented and exist to allow FreeBSD DTrace probes to be compatible with other systems.
ipinfo_t contains IP fields common to both IPv4 and IPv6 packets, and the ifinfo_t field contains information about the outgoing and incoming interfaces for packets.
ipv4info_t and ipv6info_t contain the IPv4 and IPv6 header fields for packets; the corresponding field is NULL for each address family (i.e. for v6 packets ipv4info_t is empty).
We can use the information from the man page and some DTrace magic to look at the IP traffic we are receiving. This one-liner shows who is sending to you and counts how many packets are received for each sender:
dtrace_ip(4) documents the fields which are exposed by the probes in the ip provider. In the above example we are selecting the third argument which is the ipinfo_t struct and using the ip_saddr member to aggregate senders.
Below, we have an example which counts the distribution of IP payload size by destination address on send:
The DTrace tcp provider offers 14 probes that we can look at, 8 are documented in the man page and the others are debug probes. There are more probe points exposed for TCP because it is a connection oriented, reliable protocol which leads to there being more sources for errors and stalls in connections.
Each of the tcp provider probes has 5 arguments, for all apart from state-changed they are: pktinfo_t, csinfo_t, ipinfo_t,tcpsinfo_t and tcpinfo_t. pktinfo_t and csinfo_t are unimplemented and made available to allow FreeBSD’s tcp provider to be more compatible with other DTrace implementations. The state-changed probe has a slightly different set of arguments and these are documented in the man page.
The ipinfo_t argument is an address family agnostic representation of the fields from the IP header and is described in the dtrace_ip(4) man page.
tcpsinfo_t is a stable representation of the TCP connection state, and the tcpinfo_t argument exposes the fields from the TCP header on the received segment in host order.
With this information, we can inspect the TCP connections to the machine. The following one-liner shows new successful connections by remote address and local port number (who is connecting to you and what):
This one-line shows that there is traffic to port 22, likely ssh scans, and to port 4000, where I am running a development web server.
TCP is governed by a state machine and the probes that fire during a sampling period can give you a good idea of what is happening on the machine. The following examples traces the TCP Probes that fire and gives a count by name:
Again, pktinfo_t and csinfo_t are offered to provide compatibility with other systems. The ipinfo_t argument offers access to the packets IP header information.
udpsinfo_t argument contains the state of the UDP connection associated with the packet and provides a point to the struct inpcb for any associated socket. udpinfo_t provides access to the raw UDP header from the datagram.
The following one-liner will count the number of UDP datagrams for each destination port from the host:
There is a lot of information available to help you understand how to use DTrace. The 2011 DTrace book by Brendan Gregg and Jim Mauro is a good resource for looking at DTrace, how the D language works and provides a ton of examples of usage. The Illumos dtrace documentation is quite up to date and offers a good intro to the D language, with examples.
awesome-dtrace.com hosts a great collection of DTrace resources and is worth a look for examples and scripts. Finally, the 2021 Systems Performance book by Gregg is a great introduction to interrogating operating systems and applications, while it is Linux and BPF heavy a lot of the information transfers well to FreeBSD and is a great resource for structuring further explorations into performance analysis.
DTrace is a very powerful tool for observing what the operating system is up to. Using the full power of FDT probes requires familiar knowledge with how things are implemented. SDT probes and specific providers like ip, tcp and udpoffer insights into the internal behavior of the kernel without requiring a high level of knowledge of the underlying code.
Meet the author: Tom Jones
Tom Jones is an Internet Researcher and FreeBSD developer that works on improving the core protocols that drive the Internet. He is a contributor to open standards in the IETF and is enthusiastic about using FreeBSD as a platform to experiment with new networking ideas as they progress towards standardisation.
When troubleshooting a Linux or FreeBSD system, you need to be able to probe the system to find answers as to why it is behaving in a particular way. In this article, we’ll provide an overview of some of the basic tools and introduce the FreeBSD equivalents of common Linux tracing and troubleshooting tools.
The popularity of package managers permeates all Unix distributions. Yet there are subtle differences in the approach that Linux vs. FreeBSD take in handling packages. How does Linux compare to FreeBSD’s way of managing packages? We have identified key points to consider in the software lifecycle management of both in the article below.
The question isn’t as much “Should I choose FreeBSD or Linux”, the question should be “Which OS fits my needs best?”. In our most recent comparison article, we go over the two implementations of the networking stack and look at how Linux implements networking and how FreeBSD fares in most cases.
Pingback: Valuable News – 2021/06/14 | 𝚟𝚎𝚛𝚖𝚊𝚍𝚎𝚗