DTrace is an observability framework that enables dynamic tracing of programs, and the FreeBSD Kernel. DTrace has been ported to many different Operating Systems, including NetBSD, Mac OS, and Windows. DTrace was originally developed for Sun’s Solaris operating system with DTrace seeing its first release in 2005. DTrace landed in FreeBSD 7.1 in 2009, this work being documented by a BSDCan Talk that explains its usage and some of its internals. DTrace offers an incredible view into the operation of programs, and is an excellent tool for debugging and performing analysis of complex software.
DTrace was the first dynamic observability framework to see large scale deployment. If you have done performance analysis on Linux in recent years, you might be familiar with extended BPF (or just BPF now). BPF offers a similar set of tools and has similar interfaces to its command line tools, but the Linux kernel developers were able to learn from a decade of production DTrace use, and BPF offers even more power and more flexibility than DTrace.
The DTrace framework is a combination of probe points compiled into software, a C like programming language called D and a large collection of scripts built on top of DTrace programs. The main interface to DTrace that we will interact with is the dtrace command line tool.
An introduction to the D language is out of scope for this article, interested readers should look at the Illumos DTrace Guide or the 2011 DTrace Book by Gregg and Mauro for background on writing D programs and how to use its powerful features.
DTrace Probe points are specially marked sections of a compiled program that allow a smart tool (in our case dtrace) to hook in and inject additional code to run when the probe is activated, but also be able to remove the code when the probe is deactivated. Probe points are marked by the compiler, and their locations are stored in metadata that the DTrace tools access. On FreeBSD, this information is stored in Compact C Type Format (CTF) and included in ELF objects.
When DTrace probes are not enabled they have effectively 0 impact on the operation of software. DTrace has been used in production for 16 years and it has been demonstrated to be a solid tool for introspecting running systems. DTrace technologies enable tools that allow systems moving Gigabit/s of traffic to be debuggable with only modest impact on CPU usage.
When the kernel is being built, probe points are created for every function entry and exit, and for specific points in the kernel marked out by developers. The entry and exit probe points are called Function Boundary Traces or FBT (entry and exit being the edges of function calls), and static probe points are called Statically Defined Tracing or SDT. A third type of probe you might see is used for static trace points in userspace programs, and these are called USDT or User Statically Defined Tracing.
FBT probes allow us to instrument any function that ends up in the final program without having to make any special effort to declare places we might be interested. Not all source code functions end up in the final program; inlined functions do not have FBT probes.
SDT probes are added by developers and offer an API for inspecting the kernel at specific points. In this article, we are going to look at statically defined probe points for the IP, TCP, and UDP network protocols.
Statically Define Probes
SDT probes allow kernel developers to offer an external API to consumers using DTrace. SDT probes have the benefit of being specifically located in key parts of the kernel and enable the tracing of code in locations other than function entry and function return.
Did you know?
Improving your FreeBSD infrastructure has never been easier. Our teams are ready to consult with you on any FreeBSD topics ranging from development to regular support.
The downside to SDT probes is that they have to be implemented to be available, and they offer a static interface. In production releases you are stuck with the probes that kernel developers placed for you to use. Probes are limited to the arguments selected for the interface and you aren’t able to add access to other structures that you might want in the future.
Adding statically probes is straight forward once their subsystem has been added. This example from the network stack shows the receive probes for UDP and UDP-Lite:
UDPLITE and UDP SDT probes from sys/netinet/udp_usrreq.c:
...
if (proto == IPPROTO_UDPLITE)
UDPLITE_PROBE(receive, NULL, last, ip, last, uh);
else
UDP_PROBE(receive, NULL, last, ip, last, uh);
if (udp_append(last, ip, m, iphlen, udp_in) == 0)
INP_RUNLOCK(last);
return (IPPROTO_DONE);
...
There can be multiple points in the kernel that will call the same probe. This means that SDT probes are good for representing state transitions of logical points of execution rather than specific pieces of code.
Providers
Access to DTrace probes is available through a 4-level hierarchy. Probes are stored in libraries called providers, and each library can have multiple sub modules - there is an optional function namespace (used for FBT probes) and finally each probe has a name. On FBT probes the function is the function from the original source code, and the probe names represent the entry and return boundary points.
DTrace probes are written in the form provider:module:function:name. DTrace will do its best to match missed sections, and allows the use of wild cards.
We can list all of the probes on our system with dtrace -l:
# dtrace -l
ID PROVIDER MODULE FUNCTION NAME
1 dtrace BEGIN
2 dtrace END
3 dtrace ERROR
4 fbt kernel camstatusentrycomp entry
5 fbt kernel camstatusentrycomp return
6 fbt kernel cam_compat_handle_0x17 entry
7 fbt kernel cam_compat_handle_0x17 return
8 fbt kernel cam_compat_handle_0x18 entry
9 fbt kernel cam_compat_handle_0x18 return
...
75272 dtmalloc entropy malloc
75273 dtmalloc entropy free
75274 dtmalloc CAM_DEV malloc
75275 dtmalloc CAM_DEV free
75276 dtmalloc PUC malloc
75277 dtmalloc PUC free
75278 dtmalloc ppbusdev malloc
75279 dtmalloc ppbusdev free
75280 dtmalloc agtiapi_MemAlloc_malloc malloc
75281 dtmalloc agtiapi_MemAlloc_malloc free
On this system, there are 75282 probes
dtrace -l | wc -l
75282
This large number of probes represents all of the traced function calls that have ended up in the kernel. We can list the probes for a provider with the -P flag:
dtrace -l -P fbt | wc -l
71324
This command shows that the vast majority of the available probes come from the fbt provider. DTrace on FreeBSD offers a large number of providers that (other than FBT) represent areas where developers think probe points would be interesting:
# dtrace -l | awk '{print $2}' | uniq
PROVIDER
dtrace
fbt
random
xbb
sched
proc
lockstat
priv
proc
racct
proc
sched
proc
callout_execute
sched
io
lock
sched
sdt
vfs
vnet
ip
sctp
tcp
udp
udplite
sctp
opencrypto
mac_framework
mac
mac_framework
vm
vfs
sdt
profile
syscall
nfscl
dtmalloc
Some DTrace providers have manual pages that offer in-depth information on how to use the provider and its probes. For network probes, dtrace_tcp(4), dtrace_ip(4), and dtrace_udp(4) offer information about the arguments passed to the probes and explain what is reachable from DTrace.
In this article, we are interested in looking at network protocols with DTrace; the SDT probes for TCP are much richer than the static probes for other protocols:
# dtrace -l -P tcp -P ip -P udp
ID PROVIDER MODULE FUNCTION NAME
71450 tcp kernel none accept-established
71451 tcp kernel none accept-refused
71452 tcp kernel none connect-established
71453 tcp kernel none connect-refused
71454 tcp kernel none connect-request
71455 tcp kernel none receive
71456 tcp kernel none send
71457 tcp kernel none siftr
71458 tcp kernel none debug-input
71459 tcp kernel none debug-output
71460 tcp kernel none debug-user
71461 tcp kernel none debug-drop
71462 tcp kernel none state-change
71463 tcp kernel none receive-autoresize
71445 ip kernel none receive
71446 ip kernel none send
71464 udp kernel none receive
71465 udp kernel none send
The number of available static probes is much smaller than the probe points we can examine using FBT probes.
# sudo dtrace -l -P tcp | wc -l
15
There are 15 probe points in the TCP provider vs 950 FBT probe points:
# dtrace -l -P fbt -m kernel | grep tcp | wc -l
950
The downside is that when using FDT probes you are at the whims of the compiler to generate the probe points, in the future those may change, therefore FBT offers an unstable API. We also have to have a good understanding of what the kernel implementation is trying to do to understand and best use the FBT probe points.
IP
DTrace offers send and receive probes in the ip provider. With the data passed as arguments to the probes, we can use these to extract a lot of interesting information about the packets the system is sending and receiving.
Both of the probes in the ip provider have the same 6 arguments:
ip:::receive(pktinfo_t *, csinfo_t *, ipinfo_t *, ifinfo_t *,
ipv4info_t *, ipv6info_t *);
ip:::send(pktinfo_t *, csinfo_t *, ipinfo_t *, ifinfo_t *, ipv4info_t *,
ipv6info_t *);
The arguments, pktinfo_t and csinfo_t, are unimplemented and exist to allow FreeBSD DTrace probes to be compatible with other systems.
ipinfo_t contains IP fields common to both IPv4 and IPv6 packets, and the ifinfo_t field contains information about the outgoing and incoming interfaces for packets.
ipv4info_t and ipv6info_t contain the IPv4 and IPv6 header fields for packets; the corresponding field is NULL for each address family (i.e. for v6 packets ipv4info_t is empty).
We can use the information from the man page and some DTrace magic to look at the IP traffic we are receiving. This one-liner shows who is sending to you and counts how many packets are received for each sender:
# dtrace -n 'ip:::receive {@[args[2]->ip_saddr] = count();}'
dtrace: description 'ip:::receive ' matched 1 probe
^C
186.33.76.219 1
40.119.108.0 1
52.17.231.73 1
85.91.1.164 1
87.198.32.53 1
40.119.104.0 3
47.105.203.221 3
222.186.180.130 7
168.63.129.16 50
62.171.156.18 259
92.26.12.92 367
dtrace_ip(4) documents the fields which are exposed by the probes in the ip provider. In the above example we are selecting the third argument which is the ipinfo_t struct and using the ip_saddr member to aggregate senders.
Below, we have an example which counts the distribution of IP payload size by destination address on send:
# dtrace -n 'ip:::send { @[args[2]->ip_daddr] = quantize(args[2]->ip_plength);}'
dtrace: description 'ip:::send ' matched 1 probe
^C
167.248.133.25
value ------------- Distribution ------------- count
8 | 0
16 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
32 | 0
168.63.129.16
value ------------- Distribution ------------- count
16 | 0
32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 10
64 |@@@@@@@ 2
128 | 0
92.26.12.92
value ------------- Distribution ------------- count
32 | 0
64 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 92
128 |@@ 6
256 | 0
62.171.156.18
value ------------- Distribution ------------- count
16 | 0
32 |@@@@@@@@@@@@@@@@@@@@ 25
64 |@@@@@@@@@@@@ 15
128 | 0
256 | 0
512 |@@@@ 5
1024 |@@@@ 5
2048 | 0
TCP
The DTrace tcp provider offers 14 probes that we can look at, 8 are documented in the man page and the others are debug probes. There are more probe points exposed for TCP because it is a connection oriented, reliable protocol which leads to there being more sources for errors and stalls in connections.
Each of the tcp provider probes has 5 arguments, for all apart from state-changed they are: pktinfo_t, csinfo_t, ipinfo_t,tcpsinfo_t and tcpinfo_t. pktinfo_t and csinfo_t are unimplemented and made available to allow FreeBSD’s tcp provider to be more compatible with other DTrace implementations. The state-changed probe has a slightly different set of arguments and these are documented in the man page.
The ipinfo_t argument is an address family agnostic representation of the fields from the IP header and is described in the dtrace_ip(4) man page.
tcpsinfo_t is a stable representation of the TCP connection state, and the tcpinfo_t argument exposes the fields from the TCP header on the received segment in host order.
With this information, we can inspect the TCP connections to the machine. The following one-liner shows new successful connections by remote address and local port number (who is connecting to you and what):
# dtrace -n 'tcp:::accept-established { @[args[3]->tcps_raddr, args[3]->tcps_lport] = count();;}'
dtrace: description 'tcp:::accept-established ' matched 1 probe
^C
49.88.112.75 22 1
62.171.156.18 22 4
92.26.12.92 4000 4
This one-line shows that there is traffic to port 22, likely ssh scans, and to port 4000, where I am running a development web server.
TCP is governed by a state machine and the probes that fire during a sampling period can give you a good idea of what is happening on the machine. The following examples traces the TCP Probes that fire and gives a count by name:
# dtrace -n 'tcp::: { @[probename] = count();}'
dtrace: description 'tcp::: ' matched 14 probes
^C
accept-established 6
receive-autoresize 10
state-change 17
debug-user 323
debug-output 2485
send 2492
debug-input 3303
receive 3306
UDP
The DTrace udp provider offers two probes similar to the ip provider, send and receive. Both probes take 5 arguments, pktinfo_t, csinfo_t, ipinfo_t, udpsinfo_t and udpinfo_t.
udp:::receive(pktinfo_t *, csinfo_t *, ipinfo_t *, udpsinfo_t *,
udpinfo_t *);
udp:::send(pktinfo_t *, csinfo_t *, ipinfo_t *, udpsinfo_t *,
udpinfo_t *);
Again, pktinfo_t and csinfo_t are offered to provide compatibility with other systems. The ipinfo_t argument offers access to the packets IP header information.
udpsinfo_t argument contains the state of the UDP connection associated with the packet and provides a point to the struct inpcb for any associated socket. udpinfo_t provides access to the raw UDP header from the datagram.
The following one-liner will count the number of UDP datagrams for each destination port from the host:
# dtrace -n 'udp:::send {@[args[4]->udp_dport] = count();}'
dtrace: description 'udp:::send ' matched 1 probe
^C
53 8
56962 27
36944 29
Conclusion
There is a lot of information available to help you understand how to use DTrace. The 2011 DTrace book by Brendan Gregg and Jim Mauro is a good resource for looking at DTrace, how the D language works and provides a ton of examples of usage. The Illumos dtrace documentation is quite up to date and offers a good intro to the D language, with examples.
awesome-dtrace.com hosts a great collection of DTrace resources and is worth a look for examples and scripts. Finally, the 2021 Systems Performance book by Gregg is a great introduction to interrogating operating systems and applications, while it is Linux and BPF heavy a lot of the information transfers well to FreeBSD and is a great resource for structuring further explorations into performance analysis.
DTrace is a very powerful tool for observing what the operating system is up to. Using the full power of FDT probes requires familiar knowledge with how things are implemented. SDT probes and specific providers like ip, tcp and udpoffer insights into the internal behavior of the kernel without requiring a high level of knowledge of the underlying code.
What makes us different, is our dedication to the FreeBSD project.
Through our commitment to the project, we ensure that you, our customers, are always on the receiving end of the best development for FreeBSD. With our values deeply tied into the community, and our developers a major part of it, we exist on the border between your infrastructure and the open source world.
Embedded