DTrace Network Probes

DTrace Network Probes

Discover the awesome powers of Dtrace and take a deep dive into FreeBSD observability

DTrace is an observability framework that enables dynamic tracing of programs, and the FreeBSD Kernel. DTrace has been ported to many different Operating Systems, including NetBSD, Mac OS, and Windows. DTrace was originally developed for Sun’s Solaris operating system with DTrace seeing its first release in 2005. DTrace landed in FreeBSD 7.1 in 2009, this work being documented by a BSDCan Talk that explains its usage and some of its internals. DTrace offers an incredible view into the operation of programs, and is an excellent tool for debugging and performing analysis of complex software.

DTrace was the first dynamic observability framework to see large scale deployment. If you have done performance analysis on Linux in recent years, you might be familiar with extended BPF (or just BPF now). BPF offers a similar set of tools and has similar interfaces to its command line tools, but the Linux kernel developers were able to learn from a decade of production DTrace use, and BPF offers even more power and more flexibility than DTrace.

The DTrace framework is a combination of probe points compiled into software, a C like programming language called D and a large collection of scripts built on top of DTrace programs. The main interface to DTrace that we will interact with is the dtrace command line tool.

An introduction to the D language is out of scope for this article, interested readers should look at the Illumos DTrace Guide or the 2011 DTrace Book by Gregg and Mauro for background on writing D programs and how to use its powerful features.

DTrace Probe points are specially marked sections of a compiled program that allow a smart tool (in our case dtrace) to hook in and inject additional code to run when the probe is activated, but also be able to remove the code when the probe is deactivated. Probe points are marked by the compiler, and their locations are stored in metadata that the DTrace tools access. On FreeBSD, this information is stored in Compact C Type Format (CTF) and included in ELF objects.

When DTrace probes are not enabled they have effectively 0 impact on the operation of software. DTrace has been used in production for 16 years and it has been demonstrated to be a solid tool for introspecting running systems. DTrace technologies enable tools that allow systems moving Gigabit/s of traffic to be debuggable with only modest impact on CPU usage.

When the kernel is being built, probe points are created for every function entry and exit, and for specific points in the kernel marked out by developers. The entry and exit probe points are called Function Boundary Traces or FBT (entry and exit being the edges of function calls), and static probe points are called Statically Defined Tracing or SDT. A third type of probe you might see is used for static trace points in userspace programs, and these are called USDT or User Statically Defined Tracing.

FBT probes allow us to instrument any function that ends up in the final program without having to make any special effort to declare places we might be interested. Not all source code functions end up in the final program; inlined functions do not have FBT probes.

SDT probes are added by developers and offer an API for inspecting the kernel at specific points. In this article, we are going to look at statically defined probe points for the IP, TCP, and UDP network protocols.

Statically Define Probes

SDT probes allow kernel developers to offer an external API to consumers using DTrace. SDT probes have the benefit of being specifically located in key parts of the kernel and enable the tracing of code in locations other than function entry and function return.

Did you know?

Improving your FreeBSD infrastructure has never been easier. Our teams are ready to consult with you on any FreeBSD topics ranging from development to regular support.

The downside to SDT probes is that they have to be implemented to be available, and they offer a static interface. In production releases you are stuck with the probes that kernel developers placed for you to use. Probes are limited to the arguments selected for the interface and you aren’t able to add access to other structures that you might want in the future.

Adding statically probes is straight forward once their subsystem has been added. This example from the network stack shows the receive probes for UDP and UDP-Lite:

UDPLITE and UDP SDT probes from sys/netinet/udp_usrreq.c:
...
    if (proto == IPPROTO_UDPLITE)
        UDPLITE_PROBE(receive, NULL, last, ip, last, uh);
    else
        UDP_PROBE(receive, NULL, last, ip, last, uh);
    if (udp_append(last, ip, m, iphlen, udp_in) == 0)
        INP_RUNLOCK(last);
    return (IPPROTO_DONE);
...

There can be multiple points in the kernel that will call the same probe. This means that SDT probes are good for representing state transitions of logical points of execution rather than specific pieces of code.

Providers

Access to DTrace probes is available through a 4-level hierarchy. Probes are stored in libraries called providers, and each library can have multiple sub modules – there is an optional function namespace (used for FBT probes) and finally each probe has a name. On FBT probes the function is the function from the original source code, and the probe names represent the entry and return boundary points.

DTrace probes are written in the form provider:module:function:name. DTrace will do its best to match missed sections, and allows the use of wild cards.

We can list all of the probes on our system with dtrace -l:

# dtrace -l 
   ID   PROVIDER            MODULE                          FUNCTION NAME
    1     dtrace                                                     BEGIN
    2     dtrace                                                     END
    3     dtrace                                                     ERROR
    4        fbt            kernel                camstatusentrycomp entry
    5        fbt            kernel                camstatusentrycomp return
    6        fbt            kernel            cam_compat_handle_0x17 entry
    7        fbt            kernel            cam_compat_handle_0x17 return
    8        fbt            kernel            cam_compat_handle_0x18 entry
    9        fbt            kernel            cam_compat_handle_0x18 return
...
75272   dtmalloc                                             entropy malloc
75273   dtmalloc                                             entropy free
75274   dtmalloc                                             CAM_DEV malloc
75275   dtmalloc                                             CAM_DEV free
75276   dtmalloc                                                 PUC malloc
75277   dtmalloc                                                 PUC free
75278   dtmalloc                                            ppbusdev malloc
75279   dtmalloc                                            ppbusdev free
75280   dtmalloc                             agtiapi_MemAlloc_malloc malloc
75281   dtmalloc                             agtiapi_MemAlloc_malloc free

On this system, there are 75282 probes

dtrace -l | wc -l
   75282

This large number of probes represents all of the traced function calls that have ended up in the kernel. We can list the probes for a provider with the -P flag:

dtrace -l -P fbt | wc -l   
   71324

This command shows that the vast majority of the available probes come from the fbt provider. DTrace on FreeBSD offers a large number of providers that (other than FBT) represent areas where developers think probe points would be interesting:

# dtrace -l | awk '{print $2}' | uniq
PROVIDER
dtrace
fbt
random
xbb
sched
proc
lockstat
priv
proc
racct
proc
sched
proc
callout_execute
sched
io
lock
sched
sdt
vfs
vnet
ip
sctp
tcp
udp
udplite
sctp
opencrypto
mac_framework
mac
mac_framework
vm
vfs
sdt
profile
syscall
nfscl
dtmalloc

Some DTrace providers have manual pages that offer in-depth information on how to use the provider and its probes. For network probes, dtrace_tcp(4), dtrace_ip(4), and dtrace_udp(4) offer information about the arguments passed to the probes and explain what is reachable from DTrace.

In this article, we are interested in looking at network protocols with DTrace; the SDT probes for TCP are much richer than the static probes for other protocols:

# dtrace -l -P tcp -P ip -P udp
   ID   PROVIDER            MODULE                          FUNCTION NAME
71450        tcp            kernel                              none accept-established
71451        tcp            kernel                              none accept-refused
71452        tcp            kernel                              none connect-established
71453        tcp            kernel                              none connect-refused
71454        tcp            kernel                              none connect-request
71455        tcp            kernel                              none receive
71456        tcp            kernel                              none send
71457        tcp            kernel                              none siftr
71458        tcp            kernel                              none debug-input
71459        tcp            kernel                              none debug-output
71460        tcp            kernel                              none debug-user
71461        tcp            kernel                              none debug-drop
71462        tcp            kernel                              none state-change
71463        tcp            kernel                              none receive-autoresize
71445         ip            kernel                              none receive
71446         ip            kernel                              none send
71464        udp            kernel                              none receive
71465        udp            kernel                              none send

The number of available static probes is much smaller than the probe points we can examine using FBT probes.

# sudo dtrace -l -P tcp | wc -l
      15

There are 15 probe points in the TCP provider vs 950 FBT probe points:

# dtrace -l -P fbt -m kernel | grep tcp | wc -l
     950

The downside is that when using FDT probes you are at the whims of the compiler to generate the probe points, in the future those may change, therefore FBT offers an unstable API. We also have to have a good understanding of what the kernel implementation is trying to do to understand and best use the FBT probe points.

IP

DTrace offers send and receive probes in the ip provider. With the data passed as arguments to the probes, we can use these to extract a lot of interesting information about the packets the system is sending and receiving.

Both of the probes in the ip provider have the same 6 arguments:

ip:::receive(pktinfo_t *, csinfo_t *, ipinfo_t *, ifinfo_t *,
    ipv4info_t *, ipv6info_t *);

ip:::send(pktinfo_t *, csinfo_t *, ipinfo_t *, ifinfo_t *, ipv4info_t *,
    ipv6info_t *);

The arguments, pktinfo_t and csinfo_t, are unimplemented and exist to allow FreeBSD DTrace probes to be compatible with other systems.

ipinfo_t contains IP fields common to both IPv4 and IPv6 packets, and the ifinfo_t field contains information about the outgoing and incoming interfaces for packets.

ipv4info_t and ipv6info_t contain the IPv4 and IPv6 header fields for packets; the corresponding field is NULL for each address family (i.e. for v6 packets ipv4info_t is empty).

We can use the information from the man page and some DTrace magic to look at the IP traffic we are receiving. This one-liner shows who is sending to you and counts how many packets are received for each sender:

# dtrace -n 'ip:::receive {@[args[2]->ip_saddr] = count();}'
dtrace: description 'ip:::receive ' matched 1 probe
^C

  186.33.76.219                                                     1
  40.119.108.0                                                      1
  52.17.231.73                                                      1
  85.91.1.164                                                       1
  87.198.32.53                                                      1
  40.119.104.0                                                      3
  47.105.203.221                                                    3
  222.186.180.130                                                   7
  168.63.129.16                                                    50
  62.171.156.18                                                   259
  92.26.12.92                                                     367

dtrace_ip(4) documents the fields which are exposed by the probes in the ip provider. In the above example we are selecting the third argument which is the ipinfo_t struct and using the ip_saddr member to aggregate senders.

Below, we have an example which counts the distribution of IP payload size by destination address on send:

# dtrace -n 'ip:::send { @[args[2]->ip_daddr] = quantize(args[2]->ip_plength);}'
dtrace: description 'ip:::send ' matched 1 probe
^C

  167.248.133.25                                    
       value  ------------- Distribution ------------- count    
           8 |                                         0        
          16 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1        
          32 |                                         0        

  168.63.129.16                                     
       value  ------------- Distribution ------------- count    
          16 |                                         0        
          32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@        10       
          64 |@@@@@@@                                  2        
         128 |                                         0        

  92.26.12.92                                       
       value  ------------- Distribution ------------- count    
          32 |                                         0        
          64 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@   92       
         128 |@@                                       6        
         256 |                                         0        

  62.171.156.18                                     
       value  ------------- Distribution ------------- count    
          16 |                                         0        
          32 |@@@@@@@@@@@@@@@@@@@@                     25       
          64 |@@@@@@@@@@@@                             15       
         128 |                                         0        
         256 |                                         0        
         512 |@@@@                                     5        
        1024 |@@@@                                     5        
        2048 |                                         0        

TCP

The DTrace tcp provider offers 14 probes that we can look at, 8 are documented in the man page and the others are debug probes. There are more probe points exposed for TCP because it is a connection oriented, reliable protocol which leads to there being more sources for errors and stalls in connections.

Each of the tcp provider probes has 5 arguments, for all apart from state-changed they are: pktinfo_t, csinfo_t, ipinfo_t,tcpsinfo_t and tcpinfo_t. pktinfo_t and csinfo_t are unimplemented and made available to allow FreeBSD’s tcp provider to be more compatible with other DTrace implementations. The state-changed probe has a slightly different set of arguments and these are documented in the man page.

The ipinfo_t argument is an address family agnostic representation of the fields from the IP header and is described in the dtrace_ip(4) man page.

tcpsinfo_t is a stable representation of the TCP connection state, and the tcpinfo_t argument exposes the fields from the TCP header on the received segment in host order.

With this information, we can inspect the TCP connections to the machine. The following one-liner shows new successful connections by remote address and local port number (who is connecting to you and what):

# dtrace -n 'tcp:::accept-established { @[args[3]->tcps_raddr, args[3]->tcps_lport] = count();;}'
dtrace: description 'tcp:::accept-established ' matched 1 probe
^C

  49.88.112.75                                          22                1
  62.171.156.18                                         22                4
  92.26.12.92                                         4000                4

This one-line shows that there is traffic to port 22, likely ssh scans, and to port 4000, where I am running a development web server.

TCP is governed by a state machine and the probes that fire during a sampling period can give you a good idea of what is happening on the machine. The following examples traces the TCP Probes that fire and gives a count by name:

# dtrace -n 'tcp::: { @[probename] = count();}'
dtrace: description 'tcp::: ' matched 14 probes
^C

  accept-established                                                6
  receive-autoresize                                               10
  state-change                                                     17
  debug-user                                                      323
  debug-output                                                   2485
  send                                                           2492
  debug-input                                                    3303
  receive                                                        3306

UDP

The DTrace udp provider offers two probes similar to the ip provider, send and receive. Both probes take 5 arguments, pktinfo_t, csinfo_t, ipinfo_t, udpsinfo_t and udpinfo_t.

udp:::receive(pktinfo_t *, csinfo_t *, ipinfo_t *, udpsinfo_t *,
    udpinfo_t *);
udp:::send(pktinfo_t *, csinfo_t *, ipinfo_t *, udpsinfo_t *,
    udpinfo_t *);

Again, pktinfo_t and csinfo_t are offered to provide compatibility with other systems. The ipinfo_t argument offers access to the packets IP header information.

udpsinfo_t argument contains the state of the UDP connection associated with the packet and provides a point to the struct inpcb for any associated socket. udpinfo_t provides access to the raw UDP header from the datagram.

The following one-liner will count the number of UDP datagrams for each destination port from the host:

# dtrace -n 'udp:::send {@[args[4]->udp_dport] = count();}'
dtrace: description 'udp:::send ' matched 1 probe
^C

    53                8
 56962               27
 36944               29

Conclusion

There is a lot of information available to help you understand how to use DTrace. The 2011 DTrace book by Brendan Gregg and Jim Mauro is a good resource for looking at DTrace, how the D language works and provides a ton of examples of usage. The Illumos dtrace documentation is quite up to date and offers a good intro to the D language, with examples.

awesome-dtrace.com hosts a great collection of DTrace resources and is worth a look for examples and scripts. Finally, the 2021 Systems Performance book by Gregg is a great introduction to interrogating operating systems and applications, while it is Linux and BPF heavy a lot of the information transfers well to FreeBSD and is a great resource for structuring further explorations into performance analysis.

DTrace is a very powerful tool for observing what the operating system is up to. Using the full power of FDT probes requires familiar knowledge with how things are implemented. SDT probes and specific providers like ip, tcp and udpoffer insights into the internal behavior of the kernel without requiring a high level of knowledge of the underlying code.

<strong>Meet the Author</strong>: Tom Jones
Meet the Author: Tom Jones

Tom Jones is an Internet Researcher and FreeBSD developer that works on improving the core protocols that drive the Internet. He is a
contributor to open standards in the IETF and is enthusiastic about using FreeBSD as a platform to experiment with new networking ideas as they progress towards standardisation.

Like this article? Share it!

You might also be interested in

Get more out of your FreeBSD development

Kernel development is crucial to many companies. If you have a FreeBSD implementation or you’re looking at scoping out work for the future, our team can help you further enable your efforts.

More on this topic

Using the FreeBSD RACK TCP Stack

Did you know that FreeBSD has more than one TCP stack and that TCP stacks are pluggable at run time? Since FreeBSD 12, FreeBSD has support pluggable TCP stacks, and today we will look at the RACK TCP Stack. The FreeBSD RACK stack takes this pluggable TCP feature to an extreme: rather than just swapping the congestion control algorithm, FreeBSD now supports dynamically loading and an entirely separate TCP stack. With the RACK stack loaded, TCP flows can be handled either by the default FreeBSD TCP stack or by the RACK stack.

FreeBSD TCP Performance System Controls 

While new protocols are constantly being developed, the venerable Transmission Control Protocol (TCP) still accounts for most global traffic. The FreeBSD kernel TCP stack offers a lot of opportunities to tweak different performance features. The options it includes allow a lot of flexibility in the configuration of machines without having to do custom kernel builds.
Find out how to make use of the Initial Window, what the TCP Segment OffLoad is, and how to use TCP Buffer Tuning to your advantage.

freebsd networking

FreeBSD Network Troubleshooting: Understanding Network Performance

Network performance is one of the most complex topics to analyse and understand. FreeBSD has a full set of debugging features, and the network stack reports a ton of information. So much that it can be hard to figure out what is relevant and what is not. In this article, we define performance, look at how to measure what is available and how to get the system to report what it is managing to do.

One Comment on “DTrace Network Probes

  1. Pingback: Valuable News – 2021/06/14 | 𝚟𝚎𝚛𝚖𝚊𝚍𝚎𝚗

Tell us what you think!