freebsd networking

FreeBSD Network Troubleshooting: Understanding Network Performance

FreeBSD Network Troubleshooting

Learning more about network performance and the power tools available in FreeBSD for maximum network debugging

FreeBSD has a full set of debugging features, and the network stack is able to report a ton of information. So much that it can be hard to figure out what is relevant and what is not.

There are a number of things we can mean when we say a network is performing well and they don’t all have to align for a network to get a good score for a particular application. The nature of the workload is an important determinant of what is considered good. Network performance is dominated by two components that exist in symbiosis, delay and bandwidth.

Delay is the amount of time it takes for traffic to transverse the network. Most commonly we will talk about the round-trip time or ‘ping’, this measures the delay out and back to a remote host. Unless you are in a very special network (such as a VSAT link with dial up return) you can assume that the out and back delays are the same and that the one-way delay is half of the RTT. In reality, out and back routes very often differ, i.e. one path is faster than the other. Delay controls how responsive an application or protocol can be. The time between an action and the response to that action is referred to as the ‘latency’. We measure delay in milliseconds (ms).

Network capacity is the network’s ability to carry traffic over time; we normally use the term bandwidth when talking about the rates a network can send at, and in most cases these two terms are interchangeable. If you ever deal with radio systems it can be easy to get confused. Capacity governs the amount of information we can transfer in an amount of time (we use one second) and we typically talk about bandwidth in Megabit per second (Mbps) for links and Megabytes per second (MB/s) for application measures. The traffic a host or application can process in a second is also called its ‘throughput’.

We use the bandwidth and the delay together to estimate the size of a network with something called the Bandwidth Delay Product (BDP). The size of the BDP is important when calculating buffer sizes for transfers and is an area to tune. BDP governs a lot of the functionality of the congestion controller and hence the transfer speeds you can have between two hosts.

Protocols on the Internet are expected to share the available bandwidth fairly. Protocols that offer reliable delivery like TCP and QUIC use congestion controller to estimate how much bandwidth (the BDP) is available and react quickly by reducing sending rates when loss or reordering is detected. Classical congestion control algorithms grow and recover based on the number of packets that are successfully delivered at each RTT, meaning that large delays can have an impact on protocol performance.

Internet service is provided on a best effort basis, the network does not guarantee delivery of any particular packet. This is an optimization and simplification that has made our current Internet possible, but it means that errors are part of the default operating mode of the network. In a best effort network, we expect to see packet loss and protocols are either agnostic to it or have mechanisms to work around it. Protocols that offer reliable delivery, such as TCP, struggle when there is too much packet loss. High levels of packet loss (>1%) might be a sign of a network issue and will have a serious impact on transfer speeds.

Our Internet was built to be flexible to advances in host and link technology. In the 70s the routers of the network (at that point called imps) could convert between different byte orders and sizes (ever seen a non 8 bit byte?). We quickly realised that this put too much complexity into the network and made things adhere to a single standard. Link sizes were not standardised and instead we added mechanisms for hosts to learn the size of the largest packets that could be carried on a network path. When these mechanisms don’t work or aren’t usable, IPv4 routers are able to fragment packets. Unfortunately, fragments are expensive to process and require a lot of memory to reassemble and process on end hosts. For this reason, fragmentation is considered fragile and an increase in the number of fragments a host sees can indicate errors on the network.

Protocols: Hosts use many different protocols to communicate with each other on the internet and we layer these on top of each other to provide richer services to applications.

For performance we are going to concern ourselves with the network characteristics that affect the lower level protocols that carry communication. FreeBSD has a good view of the internals of these protocols and we can use system tools to understand how well they are performing. Application layer protocols such as HTTP can be a big factor in network performance, but analysis and tuning are tightly coupled to the application and out of scope for this article.

Two hosts are connected together by a series of links connected together through bridges, switches, and routers. The set of links that a packet passes through to get from one host to another is called a path. Paths should be assumed to be asymmetric (not the same in both directions) and expected to vary over time. Any measurement we take will give us values for that path, it might not apply well to other paths.

Establishing a Network Baseline

Before you can look at the performance of a system you first need to understand what ‘good’ performance for your host and network is. You need to have a set of baseline measurements from which you can determine if the network performance is lacking. The best way to do this is well in advance when machines are new and before they are seeing a deluge of traffic.

Measuring Delay

Delay is the most straight forward network characteristic to reason about, but the impact it can have on a connection’s performance can be unintuitive. The most common way to measure the delay between two hosts is to use the ‘ping’ tool. This is so common that most gamers will talk about their ‘ping’ time rather than delay at all. ping works by sending an ICMP Echo Request message and waiting for a matching ICMP Echo Reply message. The time between the request and the reply is the round-trip time.

[user@freebsd-laptop] $ ping -c 10 klarasystems.com
PING klarasystems.com (192.0.78.155): 56 data bytes
64 bytes from 192.0.78.155: icmp_seq=0 ttl=57 time=37.368 ms
64 bytes from 192.0.78.155: icmp_seq=1 ttl=57 time=140.856 ms
64 bytes from 192.0.78.155: icmp_seq=2 ttl=57 time=33.893 ms
64 bytes from 192.0.78.155: icmp_seq=3 ttl=57 time=34.486 ms
64 bytes from 192.0.78.155: icmp_seq=4 ttl=57 time=37.268 ms
64 bytes from 192.0.78.155: icmp_seq=5 ttl=57 time=35.898 ms
64 bytes from 192.0.78.155: icmp_seq=6 ttl=57 time=33.830 ms
64 bytes from 192.0.78.155: icmp_seq=7 ttl=57 time=35.521 ms
64 bytes from 192.0.78.155: icmp_seq=8 ttl=57 time=35.751 ms
64 bytes from 192.0.78.155: icmp_seq=9 ttl=57 time=34.804 ms

--- klarasystems.com ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 33.830/45.967/140.856/31.651 ms

With the ping command we learn a lot of things about the state of the network right now, the configuration of the host and the nature of the delay component of the path. Firstly, we learn that the host is connected to the network, how many network ‘hops’ away it is, if there is any packet loss, that DNS works, that we can get out of the local network and that we have valid routing. We get all of this by the fact that the ping command worked and we received responses.

Secondly, we learn several things about the nature of the delay component of our network path to this host. We can look at the final column of the ping output (the one with time=) and get a sense for how consistent the delay is. Ping also provides two summary lines at the end of a session (when you use the -c count flag, or if you press ^C).

The first summary line tells you about packet loss. Loss is a normal feature of the network; excessive levels of loss are worth investigating.

The second summary line tells you about the nature of the variance of delay in the network. This is provided in the form of the min, average, max, and standard deviation in delay. The second packet in this transfer had a time of 140.865ms, this might be significant if it happens frequently, but we can’t tell in the space of 10 packets here.

We have to consider the delay for the path for each host as a point in time measurement. Different hosts will almost always be in different physical locations and so there will be a variation in the delay characteristics between them. The path between two hosts can change at any time and when we design network protocols one requirement is to always consider that the paths will be different in different directions. This commonly happens when there are ECMP load balancers in the ingress path of a network.

For good performance it is important that delay is consistent over time. Reliable protocols, such as TCP, use the smoothed round trip time (SRTT) to establish timers that govern loss recovery and reliability mechanisms. If there are frequent delay spikes or the variation is very large these timers might fire accidentally.

Performance can also be improved by moving hosts closer together to reduce the delay. At the core, this is one of the large benefits to using a Content Delivery Network. Any transactional workload will be governed by the speed of light and any network process that reacts or grows based on round trip times will be limited by how many trips can be achieved in a second. Delay is a big issue for doing large transfers across the planet, moving 100GB on a 1Gb link is harder between Frankfurt and Sydney than it is between Frankfurt and London because it takes longer to grow the window and longer to recover after a packet loss event.

[user@freebsd-laptop] $ ping -c 1000 klarasystems.com
PING klarasystems.com (192.0.78.155): 56 data bytes
64 bytes from 192.0.78.155: icmp_seq=0 ttl=57 time=37.878 ms
64 bytes from 192.0.78.155: icmp_seq=1 ttl=57 time=36.338 ms
...
64 bytes from 192.0.78.155: icmp_seq=996 ttl=57 time=471.185 ms
64 bytes from 192.0.78.155: icmp_seq=997 ttl=57 time=34.490 ms
64 bytes from 192.0.78.155: icmp_seq=998 ttl=57 time=33.977 ms
64 bytes from 192.0.78.155: icmp_seq=999 ttl=57 time=33.887 ms

--- klarasystems.com ping statistics ---
1000 packets transmitted, 999 packets received, 0.1% packet loss
round-trip min/avg/max/stddev = 33.469/37.932/471.185/19.235 ms

The Internet is a best effort network, which means we expect packets to be lost and for delay to vary. The above trace was taken from my home network. Here we see a single packet getting dropped and one sample with a massive variation in delay. These two packets were probably victims to home WiFi.

Measuring Bandwidth

The actual speed a single connection can manage is influenced by a number of factors. The closest to the machine is the link speed of the network interface. For a wired connection such as Ethernet this will be something like 100Mbit if you are unlucky and more likely it will be 1Gb, 10Gbit or higher. Wired Network links are quite good at determining the speed actually available on the physical medium. Wireless links have a lot of factors that influence their speeds; you will probably see a link speed from 54Mbit with 80211g all the way up to a gigabit or more with WiFi 6.

The hosts link speed sets upper bound on the speed of any connection (ignoring aggregated links such as lagg(4)). A more likely factor will be the bandwidth made available via the service level agreement from the data center operator or ISP that offers the connection out to the internet. This quoted bandwidth will probably be a more realistic upper bound, with the real available value governed by network contention (how over-sold the link is) and congestion (how active the other users of the link are).

The capacity of the path is determined by the speed of the slowest link, how congested the link is, and how much buffering is on the link. Too little buffering and bursts of traffic will be dropping impacting performance and reducing throughput, but too much buffering and latency increases with traffic volume.

iperf3 (available as benchmarks/iperf3) is a standard, well-tested tool for measuring network performance. It has to be run as both the client and server and is able to report the throughput for TCP, UDP and SCTP connections and supports a host of options. There are other similar tools, it is important to use these consistently and be aware when you are comparing different versions of the software, as different versions can report different performance results.

iperf3 is also able to report results as JSON, which can be very helpful for integrating measurements into other tools.

To measure with iperf3 we need to run a server instance:

    [user@freebsd-server] $ iperf3 -s
-----------------------------------------------------------                    
Server listening on 5201                                                       
-----------------------------------------------------------                    
And connect from the client to perform measurements:
[user@freebsd-client] $ iperf3 -c freebsd-server.example.com
Connecting to host 10.0.4.2, port 5201
[  5] local 10.0.4.1 port 35041 connected to 10.0.4.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  99.9 MBytes   838 Mbits/sec  331    431 KBytes       
[  5]   1.00-2.00   sec   193 MBytes  1.62 Gbits/sec    0    855 KBytes       
[  5]   2.00-3.00   sec   197 MBytes  1.65 Gbits/sec   16    174 KBytes       
[  5]   3.00-4.00   sec   204 MBytes  1.71 Gbits/sec    0    787 KBytes       
[  5]   4.00-5.00   sec   196 MBytes  1.65 Gbits/sec    2    538 KBytes       
[  5]   5.00-6.00   sec   205 MBytes  1.72 Gbits/sec    0    934 KBytes       
[  5]   6.00-7.00   sec   217 MBytes  1.82 Gbits/sec    2    710 KBytes       
[  5]   7.00-8.00   sec   202 MBytes  1.69 Gbits/sec    0   1.02 MBytes       
[  5]   8.00-9.00   sec   166 MBytes  1.39 Gbits/sec    0   1.21 MBytes       
[  5]   9.00-10.00  sec   165 MBytes  1.38 Gbits/sec    0   1.37 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.80 GBytes  1.55 Gbits/sec  351             sender
[  5]   0.00-10.01  sec  1.80 GBytes  1.54 Gbits/sec                  receiver

iperf Done.

You might also be interested in

Improve the way you make use of ZFS in your company

ZFS is crucial to many companies. We guide companies and teams towards safe, whitepaper implementations of ZFS that enhance and improve the way the infrastructure is enabling your business.

The client will by default print the results for each second of transfer and perform the transfer for 10 seconds. When a client connects to the server, it will log the received rate for each second of transfer.

    [user@freebsd-server] $ iperf3 -s
-----------------------------------------------------------                    
Server listening on 5201                                                       
-----------------------------------------------------------                    
Accepted connection from 10.0.4.1, port 36348                                  
[  5] local 10.0.4.2 port 5201 connected to 10.0.4.1 port 35041                
[ ID] Interval           Transfer     Bitrate                                  
[  5]   0.00-1.00   sec  98.2 MBytes   823 Mbits/sec                           
[  5]   1.00-2.00   sec   192 MBytes  1.60 Gbits/sec                           
[  5]   2.00-3.00   sec   198 MBytes  1.67 Gbits/sec                           
[  5]   3.00-4.00   sec   204 MBytes  1.71 Gbits/sec                           
[  5]   4.00-5.00   sec   196 MBytes  1.64 Gbits/sec                           
[  5]   5.00-6.00   sec   205 MBytes  1.71 Gbits/sec                           
[  5]   6.00-7.00   sec   218 MBytes  1.83 Gbits/sec                           
[  5]   7.00-8.01   sec   201 MBytes  1.68 Gbits/sec                           
[  5]   8.01-9.00   sec   166 MBytes  1.41 Gbits/sec                           
[  5]   9.00-10.00  sec   165 MBytes  1.38 Gbits/sec                           
[  5]  10.00-10.01  sec   935 KBytes   588 Mbits/sec                           
- - - - - - - - - - - - - - - - - - - - - - - - -                              
[ ID] Interval           Transfer     Bitrate                                  
[  5]   0.00-10.01  sec  1.80 GBytes  1.54 Gbits/sec                  receiver 
-----------------------------------------------------------                    
Server listening on 5201                                                       
-----------------------------------------------------------                    

By default, iperf3 sends traffic from the client to the server, this being usually backwards to what you want to measure. We can get the opposite behaviour using the -R flag. As iperf3 sends for 10 seconds, networks with a high delay component might not have long enough to get up to full speed. I find it better for comparisons to transfer a fixed amount of data using the -n flag (there is an iperf3 bug where quantities don’t work in reverse mode).

TCP carries HTTP and because it makes up the majority of traffic on the Internet it has seen a lot of optimization. TCP is more likely to have offload configured, and tests on real hardware are likely to have much higher TCP throughput than UDP.

UDP tests are also helpful to understand your network. iperf3 will be default only; try to send a 1Mbit/s with UDP and so you have to specify a target bit rate.

[user@freebsd-client] $ iperf3 -c 10.0.4.2 -u -b 1G
Connecting to host 10.0.4.2, port 5201
[  5] local 10.0.4.1 port 16765 connected to 10.0.4.2 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   119 MBytes  1000 Mbits/sec  85586  
[  5]   1.00-2.00   sec   119 MBytes  1000 Mbits/sec  85587  
[  5]   2.00-3.00   sec   119 MBytes  1.00 Gbits/sec  85642  
[  5]   3.00-4.00   sec   119 MBytes  1000 Mbits/sec  85608  
[  5]   4.00-5.00   sec   119 MBytes  1.00 Gbits/sec  85626  
[  5]   5.00-6.00   sec   119 MBytes  1000 Mbits/sec  85590  
[  5]   6.00-7.00   sec   119 MBytes  1000 Mbits/sec  85607  
[  5]   7.00-8.00   sec   119 MBytes  1.00 Gbits/sec  85621  
[  5]   8.00-9.00   sec   119 MBytes  1000 Mbits/sec  85610  
[  5]   9.00-10.00  sec   119 MBytes  1000 Mbits/sec  85725  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.16 GBytes  1000 Mbits/sec  0.000 ms  0/856202 (0%)  sender
[  5]   0.00-10.10  sec  1.01 GBytes   859 Mbits/sec  0.003 ms  113542/856202 (13%)  receiver

iperf Done.

However, traffic on the parent interface that has a VLAN tag (traffic to 10.0.64.0/24 subnet) will show up in captures on that interface with the tag:

    root@hostA # tcpdump -i vtnet1 -e  
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on vtnet1, link-type EN10MB (Ethernet), capture size 262144 bytes
    19:59:12.596875 00:a0:98:67:03:ce (oui Unknown) > 00:a0:98:05:09:2c (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 5, p 0, ethertype IPv4, 10.0.64.1 > 10.0.64.2: ICMP echo request, id 33029, seq 0, length 64
    19:59:12.600823 00:a0:98:05:09:2c (oui Unknown) > 00:a0:98:67:03:ce (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 5, p 0, ethertype IPv4, 10.0.64.2 > 10.0.64.1: ICMP echo reply, id 33029, seq 0, length 64
    ^C
    2 packets captured
    2 packets received by filter
    0 packets dropped by kernel

When measuring UDP it is normally more helpful to note the rate at which that server can receive, rather than the rate at which the client can send. The host’s UDP stack will accept packets until all buffers are full and then silently drop packets, misrepresenting to the application how many packets are actually being sent each second. On the server side we will see the packets that actually make it across the network; you can see this in the below trace from a local virtual machine network.

    [user@freebsd-server] $ iperf3 -s
-----------------------------------------------------------                                  
Server listening on 5201                                                                     
-----------------------------------------------------------                                  
Accepted connection from 10.0.4.1, port 38610                                                
[  5] local 10.0.4.2 port 5201 connected to 10.0.4.1 port 16765                              
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams         
[  5]   0.00-1.00   sec  94.9 MBytes   796 Mbits/sec  0.007 ms  9432/77555 (12%)             
[  5]   1.00-2.00   sec   103 MBytes   861 Mbits/sec  0.007 ms  11865/85575 (14%)            
[  5]   2.00-3.00   sec   104 MBytes   869 Mbits/sec  0.002 ms  11261/85638 (13%)            
[  5]   3.00-4.00   sec   104 MBytes   871 Mbits/sec  0.002 ms  11018/85613 (13%)            
[  5]   4.00-5.00   sec   111 MBytes   932 Mbits/sec  0.006 ms  5788/85597 (6.8%)            
[  5]   5.00-6.00   sec   101 MBytes   843 Mbits/sec  0.005 ms  13379/85585 (16%)            
[  5]   6.00-7.00   sec   100 MBytes   842 Mbits/sec  0.009 ms  13515/85605 (16%)            
[  5]   7.00-8.00   sec   104 MBytes   872 Mbits/sec  0.011 ms  10908/85581 (13%)            
[  5]   8.00-9.00   sec   103 MBytes   867 Mbits/sec  0.002 ms  11390/85618 (13%)            
[  5]   9.00-10.00  sec   100 MBytes   842 Mbits/sec  0.004 ms  13610/85702 (16%)            
[  5]  10.00-10.10  sec  9.41 MBytes   831 Mbits/sec  0.003 ms  1376/8133 (17%)              
- - - - - - - - - - - - - - - - - - - - - - - - -                                            
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams         
[  5]   0.00-10.10  sec  1.01 GBytes   859 Mbits/sec  0.003 ms  113542/856202 (13%)  receiver
-----------------------------------------------------------                                  
Server listening on 5201                                                                     
-----------------------------------------------------------                                  

Because UDP is less likely to have effective offload support you will see that for the same transfer rate that TCP has much better CPU performance with iperf3. If you test on smaller devices (APU2 boards are a good test tool for this) you will observe that UDP only manages 400Mbit/s while TCP can saturate a Gigabit link. This happens because iperf3 becomes CPU limited for UDP, but TCP can offload a lot of the work to the network interface.

Passive Network Performance Information

The FreeBSD network stack gathers a lot of information as packets pass through, and these are exposed as statistic counters. You can access this information with netstat and the -s flag, and it is likely to give you much more than you can digest:

[user@freebsd-server] $ netstat -s
tcp:
    26734809 packets sent
        20912618 data packets (16879154564 bytes)
        11369 data packets (2782329 bytes) retransmitted
        90 data packets unnecessarily retransmitted
        0 resends initiated by MTU discovery
        5669526 ack-only packets (72899 delayed)
        0 URG only packets
        161 window probe packets
        3609 window update packets
... 461 more lines ...

    0 mbufs inserted during makespace
rip6:
    0 messages received
    0 checksum calculations on inbound
    0 messages with bad checksum
    0 messages dropped due to no socket
    0 multicast messages dropped due to no socket
    0 messages dropped due to full socket buffers
    0 delivered
    0 datagrams output

You can grep for what you want in the netstat output and you can filter for specific protocols by using the -p flag.

[user@freebsd-server] $ netstat -s -p ip 
ip:     
    35934055 total packets received
    0 bad header checksums
    0 with size smaller than minimum
    0 with data size < data length
    0 with ip length > max ip packet size
    0 with header length < data size
    0 with data length < header length
    0 with bad options
    0 with incorrect version number
    0 fragments received
    0 fragments dropped (dup or out of space)
    0 fragments dropped after timeout
    0 packets reassembled ok
    31395385 packets for this host
    0 packets for unknown/unsupported protocol
    0 packets forwarded (0 packets fast forwarded)
    0 packets not forwardable
    0 packets received for unknown multicast group
    0 redirects sent
    38879648 packets sent from this host
    0 packets sent with fabricated ip header
    0 output packets dropped due to no bufs, etc.
    31 output packets discarded due to no route
    0 output datagrams fragmented
    0 fragments created
    0 datagrams that can't be fragmented
    0 tunneling packets that can't find gif
    0 datagrams with bad address in header

Did you know?

We wrote about VNETs in “Virtualising Networks with FreeBSD VNET Jails

Statistic counters can be very helpful in monitoring activity, and detecting network failures and interface errors. In the ip, statistics above the total packets received and total packets sent tell us how active this host is. Any of the error condition counters such as ‘bad header checksums’, ‘bad options’ or ‘ip length > max ip size’ being non-zero or increasing can indicate misbehaving software and failing or badly configured network devices.

The IP counters for forwarding can be helpful to track and diagnose routing issues for which traces might be hard to capture. Netstat can also reveal if Path MTU Discovery is failing by showing both the incoming and outgoing fragmentation counters. While fragmentation is a requirement for the network to function, it is frequently an indicator of configuration issues somewhere on the path.

The FreeBSD tool systat makes it easier to track how network statistics change over time. systat offers a live display of network statistics and updates this information similar to how top behaves.

systat will provide information for network interfaces (via ifstat) and for network protocols (providing icmp, icmp6, ip, ip6, sctp and tcp).

systat has a command entry window which is accessed by typing a colon ‘:’ followed by a command string. This command window can be used to move between the different displays by typing their names. ‘:ifstat’ will take you to the ifstat display and ‘:tcp’ will take you to the TCP display.

systat -ifstat will provide current and peak transfer rates for all of the interfaces on a host marked as up, and will show a total amount of data transferred over the interface since systat was started. ifstat mode can show network information with different units with the scale command or as packets per second. It can also filter the interfaces shown, which is helpful on hosts with many virtual machines or jails.

                /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
 Load Average   ||||

  Interface           Traffic               Peak                Total
        lo0  in      0.000 KB/s          0.000 KB/s           51.187 KB
             out     0.000 KB/s          0.000 KB/s           51.187 KB

     vtnet0  in    204.499 MB/s        224.708 MB/s            4.985 GB
             out     4.459 MB/s          4.900 MB/s          111.414 MB

systat can be helpful to determine how much traffic is passing through a host. With real traffic this can be used as a good comparison with the measurements taken with iperf3, as well as previous measurements of the host’s activity.

systat has six screens of network protocol output to show you. UDP information is shown alongside ip and ip6. All of these screens operate in one of four modes: rate, delta, since or absolute.

                /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
 Load Average   |||

         TCP Connections                       TCP Packets
    7752 connections initiated          297828 total packets sent
     654 connections accepted           118740 - data
    8186 connections established           529 - data (retransmit by dupack)
     412 connections dropped                63 - data (retransmit by sack)
      26 - in embryonic state           159860 - ack-only
       8 - on retransmit timeout             0 - window probes
       0 - by keepalive                   3035 - window updates
       0 - from listen queue                 0 - urgent data only
                                         15663 - control
                                             0 - resends by PMTU discovery
         TCP Timers                     283757 total packets received
  119689 potential rtt updates          176485 - in sequence
  132848 - successful                     2184 - completely duplicate
   74552 delayed acks sent                   2 - with some duplicate data
     404 retransmit timeouts              1046 - out-of-order
       0 persist timeouts                 2777 - duplicate acks
       0 keepalive probes               133702 - acks
       0 - timeouts                          0 - window probes
                                           865 - window updates
                                          5829 - bad checksum

‘absolute’ mode shows the values of the counters as they are, the network protocol modes will start by showing you the absolute counters by default and then will show you rate of change on the first refresh. The other modes will show you how the counters are changing over time or since a point in time. The reference point for these can be set with the ‘reset’ command.

While you can watch the counters change, interpreting the information they provide requires understanding of the protocols and what the different counters represent.

In the TCP output above, the ‘TCP Connections’ and ‘TCP Timers’ sections tell us about connections and their status. Large numbers of connections in the ‘embryonic state’ could indicate a resource limit, a busy server or an attack. A large ‘listen queue’ might be a sign of a similar issue. Lots of connections in ‘keepalive’ might indicate a network or application protocol error. ‘persistent timeouts’ can show us that there is packet loss happening somewhere and we might want to look at performance on the path.

The ‘TCP Packets’ section tells us more raw information about the packets that make up our TCP connections. ‘sent’ and ‘received’ should be clear and represent the number of TCP segments sent and received. The ‘out-of-order’ and ‘duplicate acks’ fields can be signs of error. TCP is sensitive to reordering (hence out-of-order packets are an indication of a problem) and loss; loss typically appears as duplicate acks. This is a very quiet host on the Internet, with a long uptime and no performance issues I am aware of, so these numbers are not a concern.

                /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
 Load Average   |

      ICMP Input                         ICMP Output
268946 total messages			226150 total messages
0 with bad code				8655 errors generated
9 with bad length				0 suppressed - original too short
5 with bad checksum			0 suppressed - original was ICMP
0 with insufficient data		217495 responses sent
						0 suppressed - multicast echo
						0 suppressed - multicast tstamp
      Input Histogram                    Output Histogram

  326 echo response			217485 echo response
217485 echo request			0 echo request
50469 destination unreachable		8655 destination unreachable
13 redirect					0 redirect
608 time-to-live exceeded		0 time-to-live exceeded
0 parameter problem			0 parameter problem
0 router advertisement			0 router solicitation

Conclusion

“Benchmarks are hard to get right, and network benchmarks are harder”

The Internet we have built is one of humanity’s most impressive achievements, but that achievement is built on a lot of complexity. This article has touched a little bit on how we define performance and the factors we control that influence the available performance. We have looked at how we can measure what is available and how we can get the system to report what it is managing to do. We have looked at some indicators of errors, but as happy families are all the same, all unhappy networks are unique in their own way.

FreeBSD does a very good job of pushing packets and offers a lot of facilities to help you improve what it is doing. This article has only touched on the surface of network performance analysis.

<strong>Meet the Author</strong>: Tom Jones
Meet the Author: Tom Jones

Tom Jones is an Internet Researcher and FreeBSD developer that works on improving the core protocols that drive the Internet. He is a
contributor to open standards in the IETF and is enthusiastic about using FreeBSD as a platform to experiment with new networking ideas as they progress towards standardisation.

Like this article? Share it!

You might also be interested in

Get more out of your FreeBSD development

Kernel development is crucial to many companies. If you have a FreeBSD implementation or you’re looking at scoping out work for the future, our team can help you further enable your efforts.

More on this topic

Explaining top(1) on FreeBSD

We all know and have at least once used the top(1) command on FreeBSD to track information about our cpu and processes, but how many of you know what each field means? By default, top(1) displays the ‘top’ processes on each system and periodically updates this information every 2.0 seconds using the raw cpu use percentage to rank the processes in the list.
This article will give you some insight on how to better understand top (1).

Manipulating a Pool from the Rescue System

We’ve all been there: that moment of panic when a system fails to boot back up. Perhaps there was a glitch with an upgrade. Maybe you’re wondering if you fumble-fingered a typo when you made that last change to loader.conf.
Fortunately, with FreeBSD and its built-in rescue mechanisms it is possible to quickly recover from most scenarios that prevent a system from booting into normal operation. And if you’re using OpenZFS, you can rest assured that your data is intact.
With this article, let’s take a look at some common recovery scenarios.

FreeBSD Developer Workstation Setup

If you are eager to start FreeBSD development, but don’t know where to begin, this short guide is for you.
This article will provide some pointers into how some FreeBSD committers work on the Operating System and ports collection. We will discuss the hardware configurations that are used day to day to implement, build and test the operating system.

2 Comments on “FreeBSD Network Troubleshooting: Understanding Network Performance

  1. Pingback: Valuable News – 2021/03/08 | 𝚟𝚎𝚛𝚖𝚊𝚍𝚎𝚗

  2. Pingback: FreeBSD TCP Performance System Controls  | Klara Inc.

Tell us what you think!