FreeBSD in the cloud

The Next Level – FreeBSD on arm64 In The Cloud

The Next Level

FreeBSD on arm64 in the Cloud

FreeBSD/arm64 is the FreeBSD port to the 64-bit ARM architecture, also known as AArch64 or ARMv8. All supported FreeBSD releases include support for ARMv8 and there are many packages and ports (3rd party applications) available to support the software you normally deploy with FreeBSD.

arm64 extends and modernizes the ARM architecture, adding specialized instructions and accelerators, as well as modern virtualization capabilities. The arm64 hardware, on which FreeBSD runs, ranges from small single board computers to mammoth multi-processor server machines with more than 100 cores on a single socket.

arm64 supports two models for describing hardware. Most single board computers (such as the raspberry pi or ROCK64) use flattened device tree (FDT) to describe the hardware, while other devices use the same ACPI (Advanced Configuration and Power Interface) as x86 CPUs. arm64 scales up to server platforms and the ARM specifications intend to make it possible to have images that ‘run anywhere’ through the SBSA (Server Base System Architecture). This standardizes the interface the machine uses to boot and interface with hardware. With SBSA and SBBR (Server Base Boot Requirement) arm64 hosts are able to boot using UEFI and discover hardware through ACPI. This makes it easier to deploy single images that work on many different hardware and virtual platforms, and brings arm64 deployment in line with amd64.

FreeBSD/arm64 has had support for server class platforms for a long time. The FreeBSD project uses Ampere servers to build packages for ARMv7 and ARMv8. These machines can be expensive and difficult to acquire, hence for experimentation it has been easier to pick up single board computers that are available for less than $50.

If you aren’t an embedded device vendor, then single board computers are not a proof-of-concept platform for more than the smallest experimentation. Thankfully FreeBSD/arm64 runs really well in the cloud and we can use cloud services to show that FreeBSD/arm64 is just as capable a server platform as any other of the supported architectures.

Cloud Services

arm64 is still breaking into the world of cloud hosting for virtual machines. AWS is the major host of virtual machines that will run arm64 images. At the time of writing Azure is experimenting with offering deployment pipelines on arm64, and other providers have experimented with arm64 virtual machines and full arm64 machine provisioning.

FreeBSD runs very well on the Amazon Web Services (AWS) with their Graviton 2 hosts. Graviton 2 is the second generation of general access arm64 servers that amazon has offered in the cloud.

AWS introduced their Graviton platform in November 2018 with their first generation of hardware using Annapurna Labs processors. This platform featured 16 Cortex-A72 cores organized as four quad-core clusters, all operating at 2.3 GHz.

At ReInvent, in December 2019, AWS introduced the Graviton2 platform built on top of ARM Neoverse N1 processors. These processors quadruple the number of cores compared to Graviton 1 from 16 to 64 and claim to offer up to 7x the performance of the first generation with the newer 7nm process.

Graviton2 processors power EC2 T4g, M6g, C6g and R6g instances, and their variants offer high performance NVMe-based storage and high-performance networking with Elastic Network Adapters (ENA).

Did you know?

Until the 30th of June 2021 AWS is offering free trial of the Graviton 2 t4g.micro instance size with 750 hours of free time available each month of the trial.

This is the perfect time to try running FreeBSD/arm64 in EC2.

Adding SSH Keys

We are going to access the virtual machine using ssh. EC2 can generate a key pair for each AMI as we launch it or we can upload our own. For most users with ssh keys, using an existing public key will be the easiest option. 

If you do not already have a ssh key pair you can generate one with the ssh-keygen command: 

[user@computer] $ ssh-keygen  
Generating public/private rsa key pair. 
Enter file in which to save the key (/home/user/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase):  
Enter same passphrase again:  
Your identification has been saved in /home/user/.ssh/id_rsa. 
Your public key has been saved in /home/user/.ssh/id_rsa.pub. 
The key fingerprint is: 
SHA256:M5nvBGhF6uWTnBseST7X75Qd0Cld6e5S2xD1t27rrCw user@computer.local 
The key's randomart image is: 
+---[RSA 3072]----+ 
|        .       o| 
|       o      o.+| 
|      . +    o.=.| 
|     . O * .  +.o| 
|      + S . . .oo| 
|     . . %   ..*.| 
|        o o   B.+| 
|         o E.+.=.| 
|          . .o*+.| 
+----[SHA256]-----+ 

We need to add the ssh public key (id_rsa.pub here) to AWS so EC2 can add the key to the virtual machine when it is created. We do so by visiting the EC2 console and selecting key pairs and selecting import key pair: 

It is good practice to give key pairs meaningful names especially if they are generated for particular hosts (you might have a different key for a laptop and a desktop). I call this key ‘arm64 testing’ as that is what it is used for. I prefer to paste the public key into the box:

Running FreeBSD on EC2

EC2 t4g instances offer great general workload performance with the ability to burst performance of CPU, storage and network. CPU burst credits are determined by instance size with all sizes of instance being able to burst network performance up to 5 Gigabit/s.

The current release of FreeBSD (12.2) is available as an official image produced by the FreeBSD Release Engineering team and offered on the Amazon Market Place by the FreeBSD Foundation.

Launching a t4g.micro instance is a matter of hitting the launch button from the EC2 web interface, searching for FreeBSD AMIs, selecting FreeBSD/ARM 12 AMI type and launching an instance.

When we launch the virtual machine we need to select the key pair we added:

After a few minutes the machine will be available in the instance dashboard and you will be able to ssh to the virtual machine with your ssh key. FreeBSD AMIs are configured with a user named ec2-user which is available for use with ssh. Once you connect you will be shown the motd as on any other FreeBSD host:

[user@computer] $ ssh ec2-user@3.249.84.154
Last login: Mon Mar 15 10:21:09 2021 from tmux(2897).%0
FreeBSD 12.2-RELEASE r366954 GENERIC

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories:   https://www.FreeBSD.org/security/
FreeBSD Handbook:      https://www.FreeBSD.org/handbook/
FreeBSD FAQ:           https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
FreeBSD Forums:        https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with:  pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed:  freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages:  man man
FreeBSD directory layout:      man hier

Edit /etc/motd to change this login announcement.
To easily configure your installed FreeBSD use bsdconfig(8).

        -- Lars Engels <lme@FreeBSD.org>
ec2-user@freebsd:~ $ uname -a
FreeBSD freebsd 12.2-RELEASE FreeBSD 12.2-RELEASE r366954 GENERIC  arm64

The ec2-user is in the wheel group and you can elevate to root by running the su command without a password.

Software Side

A lot of work has gone into FreeBSD/arm64 from both src and ports committers to make the platform a high-quality experience. Because of all of this hard work, using FreeBSD/arm64 in the cloud should not be very different than using amd64 based cloud images in any provider.

I ran some tests services with nginx web workloads and some iperf3 network performance tests without any issue on the FreeBSD/arm64 EC2 instances. The places you may encounter architecture related issues will be when software has not been written with portability or where FreeBSD/arm64 is not well supported upstream. 

The second area of where you may encounter issues is when ports have been marked as amd64 only because in the past amd64 was the main platform these tools would be run. With the SBSA, arm64 is much more similar to amd64 server hardware, and most of these tools will run without any problem once the port metadata has been updated to include arm64.

t4g Burstable Instances

EC2 t4g instances are burstable, when the CPU, storage (EBS) or network (ENA) is not running above a baseline capacity the instance accrues credits which can later be used when there is an increase in workload. When the instance runs above the baseline capacity it will spend accrued credit.

The free trial t4g.micro instance we deployed in this example will accrue CPU credits when the CPU load is below the baseline of 10% utilization per vCPU. When below the baseline the instance will accrue 12 credits per hour up to a maximum of 288 credits. Credits are spent at the millisecond interval by EC2 and information about credits is made available at 5-minute intervals through cloudwatch.

Starting with ENA version 2.3.0 t4g-network, FreeBSD supports exposing ENA network metrics via sysctl. You first have to start collecting metrics by setting the sysctl that governs the metrics interval:

# sysctl dev.ena.1.eni_metrics.sample_interval=10
# sysctl dev.ena.network_interface.en1_metrics

I was unable to test this feature; all of the instances I started came up with ENA version 2.2.0. 

Disk and network are also burstable and are throttled to a base line when their credit has been exhausted. The t4g burstable instances are a great way to experiment with FreeBSD/arm64 in the cloud, but don’t expect them to be able to handle sustained load as well as non-burstable instance types.

Conclusion

FreeBSD/arm64 is a growing platform, and I would expect to see more cloud providers offering services on arm64 and more and more powerful server hardware to come on to the market. arm64 is already able to provide significant power savings and new server offerings from companies such as Ampere offer very high core counts as well.

There is ongoing discussion in the FreeBSD project to promote FreeBSD/arm64 to a Tier 1 platform in FreeBSD 13. There is clear enthusiasm from FreeBSD developers for the arm64 architecture and it looks to have a great future in the project.

<strong>Meet the Author</strong>: Tom Jones
Meet the Author: Tom Jones

Tom Jones is an Internet Researcher and FreeBSD developer that works on improving the core protocols that drive the Internet. He is a
contributor to open standards in the IETF and is enthusiastic about using FreeBSD as a platform to experiment with new networking ideas as they progress towards standardisation.

Like this article? Share it!

You might also be interested in

Get more out of your FreeBSD development

Kernel development is crucial to many companies. If you have a FreeBSD implementation or you’re looking at scoping out work for the future, our team can help you further enable your efforts.

More on this topic

FreeBSD TCP Performance System Controls 

While new protocols are constantly being developed, the venerable Transmission Control Protocol (TCP) still accounts for most global traffic. The FreeBSD kernel TCP stack offers a lot of opportunities to tweak different performance features. The options it includes allow a lot of flexibility in the configuration of machines without having to do custom kernel builds.
Find out how to make use of the Initial Window, what the TCP Segment OffLoad is, and how to use TCP Buffer Tuning to your advantage.

freebsd networking

FreeBSD Network Troubleshooting: Understanding Network Performance

Network performance is one of the most complex topics to analyse and understand. FreeBSD has a full set of debugging features, and the network stack reports a ton of information. So much that it can be hard to figure out what is relevant and what is not. In this article, we define performance, look at how to measure what is available and how to get the system to report what it is managing to do.

freebsd network virtualized

Routing and Firewalling VLANS with FreeBSD

VNET virtual network stacks are a powerful network stack isolation technology that gives FreeBSD jails super powers. Follow our guide to use VLANs on FreeBSD, combine VLANs and VNETs and use VLANs with VNET Jails. Learn useful tricks with many exemplifying instances.

2 Comments on “The Next Level – FreeBSD on arm64 In The Cloud

  1. Pingback: Valuable News – 2021/03/29 | 𝚟𝚎𝚛𝚖𝚊𝚍𝚎𝚗

  2. Pingback: FreeBSD arm64 Performance – Getting more out of your FreeBSD Deployment | Klara Inc.

Tell us what you think!