Routing and Firewalling VLANS with FreeBSD

In this article we are going to look at and integrate two network isolation technologies, VLANs and VNET. VLANs are common place, and if you have done some network management or design then you are likely to have interacted with them. The second are FreeBSDs VNET virtual network stacks, a powerful network stack isolation technology that gives FreeBSD jails super powers.

Ethernet VLAN (standardised by IEEE 802.1Q) are an extension to Ethernet and provide an essential method for scaling network deployments. They are used in all environments to enable reuse of common infrastructure by isolating portions of networks from each other. VLANs allow the reuse of common cables, switches and routers to carry completely different networks. It is common to have data that must be separated from different networks carried on common cables until their VLAN tags are finally stripped at a gateway switch or router.

VLANs are implemented by inserting a 4-byte (32 bit) field into the Ethernet header, this field is referred to as the VLAN tag. The VLAN tag has a 12 bit VLAN ID field that carries a VLAN number. There can be up to 4096 VLAN IDs present in a network. Two VLAN IDs, 0 (0x0) and 4095 (0xFFF) are reserved. 0 is used to indicate that no VLAN ID is in use and the use of 0xFFF can be implementation defined.

VLAN tags are automatically added and removed by devices with VLAN support. FreeBSD implements VLANs as child devices cloned from a parent device (this can be a real network interface such as an em(4) device or virtual vtnet(4) device). Typically, we name these cloned devices with the VLAN ID or number that the handle. In this article we will talk about VLAN interfaces that are created with this scheme, so VLAN number 5 on vtnet0 is vtnet0.5. This convention makes it much easier to track which devices are doing what in a system.

Using VLANs on FreeBSD

There are a few ways to build test networks to experiment with VLANs and FreeBSD, we can use real hardware; a couple of machines and some switches, but that makes it hard to provide an example that can be reproduced by everyone. We can also use a FreeBSD host, either physical or virtual with bridge, epair and tap interfaces. If you have a physical machine for testing we can use bhyve virtual machines, but if all you have is a virtual FreeBSD machine you can reproduce this network using VNET jails.

Logically our test setup looks like two hosts connected directly with an Ethernet cable so if this network resembles your environment it should be plenty:

For the examples in this article we are going to use virtual machines connected together using tap and bridge interfaces and jails with epair interfaces on those virtual machines. Our example machines are called hostA and hostB, they are virtual machines with tap interfaces connected together through a bridge interface. On top of this we will build up a larger network inside using epair, bridge and jails. This setup could carry over to physical machines in the same data center or if connected together with a tunnel like GRE or IPsec, different data centers.

The vtnet1 interfaces on both hostA and hostB are connected together with a bridge. Both are in the 10.0.128.0/24, with the hostA at 10.0.128.1 and hostB at 10.0.128.2. We can ping to perform a simple test to verify that traffic can pass between the two virtual machines via the bridge before we add VLANs into the mix.

    root@hostA #  ping -c 1 10.0.128.2
    PING 10.0.128.2 (10.0.128.2): 56 data bytes
    64 bytes from 10.0.128.2: icmp_seq=0 ttl=64 time=2.647 ms

    --- 10.0.128.2 ping statistics ---
    1 packets transmitted, 1 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 2.647/2.647/2.647/0.000 ms

We create a VLAN interface from a parent device dynamically using ifconfig or statically using cloned_interfaces in rc.conf (see the FreeBSD handbook and vlan(4) man page for more information). We create the VLAN device with a name and indicate the parent device and VLAN number that will be used.

    root@hostA # ifconfig vtnet1.5 create vlan 5 vlandev vtnet1 
    root@hostA # ifconfig vtnet1.5 inet 10.0.64.1/24 up

Once the interface is created we can configure it the way we would with any other interface in FreeBSD. We need to create a matching VLAN device on the hostB so we can experiment with VLAN tags.

    root@hostB # ifconfig vtnet1.5 create vlan 5 vlandev vtnet1 
    root@hostB # ifconfig vtnet1.5 inet 10.0.64.2/24 up

Packet Captures for Debugging VLANs

While VLANs are not particularly complex technically, their ability to add new isolated networks also tends to add an extra layer of complexity and really reinforces confusion when you have to debug networking issues.

For a parent interface we can see all the traffic that passes through it along with its link layer headers by running tcpdump with with -e flag. To avoid locking out the console on a busy remote system it can be a good idea to only capture a limited number of packets (add the -c flag with a count such as 1000 to do this).

    # tcpdump -c 1000 -i vtnet1 -e

Traffic from the parent interface, captured on the parent interface (in our example traffic to the 10.0.128.0/24 subnet) will appear without any VLAN tag and so will traffic captured on the VLAN interface (traffic on the 10.0.64.0/24 subnet). This is because the parent device will send and receive raw untagged traffic and the child VLAN device will strip away the tag before processing any further:

    root@hostA #  tcpdump -i vtnet1 -e
    Password:
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on vtnet1, link-type EN10MB (Ethernet), capture size 262144 bytes
    19:49:55.906340 00:a0:98:67:03:ce (oui Unknown) > 00:a0:98:05:09:2c (oui Unknown), ethertype IPv4 (0x0800), length 98: 10.0.128.1 > 10.0.128.2: ICMP echo request, id 38148, seq 0, length 64
    19:49:55.908913 00:a0:98:05:09:2c (oui Unknown) > 00:a0:98:67:03:ce (oui Unknown), ethertype IPv4 (0x0800), length 98: 10.0.128.2 > 10.0.128.1: ICMP echo reply, id 38148, seq 0, length 64
    ^C
    2 packets captured
    2 packets received by filter
    0 packets dropped by kernel

    root@hostA # tcpdump -i vtnet1.5 -e
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on vtnet1.5, link-type EN10MB (Ethernet), capture size 262144 bytes
    19:58:17.234282 00:a0:98:67:03:ce (oui Unknown) > 00:a0:98:05:09:2c (oui Unknown), ethertype IPv4 (0x0800), length 98: 10.0.64.1 > 10.0.64.2: ICMP echo request, id 30725, seq 0, length 64
    19:58:17.238527 00:a0:98:05:09:2c (oui Unknown) > 00:a0:98:67:03:ce (oui Unknown), ethertype IPv4 (0x0800), length 98: 10.0.64.2 > 10.0.64.1: ICMP echo reply, id 30725, seq 0, length 64
    ^C
    2 packets captured
    2 packets received by filter
    0 packets dropped by kernel

However, traffic on the parent interface that has a VLAN tag (traffic to 10.0.64.0/24 subnet) will show up in captures on that interface with the tag:

    root@hostA # tcpdump -i vtnet1 -e  
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on vtnet1, link-type EN10MB (Ethernet), capture size 262144 bytes
    19:59:12.596875 00:a0:98:67:03:ce (oui Unknown) > 00:a0:98:05:09:2c (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 5, p 0, ethertype IPv4, 10.0.64.1 > 10.0.64.2: ICMP echo request, id 33029, seq 0, length 64
    19:59:12.600823 00:a0:98:05:09:2c (oui Unknown) > 00:a0:98:67:03:ce (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 5, p 0, ethertype IPv4, 10.0.64.2 > 10.0.64.1: ICMP echo reply, id 33029, seq 0, length 64
    ^C
    2 packets captured
    2 packets received by filter
    0 packets dropped by kernel

Comparing these two captures you can see that when the VLAN tag isn't present (such as when we capture on vtnet1.5) tcpdump doesn't tell us there is no tag, it doesn't say anything about VLANs at all. If you find yourself lost and start to wonder if VLANs have broken down, double check you are capturing in the correct place.

If you are not used to looking at its output, tcpdump isn't the friendliest. You can always write the capture to a file (with the -w flag) and pull it back to your desktop and view with Wireshark or use tshark with the -V flag to get very verbose per field descriptions.

Our test network has two distinct /24 networks running over the same shared infrastructure (here a bridge device).

Combining VLANs and VNETs

FreeBSD jails are an excellent way to isolate software and give us the ability to run multiple logical machines on a single host with minimal overhead, VNETs enhance jails further and give them a full network stack. The network stack that the jail sees is almost identical to one that the host machine would see and the jail is able to manage and firewall this network in the same ways.

In a similar way to how we can isolate traffic from multiple applications or customers we can use jails and VNETs to give root level control over how that traffic is managed. VNET jails and VLANs make it possible for us to allow customers in a multi-tenant hosting scenario to have firewall level access over their traffic while isolating other customers traffic from their view. We might use this as we connect hosted applications together over physically separate machines either in another rack or in a data center on the other side of the planet.

Using VLANs with VNET Jails

Here we have a short example how to integrate VLANs into a jailed setup that will allow a customer to control the firewalling of traffic (there are more detailed instructions in this article on "Virtualising Networks with FreeBSD VNET Jails".

Did you know?

We wrote about VNETs before in "Virtualising Networks with FreeBSD VNET Jails"

Check it out!

When we create a VNET jail we can provide it with a network interface, this interface stops being visible to the host and comes under the control of the jail. As we get an interfaces for each VLAN we can give this interface to a VNET jail giving the jail an interface on our shared infrastructure, but with the ability to only see traffic for its configured VLAN.

We can try this set up by creating a new jail and giving it our VLAN 5 interface:

    root@hostA # jail -c name=vlanjail persist vnet vnet.interface=vtnet1.5
    root@hostA # jls -v
       JID  Hostname                      Path   
            Name                          State  
            CPUSetID                             
            IP Address(es)                       
         1                                /      
            vlanjail                      ACTIVE 
            3

Now we can enter the jail and configure the interface:

    root@hostA # jexec vlanjail /bin/sh


    root@vlanjail# ifconfig
    lo0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
            options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
            groups: lo
            nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    vtnet1.5: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
            options=80000<LINKSTATE>
            ether 00:a0:98:67:03:ce
            groups: vlan
            media: Ethernet 10Gbase-T <full-duplex>
            status: active
            nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

    root@vlanjail # ifconfig vtnet1.5 inet 10.0.64.1/24 up
    root@vlanjail # ping 10.0.64.2
    PING 10.0.64.4 (10.0.64.2): 56 data bytes
    64 bytes from 10.0.64.2: icmp_seq=0 ttl=64 time=2.643 ms

    --- 10.0.64.2 ping statistics ---
    1 packets transmitted, 1 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 2.643/2.643/2.643/0.000 ms

Notice that when we move the VLAN interface into the jail all previous configuration on that interface is lost and we need to recreate it.

Firewalling VLAN subnetworks with VNET jails

We can use this pattern to create more complicated networks as well. A VNET jail has the same access and control over network as a full FreeBSD host so once we have done the required initial configuration on the host we can load firewalls as we wish inside these jails.

We can then use VNET jails to allow per VLAN firewalling if we take the vlanjail we created before, but configure it with half of an epair we are able to accept traffic on the VLAN, firewall it using ipfw and then forward it out of the jail.

The host must have the kernel modules loaded for the firewall, and to avoid getting locked out of the machine we set the default for ipfw to allow (you might not need to do this on a machine already using ipfw).

Firewalling with ipfw

    root@hostA # kenv net.inet.ip.fw.default_to_accept=1
    root@hostA # kldload ipfw ipfw_nat if_epair
    root@hostA # ifconfig epair create
    epair0a

    root@hostA # jail -c name=ipfwjail persist vnet vnet.interface=vtnet1.5 vnet.interface=epair0b

To show the network functioning with NAT we also need an application jail:

 
    root@hostB # jail -c name=appjail persist vnet vnet.interface=epair0a
    root@appjail # ifconfig epair0a inet 10.0.32.2/24 up
    root@appjail # route add default 10.0.32.1

This time when we create the VNET jail we give it two interfaces, the VLAN as before and half of an epair interface. epair interfaces come as two parts and model an Ethernet patch cable. We give the b part of the epair to the jail and the a of the epair remains visible on the host.

    root@hostA # jexec ipfwjail /bin/sh

    root@ipfwjail # ifconfig vtnet1.5 inet 10.0.64.1/24 up
    root@ipfwjail # route add default 10.0.64.1
    root@ipfwjail # ifconfig epair0b inet 10.0.32.1/24 up

With this configuration we are able to firewall any traffic from VLAN 5. With a rule set like the one below we can NAT the traffic from the hosts on the epair0b interface:

    root@ipfwjail # sysctl net.inet.ip.forwarding=1
    root@ipfwjail # ipfw nat 1 config if vtnet1.5 
    root@ipfwjail # ipfw add 101 nat 1 ip from 10.0.32.0/24 to any out via vtnet1.5
    root@ipfwjail # ipfw add 103 nat 1 ip from any to any in via vtnet1.5

In this set up we have a firewall that can be managed as its own logical host. Given a root file system and a control interface it can be managed by a customer to control all traffic to the jails or virtual machines connected to the epair0a side of the interface.

Firewalling with pf

All three of the firewalls supported by FreeBSD can be used with VNET jails, we can configure the same set up with a jail on hostB, but using pf:

    root@hostB # kldload pf if_epair

    root@hostB # ifconfig epair create
    epair0a

    root@hostB # jail -c name=pfjail persist vnet vnet.interface=vtnet1.5 vnet.interface=epair0b

We perform the same actions as we would for ipfw with the only difference being the kernel modules we have to load on the host.

    root@hostB # jexec pfjail /bin/sh

    root@pfjail # sysctl net.inet.ip.forwarding=1
    root@pfjail # ifconfig vtnet1.5 inet 10.0.64.2/24 up
    root@pfjail # ifconfig epair0b inet 10.0.16.1/24 up

    root@pfjail # pfctl -e
    root@pfjail # pfctl -f "nat pass on vtnet1.5 inet from 10.0.16.0/24 to any -> (vtnet1.5)"

To show the network functioning with NAT we again need an application jail:

    root@hostB # jail -c name=appjail persist vnet vnet.interface=epair0a
    root@appjail # ifconfig epair0a inet 10.0.16.2/24 up
    root@appjail # route add default 10.0.16.1

With the two sub networks configured we have the same infrastructure to offered NAT'd services with a firewall on two distinct hosts with a shared, but private link connecting them together.

Conclusion

When first experimenting, it is important to start with something simple. It can sometimes be far too easy to model very complex setups and then have to spend a lot of time debugging to understand what is not configured correctly.

These example networks offer both an introduction on how to set up VNET jails with VLANs and show some of the power of their use. A production network built from this would want to give each jail its own file system, this step was skipped to make it easier to follow along.

The BSD Router project has an example VLAN and VNET multi-tennant set up on their website that includes multiple different virtual machine frameworks. This example is well worth study and this article has hopefully provided the background to help you understand how this network is set up.

You might also be interested in

Get more out of your FreeBSD networking

Networking is crucial to many companies. If you have a FreeBSD networking implementation or a driver you're looking at, our team can help you further enable your efforts.

Read More