Using Object Storage with OpenZFS and SeaweedFS

Object storage has turned into the default persistence layer for modern infrastructure, especially for cloud native applications. By combining OpenZFS and SeaweedFS, we can provide data integrity, deployment, flexibility, and high performance with open source software. Backup systems, media archives, CI/CD pipelines, AI datasets, Kubernetes workloads, and distributed apps now tend to expect an S3-compatible API rather than a classic POSIX filesystem. 

Most traditional filesystems were built for a world where data came in larger, steadier chunks and where locality was easier to assume. Once workloads move into object storage patterns, you get millions or billions of objects, mixed write patterns, heavy metadata lookups, and replication traffic that never really stops. Small random reads sit next to large sequential writes and metadata starts to dominate cost and latency. At that point, older filesystem designs start to feel strained because they were not built for this kind of load profile. 

A modern object layer paired with ZFS changes the equation in a practical way. SeaweedFS fits into this space as a object storage system designed around simplicity and high throughput rather than heavy orchestration. In the sections that follow, we will look at SeaweedFS starting from its architecture and then move toward how SeaweedFS can be deployed on top of ZFS. 

Why SeaweedFS? 

SeaweedFS takes a different route compared to heavier systems in the same space. Instead of building a deep stack of orchestration layers, it keeps the design light and focuses on keeping metadata overhead low and disk writes mostly sequential.  

When you compare SeaweedFS with systems like MinIO or Ceph, the difference is easy to notice. Those systems are powerful but tend to come with more moving parts and a heavier setup footprint. SeaweedFS can be deployed in a smaller footprint and brought online quickly. That matters in environments where teams want object storage without turning it into a full time infrastructure domain. 

The license model also plays a quiet but important role in adoption. SeaweedFS uses the Apache 2.0 license which is permissive and low friction for both private and commercial use. It allows teams to build on top of it, ship products, and run large deployments without strict copyleft constraints. In practice, this reduces legal overhead and makes it easier for infrastructure teams to adopt it inside larger systems without negotiation loops around licensing boundaries. 

The Architecture of SeaweedFS 

SeaweedFS derives much of its efficiency from how it stores data on disk. When OpenZFS and Seaweedfs are paired together, they cooperate to deliver append-oriented, high-throughput object storage. Instead of scattering individual objects across large directory trees, it aggregates many objects into larger sequential volume files. This dramatically reduces random write amplification and minimizes filesystem metadata churn. 

The design follows principles established by the Haystack architecture originally developed for large-scale photo storage by Facebook. Sequential append-heavy workloads are significantly friendlier to disks than millions of small random writes. Traditional filesystems often degrade under workloads dominated by tiny objects because every object creation generates metadata updates, allocation changes, and directory operations. SeaweedFS avoids much of this overhead by treating storage volumes as append-oriented containers. 

This approach maps extremely well onto ZFS behavior as it  performs best when workloads are predictable and sequential. Large streaming writes reduce fragmentation and improve transaction group efficiency. SeaweedFS naturally generates these patterns, especially when volume sizes and ZFS recordsize settings are aligned correctly. 

Another important architectural characteristic is the separation between metadata and bulk object data. SeaweedFS distinguishes between filer metadata and object volume storage. This separation allows you to place metadata on lower-latency media while keeping bulk object storage on higher-capacity devices. With ZFS datasets, this becomes straightforward to implement operationally. 

For example: 

• A high-performance NVMe-backed dataset can store filer metadata. 

• Large HDD RAIDZ pools can store object volumes. 

• Different compression policies can be applied independently. 

• Dataset-level snapshot schedules can vary by workload sensitivity. 

This flexibility is one of the most practical advantages of ZFS in object storage environments. Storage policies are not globally rigid and they can be adapted per dataset according to access patterns and operational priorities. 

Optimizing ZFS for SeaweedFS 

A SeaweedFS deployment running on default ZFS configuration will function correctly, but performance characteristics improve considerably when the storage layout is tuned around SeaweedFS write behavior. 

One of the most important adjustments involves recordsize alignment. SeaweedFS writes large sequential volumes and matching ZFS recordsize to this behavior reduces fragmentation and minimizes unnecessary metadata overhead. In most object storage deployments, a 1M recordsize provides a strong balance between throughput and storage efficiency. 

recordsize = 1M

Smaller recordsizes force ZFS to manage additional metadata and can fragment large sequential writes into unnecessarily small blocks. This becomes especially costly during sustained ingest workloads or replication activity. 

VDEV topology is equally important as mirror-based pools generally provide lower write latency and faster resilvering characteristics. They are often preferable for workloads emphasizing performance consistency, metadata responsiveness, or high concurrency. RAIDZ configurations, on the other hand, provide significantly higher usable capacity efficiency and may be preferable for large archival clusters prioritizing storage density. 

There isn’t any principled correct choice for all workloads.  Mirrors typically improve latency and rebuild performance and RAIDZ generally improves capacity utilization. The correct choice depends on workload expectations, fault domain planning, and hardware budget. 

Metadata performance deserves particular attention in object storage systems because metadata operations frequently become the hidden bottleneck. Special VDEVs provide an better solution within ZFS as by placing metadata and optionally small blocks on SSD or NVMe storage, lookup latency improves dramatically even when bulk object data resides on slower disks. 

For SeaweedFS filer workloads, this can substantially accelerate: 

• Directory traversal 

• Object listing operations 

• Bucket metadata queries 

• Small object retrieval 

• Namespace operations 

In practice, relatively modest NVMe devices can significantly improve responsiveness across the entire storage platform. 

Deployment on Linux:  

On Linux, SeaweedFS deployments commonly use containers for orchestration and service isolation. To get S3 working, you need four components running together. In a single-node setup, you run these in one Docker Compose stack: 

• Master: it  tracks where data lives. 

• Volume: it stores the actual data on your ZFS pool. 

• Filer: it turns raw data into a directory structure. 

• S3: the "gateway" that provides the S3 API. 

Create the ZFS Pool and Dataset: 

sudo zpool create -f seaweed-pool /dev/da2
# Create the dataset with optimizations for SeaweedFS's large volume files 
sudo zfs create -o compression=zstd -o recordsize=1M -o xattr=sa -o atime=off seaweed-pool/data 

# Set permission for Docker  
sudo chown -R 1000:1000 /seaweed-pool/data 

Create the docker-compose.yml: 

services: 
  master: 
    image: chrislusf/seaweedfs:latest 
    command: master -ip=master 
    ports: 
      - "9333:9333" 

  volume: 
    image: chrislusf/seaweedfs:latest 
    command: 'volume -mserver="master:9333" -port=8080 -dir=/data -max=20' 
    ports: 
      - "8080:8080" 
    volumes: 
      - /seaweed-pool/data:/data 
    depends_on: 
      - master 

  filer: 
    image: chrislusf/seaweedfs:latest 
    command: 'filer -master="master:9333"' 
    ports: 
      - "8888:8888" 
      - "18888:18888" 
    depends_on: 
      - master 
      - volume 

  s3: 
    image: chrislusf/seaweedfs:latest 
    command: 's3 -filer="filer:8888" -ip.bind=0.0.0.0' 
    ports: 
      - "8333:8333" 
    depends_on: 
      - filer 

Launch the stack: 

docker compose up -d 

Deployment on FreeBSD 

On FreeBSD, rather than abstracting services behind container layers, running OpenZFS and SeaweedFS inside the native jails framework managed using Bastille or iocage, preserving direct integration with the host operating system. 

This model aligns naturally with ZFS because it is deeply integrated into the FreeBSD storage stack. SeaweedFS services running inside jails maintain near-native performance characteristics while avoiding much of the complexity associated with Docker container networking and storage layers.  

To setup SeaweedFS on FreeBSD: 

Create a dedicated dataset: 

umair@freebsd:~ # sudo zfs create -o compression=zstd \ 
                -o recordsize=1M \ 
                -o xattr=sa \ 
                -o atime=off \ 
                zroot/seaweeddata 

Install SeaweedFS Binary:  

umair@freebsd:~ # fetch https://github.com/seaweedfs/seaweedfs/releases/download/3.81/freebsd_amd64.tar.gz 
tar -xzvf freebsd_amd64.tar.gz 
sudo mv weed /usr/local/bin/ 

Verify version: 

umair@freebsd:~ # weed version 
version 2TB 3.81 39c1dde9df533475e53ce1d6f07996490543fe1b freebsd amd64 

Create a FreeBSD rc.d Startup Script: 

To make SeaweedFS act like a real service that starts at boot, you need an rc.d script. 

Create the script:  

#!/bin/sh -e 
# PROVIDE: seaweedfs 
# REQUIRE: LOGIN cleanvar 
# KEYWORD: shutdown 

. /etc/rc.subr 

name="seaweedfs" 
rcvar="seaweedfs_enable" 
seaweedfs_user="root" 
# 'server' mode runs Master, Volume, and Filer in one command 
command="/usr/local/bin/weed" 
command_args="server -dir=/zroot/seaweeddata -s3 -filer=true &" 

procname="/usr/local/bin/weed" 

load_rc_config $name 
run_rc_command "$1" 

Make it executable and enable it: 

umair@freebsd:~ sudo chmod 755 /usr/local/etc/rc.d/seaweedfs 
umair@freebsd:~ echo 'seaweedfs_enable="YES"' | sudo tee -a /etc/rc.conf 
umair@freebsd:~ sudo service seaweedfs start 

To verify if everything is working, check the listening ports: 

umair@freebsd:~ # sockstat -4 -l 
USER COMMAND     PID FD PROTO LOCAL ADDRESS         FOREIGN ADDRESS 
root weed       1437  6 tcp4  192.168.1.7:9333      *:* 
root weed       1437  7 tcp4  127.0.0.1:9333        *:* 
root weed       1437  9 tcp4  192.168.1.7:19333     *:* 
root weed       1437 10 tcp4  127.0.0.1:19333       *:* 
root weed       1437 15 tcp4  192.168.1.7:18080     *:* 
root weed       1437 16 tcp4  192.168.1.7:8080      *:* 
root weed       1437 54 tcp4  192.168.1.7:8888      *:* 
root weed       1437 55 tcp4  127.0.0.1:8888        *:* 
root weed       1437 56 tcp4  192.168.1.7:18888     *:* 
root weed       1437 58 tcp4  127.0.0.1:18888       *:* 
root weed       1437 64 tcp4  192.168.1.7:8333      *:* 
root weed       1437 65 tcp4  127.0.0.1:8333        *:* 
root weed       1437 66 tcp4  192.168.1.7:18333     *:* 
root weed       1437 67 tcp4  127.0.0.1:18333       *:* 
root syslogd    1258  7 udp4  *:514                 *:* 
Umair@freebsd:~ # 

Linux vs FreeBSD: Choosing Your Platform 

Choosing between Linux and FreeBSD for SeaweedFS deployments is less about ideology (at least it should be) and more about operational priorities. 

Linux environments generally emphasize orchestration flexibility and horizontal scalability. Container ecosystems are mature, widely supported, and well integrated into cloud-native tooling. Kubernetes integration, automation tooling, and third-party observability stacks are usually easier to deploy on Linux. 

FreeBSD environments prioritize efficiency, predictability, and deep storage integration. Jails remain remarkably lightweight compared to many container environments and ZFS integration is exceptionally mature and you tend to feel more cohesive because the operating system and filesystem evolve together rather than as loosely coupled layers. 

From a pure storage efficiency perspective, FreeBSD deployments can be extremely compelling but Linux offers broader ecosystem compatibility. 

Wrapping Up  

OpenZFS and SeaweedFS complement each other. SeaweedFS reduces object storage overhead through efficient append-oriented design, while ZFS provides the integrity guarantees, compression capabilities, caching behavior, and storage flexibility required for reliable long-term operation. The result is an object storage platform that scales cleanly without inheriting the operational weight associated with many traditional distributed storage systems. 

Whether deployed through Docker on Linux or jails on FreeBSD, using SeaweedFS with ZFS provides strong integrity guarantees, efficient metadata handling, and flexible storage policies.

So if you are a organizations building scalable backup infrastructure, artifact repositories, archival systems, or private S3 compatible platforms, this pairing offers you a practical balance between throughput, and durability. It avoids unnecessary abstraction layers while still supporting large scale object workloads. 

Klara Systems works with teams that design and operate this kind of infrastructure, providing consulting, architecture design, ZFS expertise, and FreeBSD based systems engineering for production storage platforms where reliability and clarity of design matter.