Half-duplex vs Full-duplex: Half-duplex can be a serious performance bottleneck; read and write functions are not executed simultaneously since only one lane or direction can be used for transferring data. Servers with multicore processors and plenty of RAM will be waiting for data transactions (reads or writes) to complete, resulting in an underutilization of compute resources.

Cost: While SATA SSDs are relatively low cost to purchase, you lose those cost savings in performance and underutilization of other components. While NVMe has a higher purchase cost, it provides other cost gains: reduced power consumption and maximum throughput that doesn’t bottleneck other purchased components.

Purchase Availability: Storage technology is definitely heading towards NVMe and the pricing for NVMe SSDs is starting to reflect that. As you might expect from any emerging technology, there are a number of competing formats, including the M.2 format popular with personal devices, and a number of different form factors for servers, the most popular of which is U.2. At the same time, the availability of SAS SSDs is starting to decline. For organizations that want to future-proof their storage infrastructure, NVMe will provide the most operational flexibility in the years to come.

Switch From SAS to NVMe?

If you’re planning a new storage infrastructure with hot storage, it makes sense to consider building it around NVMe. You’ll have a future-proof build that provides excellent performance now. If you already have an existing storage infrastructure, it’s probably running SAS. SAS connections and backplanes understand SATA, meaning a mix of SAS and SATA disks can be used in the same drive bay. SATA disks can easy be swapped out for SAS disks or SSDs, with no other infrastructure changes required.

Unfortunately, that same ease does not apply when upgrading to NVMe. Differing hardware technologies mean you can’t just swap a SAS drive for an NVMe one. NVMe drives often do not require an HBA, or require HBAs that support the NVMe protocol. Additionally, they may require upgrading existing backplanes to get enough NVMe ports.

Which begs the question: “when does it make sense to invest in an NVMe infrastructure and phase out an existing SAS infrastructure?”. You will definitely need to balance the need for performance gain with cost. Taking a close look at the following will both help in the decision-making process and identify areas that could be improved while planning for the new infrastructure:

• performance analysis: where are the current bottlenecks? Will higher throughput resolve these or uncover bottlenecks in other components that will also need an upgrade?
• layout: does the current layout adequately separate hot and cold storage? Are there sufficient redundant paths? Are the current RAIDZ choices appropriate?
• inventory: which ports are available on the backplanes? Are there older backplanes reaching EoL? Are the sizes and number of current storage drives sufficient? Is there sufficient room in the chassis?

Some recently addressed NVMe limitations can also help you make an informed decision.

NVMe Form Factors

The older NVMe M.2 form factor is similar to memory sticks that slide parallel into M.2 slots. This form factor, plus the fact that it is not hot-swappable and only supports 3.3 V power, pretty much made NVMe a no-go as a SAS replacement in the enterprise.

The new NVMe U.2 form-factor changes that. This form factor:

• is the same size as most SATA SSDs
• uses a cable, allowing it to ride separately from the board so that the extra heat source isn’t laying directly on the board; unlike SATA cables, U.2 cables carry both power and data
• has increased speed and can make use of up to four PCIe lanes and two SATA lanes
• has larger capacities
• can use 3.3 V or 12 V power
• is hot-swappable
• has dual-port support

So, pretty much anything that SAS can do, NVMe U.2 can do faster. The current disadvantage is what one would expect of a recent technology advance: the expense and availability of U.2 backplanes. That new technology premium will decrease over the next few years as costs decrease and availability increases.

Dual Port and High Availability

You may have noticed that one of the features provided by U.2 is dual port, meaning that the drive has two independent physical channels. This can be used to maintain I/O to and from each connection independently and simultaneously. It can also be used to provide redundancy.

Most of today’s high availability storage solutions are based on dual-port SAS drives and an enclosure and HBA capable of configuring the second port to be used for redundancy, so that a disk is still available after a path or controller failure. Until U.2, NVMe did not support this type of dual-port architecture, making it unsuited for HA environments. 

If HA is a requirement for your NAS, it might make sense to wait a bit for increased availability of HA enclosures that support NVMe.

Building the Chassis for Your ZFS NAS

Before purchasing hardware, take the time to review any existing ZFS topology and plan out what that topology should look like to meet the capacity needs for the next few years. Knowing how many disks you’ll need to support that topology will impact the choice of chassis and the number of required bays. Remember, the amount of required disks includes:

• vdevs for data storage: See our article on Choosing the Right Storage Layout. A RAIDZ Capacity Calculator can also assist in determining the required number of disks.
• support vdevs: these include disks for L2ARC, SLOG, and special vdevs for metadata. To learn more about L2ARC and SLOG, see our articles OpenZFS: All about the cache vdev or L2ARC and Understanding OpenZFS SLOGs. A special vdev is device dedicated solely for allocating various kinds of internal metadata, or OpenZFS allocation classes. rsync.net recently published a technical note on their observations about using special vdevs.

One more thing…

The experts at Klara have been designing NAS solutions for over two decades and have in-the-trenches experience with using new technologies and planning for technology upgrades. Reach out to us if you would like to discuss the practicalities of creating your own NAS solution.