Shared Storage for the Fleet: What the Disks Are, the One Way Through, and the NVMe for a Small Ceph

Abstract

Paul asked for shared storage across the fleet:

• a small Ceph cluster that every machine contributes a little to;
• direct shares of the ZFS and btrfs pools already there;
• a way in for the Mac, the Asustor NAS and a virtual machine;
• all of it driven from orrery, "like a patch bay to ensure and monitor and add connectivity of disk".

The fleet was surveyed first, as the user, with no root and nothing changed:

• 9 machines, 45 disks and 20 Ceph candidates, among them twelve disks in two btrfs pools that are not mounted (calber's six SATA SSDs, whitewolf's six 1 TB drives).
• One ZFS pool: boburner's Hexamir, 7.25 TB, 3.3 TB used.
• The 10 Gb backbone is two machines today, boburner and whitewolf (Mellanox ConnectX). jaguaribe has the same card, its link down.
• whitewolf's 10 Gb port has its own address (192.168.8.195). The fleet reaches it at the 1 Gb port's .124.

The recommendation is one way through for each job:

• NFS 4.2 with nconnect between the Linux machines: the kernel's own server, no encryption on a trusted network, several connections a mount.
• SMB only for the Mac and the Asustor.
• sshfs as the way that needs nothing installed on the server.
• Ceph (CephFS) for the small replicated space every machine keeps. It runs on the Ubuntu machines (cephadm), with one NVMe OSD on each of three nodes with 10 Gb, and the Copal machines mount it as clients.

orrery gains a Storage view (Machines → Storage):

• a survey;
• each machine's links and disks, and what each disk is for;
• a Ceph candidates page, ranked fastest first;
• a patch bay, a row a share and a column a machine, whose + mounts a share;
• Share… and Unmount… buttons. Each is staged as a root script and typed into the machine's Sessions tile, for the person's password.

Nothing in it formats a disk.

I. Objective

1. Find what the fleet's disks and links are, without root.
2. Choose the fastest, lowest-overhead way to share for this network.
3. Plan the small Ceph cluster, and the NVMe it needs.
4. Give orrery the views and buttons to share, mount and watch.
5. Say how the Asustor, the Mac, a VM and XCP-ng fit.

II. Materials

Read by tools/storage.py survey over SSH (lsblk -J, /sys/class/net/*/speed, zpool list, /proc/mounts, /etc/exports) in about 5 seconds for all eight Linux machines. The card names come from each port's kernel driver (mlx4_core is Mellanox ConnectX, r8169 Realtek, e1000e Intel). Alpine's lsblk does not report filesystems without root, so a Copal disk with no visible filesystem is marked "root's blkid says for sure".

III. What the survey found

Two btrfs pools are idle. On calber and on whitewolf, all six disks report btrfs, and none is mounted. Each set is most likely one six-device btrfs filesystem, which would hold about 2.8 TB of SSD and about 6 TB of HDD before any redundancy. They may hold data, so they are marked idle pool (amber), not candidate. The first step is to mount them and look (btrfs filesystem show, as root), not to wipe them.

The ZFS pool is healthy and half full. Hexamir on boburner: 7.25 TB, 3.32 TB used, ONLINE, on the 10 Gb machine. It is the fleet's best existing share source.

jaguaribe has eight unclaimed disks. Six 2 TB SAS drives and two 480 GB SSDs hold partitions but nothing mounted. They are the largest pool of free capacity: 12 TB raw on HDD, 0.96 TB on SSD.

saskatchewan's flash is mounted but empty. It has 3.7 TB free of 4 TB on one NVMe, 932 GB of 1 TB on another, and 1.9 TB of 2 TB on a SATA SSD. They are mounted under Windows-style "New Volume" names, which suggests filesystems from another system. They are "in use", so they are not candidates until a person decides otherwise.

The links decide more than the disks. Speeds of one link:

On 1 Gb, any one disk outruns the network. Only boburner and whitewolf can move more than one SATA SSD's worth of data. A Ceph write with three copies sends the data twice more between servers, so on 1 Gb a replicated write tops out near 40–55 MB/s.

IV. The one way through, for each job

Why not iSCSI on the 10 Gb backbone for XCP-ng? It works: LIO (targetcli) can export a ZFS zvol from boburner. But an XCP-ng iSCSI repository is thick LVM, one host owns it at a time, and on this network it gains nothing over NFS. iSCSI is worth it when a block device must look local, such as a VM's boot disk on a host with no disk of its own.

V. The small Ceph: where it runs, and what NVMe it wants

Where the daemons run. Ceph's packages exist on Alpine, but no deployment tool works there. The Alpine wiki's own procedure is a manual bootstrap with OpenRC services, marked work in progress [1]. Ceph's standard tool, cephadm, wants systemd and containers. So:

• Servers (MON, MGR, MDS, OSD): the Ubuntu machines. boburner, whitewolf and calber have 31, 63 and 63 GB of memory. cephadm deploys and upgrades them.
• Clients: the Copal machines. They mount CephFS with the kernel client (mount -t ceph, Alpine's ceph-common). jaguaribe and saskatchewan could be OSD hosts by hand later, by the wiki's procedure.

The sizing rules [4], [5]:

• three nodes at least, an odd count of monitors;
• about 4 GB of RAM an OSD (osd_memory_target), and 1–2 GB for each monitor and manager;
• 4–6 CPU threads for each NVMe OSD;
• 10 Gb between the nodes at least.

The NVMe plan. One PCIe NVMe added to each of three 10 Gb nodes, a whole device each, BlueStore:

"Each keeps a small amount" maps to Ceph this way. Every server gives one device (or an LVM volume of one) as an OSD. The pool keeps three copies spread across hosts (failure domain: host). Every machine, server or client, mounts the same CephFS tree. A machine on 1 Gb can be a client but should not hold an OSD. Its replica traffic, and its share of recovery after a failure, would run at a tenth of the speed and hold the rest back.

What it should hold: the fleet's small shared state. That means the queue's results, census binaries, model slots, configuration and datasets that every machine reads. It should not hold bulk assets: those stay on Hexamir and the btrfs pools, shared over NFS and mirrored to the Asustor.

VI. What was built: the Storage view and its tools

tools/storage.py:

• survey: reads each machine as the user;
• status: the document the view draws, with each machine's role and the candidates ranked;
• stage-share, stage-mount, stage-unmount: copy a root script to the machine and print the command that goes into its Sessions tile.

Each mount uses the address on the server's fastest link. That puts whitewolf at .195, so its 10 Gb port carries the data.

tools/share-nfs.sh PATH NET (root):

• installs the NFS server if it is missing (Alpine nfs-utils, Ubuntu nfs-kernel-server);
• writes one export line, rw,sync,no_subtree_check,root_squash, for the fleet's /24 alone;
• starts the service at boot;
• refuses /, and --undo takes the share away.

tools/mount-share.sh KIND SOURCE MOUNTPOINT USER (root):

• nfs: nfs4 vers=4.2,nconnect=8,hard,noatime,_netdev;
• sshfs: with the user's key, allow_other, reconnect;
• smb: SMB 3.1.1, with a credentials file the person writes first.

One fstab line each, mounted now and at every boot (OpenRC's netmount, or systemd). The mount point must be under /mnt, /media or /srv, and --undo takes it away.

The Storage view (src/storage.rs, Machines → Storage):

Every change is confirmed and staged, never run from the console. Share…, + and Unmount… open a confirmation that says what will happen and on which machine. Stage it copies the script and types its command into that machine's Sessions tile, where the person types the password. That is the rule the Machines view already keeps.

Tests

The NFS server and the mounts were not run against a real kernel here, because a container has no NFS server to run. The first real run is boburner's /Hexamir, which is staged and waiting.

VII. The order of work

1. Share Hexamir (staged): boburner's /Hexamir, over NFS, mounted on whitewolf, 10 Gb to 10 Gb. Measure it with fio and nfsstat -m, with and without nconnect.
2. Cable jaguaribe's ConnectX (eth2) to the switch's SFP+ (fiber or DAC) port. That makes three 10 Gb machines. Consider moving whitewolf's fleet address to .195, so the keeper's own traffic uses the fast port.
3. Open the idle btrfs pools on calber and whitewolf: mount, look, then share them or retire them.
4. Add the three NVMe, then deploy Ceph with cephadm on the Ubuntu nodes: three MONs, an OSD each, CephFS. The Copal machines mount it as clients. Ceph buttons go into the Storage view only after the first deployment has been run by hand.
5. The Asustor mirror: nightly zfs snapshot, then rsync -aH --delete over SSH from the snapshot to the Asustor's rsync module.
6. Samba on boburner for the Mac, and an NFS SR for XCP-ng if it joins.

VIII. Discussion

The network is the storage system here. Every disk in the fleet outruns its machine's link but boburner's and whitewolf's. A design built for the disks would be built for a speed the wires cannot carry. So the choices follow the links:

• NFS, because it adds the least to each byte;
• Ceph only between 10 Gb nodes;
• the 1 Gb machines as clients, which is what their links let them be.

The most valuable finds were not disks. The finds that matter most for the backbone are three facts about links:

• jaguaribe's 10 Gb card, unplugged;
• whitewolf's 10 Gb port at an address the fleet never uses;
• saskatchewan at 2.5 Gb, where the plan assumed 10.

Each costs a cable or an address, not money.

A list, not a button, for the disks. Turning a disk into a Ceph OSD erases it. The survey cannot see filesystems on Copal without root, and two of the largest sets of disks are btrfs pools that may hold data. So the console lists candidates and says why. It does not offer to format them. The Ceph buttons come after a person has deployed the first cluster by hand and the steps are known on this hardware.

Still open:

• Filesystems on Copal disks need root's blkid. A root survey would replace "unknown".
• The patch bay knows NFS exports. Samba shares and CephFS are not yet rows.
• No mount is watched for health between surveys. A stale NFS mount (hard) hangs the processes that touch it, so the survey should report mount health.

References

[1] Alpine Linux wiki, "CEPH" (work in progress): the packages, manual monitor and OSD bootstrap with OpenRC, ceph-volume lvm create --no-systemd. https://wiki.alpinelinux.org/wiki/CEPH

[2] "NAS performance: NFS vs SMB vs SSHFS" (1 GbE, fio and rsync), segmentfault.com. https://segmentfault.com/p/1210000047069281

[3] TrueNAS forums, "Poor NFSv4 performance" (nconnect: 97 to 455 MB/s) and "NFS vs SMB network performance benchmarking", forums.truenas.com; "Performance gains after switching from SMB to NFS for TrueNAS", techenclave.com.

[4] Computing for Geeks, "Ceph homelab hardware requirements": osd_memory_target, CPU threads an NVMe OSD, 10 Gb networking. https://computingforgeeks.com/ceph-homelab-hardware-requirements/

[5] Proxmox forum, "FabU: can I use Ceph in a very small cluster?": three nodes against four, memory a daemon. https://forum.proxmox.com/posts/733185/

[6] orrery docs/copal-machines-lab-report.md: the recipes and the Sessions-tile rule the staging follows.