Two Copal Machines Join the Fleet: Keys, Setup, Errands and Updates by Themselves

Abstract

Two machines with Copal's full install were added to the fleet:

• xinanjiang (192.168.8.217): an i5-7200U laptop, 4 threads, 7 GB;
• jaguaribe (192.168.8.245): two Xeon X5670s, 24 threads, 47 GB.

Copal is locked down to SSH keys, and only saskatchewan's key was on them. Paul's own key reached them through saskatchewan, with no private key needing to move. Both joined the queue as CPU lanes. jaguaribe now reads 961 seeds an hour, about 18% more for the whole fleet.

Joining exposed three faults in orrery:

1. Every setup recipe was Ubuntu's. A Copal machine had no button but power-off, and Join sent it to a recipe marked "not for it".
2. The queue knew one machine's power-off command, whitewolf's. Join had copied each command once. A rule applied after joining was never seen, so a fleet shutdown would have left five machines running.
3. The Media pane stopped at "the card ledger cannot be read" when answers.txt described a standalone machine, with no way forward.

Each is fixed. There is a Copal setup recipe, two errands on every machine (update the packages, pull every project), and a recipe that keeps a machine up to date by itself, daily and at boot. The probe now keeps the power-off command current, and the Media pane walks through the steps. All eight Linux machines are now set to update daily and have a working power-off command.

I. Objective

1. Let Paul's Mac log into the two new machines.
2. Make them fleet nodes and join them to the queue.
3. Give a Copal machine the same setup buttons an Ubuntu machine has, and make every button in the Machines view lead somewhere.
4. Add errands for every machine: update the packages, pull the projects.
5. Keep every machine up to date by itself, daily and after a boot.
6. Turn the Media pane's dead end into steps a person can follow.

II. Materials

III. The key, through the one machine that was let in

The new machines trusted saskatchewan's key only. The method used sends the Mac's public key along that trusted path:

ssh user@192.168.8.235 "ssh user@192.168.8.217 'cat >> ~/.ssh/authorized_keys'" < ~/.ssh/id_ed25519.pub

The Mac pipes its public key to saskatchewan. Saskatchewan logs into the new machine with its own key and appends what it was given. After that the Mac logs in directly.

Three attempts failed before this one, each on how a command was typed, not on the lockdown:

• the loop was run on saskatchewan, so it tried to log into itself;
• a pasted line broke after cat >>, so the remote shell received an unfinished command;
• the two addresses were swapped, so the outer hop went to the machine that did not yet know the key.

One attempt did succeed unintentionally: the Mac's private key was copied to saskatchewan as ~/.ssh/id_mac. That copy is not needed, and it should be removed (ssh user@192.168.8.235 'rm ~/.ssh/id_mac').

The lesson for the instructions: commands pasted from a wrapped message pick up line breaks. The working forms were one line, and short: ssh -t user@192.168.8.$h doas sh orrery-recipes/fleet-auto-update.sh.

IV. Making a Copal machine a node

What a node needs. Rust, git, tmux, the treadcore checkout, ~/fleet and the power-off rule. Copal's full install has all of these but the checkout, ~/fleet and the rule. For this first pair the checkout was cloned by hand, and the two joined at once.

The recipe copal-node.sh (Make it a fleet node (Copal)) does the same on any Copal machine. Each step is checked first, so running it again is safe:

1. it installs the build tools missing from Alpine's own repositories;
2. as the user, it clones treadcore over HTTPS and makes ~/fleet;
3. it adds the power-off rule.

It changes nothing else: no service, firewall or SSH setting, so Copal's lockdown stays as it is. On both new machines it reported "the build tools are there", found the checkout, and wrote the rule.

Join's advice now names the recipe that fits the machine's system: copal-node on Alpine, fleet-node on Ubuntu.

V. The power-off command, which only one machine had

A fleet shutdown (shutdown-now) powers each machine off by the command in that machine's config.json entry. It leaves a machine without one running, and says so.

Join copied the command into that entry at the moment of joining, from the probe's reading. A rule applied after joining was never copied. When every machine was probed on 29 October, the queue held a command for whitewolf alone. Saskatchewan, boburner, columbia, calber and nasser all had working rules, and the queue would have left every one of them running.

The fix (sync_poweroff): every probe writes the command the machine's rule allows into its entry, and removes it if the rule is gone. After one probe of all machines, every machine but the Mac has its command.

VI. Errands: update the packages, pull the projects

An errand is not a state. It has no "applied", only a last run, so the recipe's state reads "ready to run" and its button says Run…, not Apply….

Pull never merges, rebases, stashes or resets. A checkout with changes of its own, no upstream, or a branch that has diverged is left as it is, and its line says so. The script is sent on stdin, so nothing is left on the machine. Its lines go into the machine's log under "Done here". jaguaribe's first pull: code/treadcore: up to date (37b6b65).

Neither restarts anything. A new kernel waits for the next boot, and the output says so.

VII. Up to date by itself, daily and at boot

fleet-auto-update.sh (Update daily and at boot) is run once as root. It installs fleet-update.sh --auto at /usr/local/sbin and schedules it:

The automatic runs hold back the NVIDIA driver and CUDA on Ubuntu. A driver upgraded under a running machine leaves its library out of step with the module loaded. Every CUDA program then fails until the boot, with "Driver/library version mismatch". Crucible's lanes would stop overnight for an update no one chose that day. The packages are held for the run and released after it. A package held before the run stays held. A by-hand Run upgrades everything.

Each run leaves one line in /var/log/fleet-update.log. The probe shows it as updated, and the daily job as daily update. --undo takes it all away.

Tests

In containers on the Mac, before any machine ran it:

The first test caught a fault. Busybox's install does not create a missing directory, and /usr/local/sbin did not exist. That turned out to be true on Copal too (saskatchewan and jaguaribe have none). Without the test the recipe would have failed on every Copal machine. It now makes the directory.

On the machines: all five Copal machines reported the daily job and the boot run, and all three Ubuntu machines enabled their timers. The Ubuntu timers fired at once, since each machine had been up longer than five minutes. Each reported "apt up to date", with the NVIDIA packages held back (65 on boburner, 49 on calber and whitewolf) and released. calber's card still reports driver 580.178.04. The Copal machines' first run is at 2:00.

VIII. Media: the steps where there was a dead end

answers.txt in ~/code/copal-alpine-linux had COPAL_FLEET empty: it described one standalone machine. The pane read that as "the card ledger cannot be read" and stopped.

It now shows the five steps, the one to take now lit:

1. the checkout (done);
2. make answers, the interview naming the fleet;
3. the image (make img-BOARD);
4. writing card 1 (make sd-BOARD, ERASE typed twice);
5. the next card (make answers-node N=2).

Start the interview runs make answers in the pane's own terminal, the one ERASE already reaches. So the questions are answered by the operator, typed there, and nothing is answered on their behalf. Show what is on file runs make answers-show first, so a standalone answers file is seen before the interview replaces it. Once a fleet is named, the card table and its existing buttons take steps 3 to 5.

Rendered with --frame, the first version cut off three lines at the right edge (the fault docs/ calls the drawing traps), and they were shortened. The interview itself has not yet been run through to the end: its answers are the operator's to type.

IX. Results

X. Discussion

A button that is absent is only honest if its absence has a way forward. The Machines view draws no button for a recipe that does not fit, and that is right: a greyed button is a promise it cannot keep. But a Copal machine then had nothing to press, and Join's own error sent the operator to a recipe the view had just said was "not for it". The fault was not the absent button. It was that no recipe fitted. Every system the fleet runs now has one.

A copy taken once is a copy that goes stale. Join copied the power-off command as a convenience, and nothing ever looked at it again. The cost stayed invisible until a shutdown would have left five machines on. The probe already read the truth every time it ran. The fix was to keep the copy in step with it, not to read it more carefully once.

An automatic update is a change no one chose that day. So the automatic runs do less than a person pressing Run. They never restart anything, and they leave alone the one package set whose upgrade breaks a running GPU. The by-hand run does everything, because a person running it is there to restart.

Still open:

• Kernel modules on Alpine. An Alpine kernel upgrade replaces the running kernel's modules. Until the boot, a module not already loaded (a USB device plugged in later) cannot load. The log says a boot is due. The fleet does not yet act on that line.
• Probe on a schedule. The Machines view shows the last update only as of its last probe, and nothing probes on a schedule yet.
• The copied private key ~/.ssh/id_mac on saskatchewan should be removed.

References

[1] Alpine Linux wiki, "Alpine Linux package management" (apk update, apk upgrade) and "cron" (busybox crond, /etc/periodic).

[2] OpenRC, "local" service: /etc/local.d/*.start at boot.

[3] systemd, systemd.timer(5): OnCalendar, OnBootSec, RandomizedDelaySec, Persistent.

[4] Debian, apt-mark(8): hold, unhold, showhold.

[5] NVIDIA, "Driver/library version mismatch": the NVML error after a driver upgrade without a reboot.

[6] orrery docs/fleet-train.md, the machines' bullets: Copal nodes, errands, daily updates, the power-off command.