Ninety-Eight Percent Busy: Where the Fleet's Waiting Goes, a Round Table to See It, and a Background Job That Asks for Nothing

Abstract

Paul looked at the Pipeline's four screens on the morning of 5 November and asked whether the fleet's waiting is a backlog of CUDA work that will clean up with time, or whether it will always be there, and, if it will, whether the waiting time can be put to use: "a constant low-priority background job that can be worked on piecemeal". The machines do not wait. Over the last twelve hours the eight CPU lanes were busy 97.7% of the time (96.2% to 98.5% each) and the three GPU lanes 99.8%; no job anywhere waited on another machine. The work waits instead, in three places: the GPU islands' fifteen queued segments, a buffer the planner keeps on purpose (six segments ahead a card, one running and five queued) that will never clear and should not; the chains of the line, whose next step is asked only at the planner's next pass, five minutes apart, so the transform's thirty steps take 3.8 hours for 54 minutes of work, and a trial's path from first read to held-out verdict spends 15 of its 36 minutes between its own jobs; and the CPU lanes' last two percent, idle about a minute at a time, because the keeper's low-water trigger counts the GPU buffer and so never sees the CPU queue run dry. The bottleneck is latency on a few chains, not capacity, and each part has a remedy with an estimate: the low water counted by kind of lane (about two percent more CPU work); chains advanced when they return, not at the next pass (the transform from 3.8 to about 1.6 hours, a trial's path from 36 to about 25 minutes); the held-out read in four pooled pieces (about 17 minutes off each generation's path); and the background job Paul asked for, designed so that every piece is stamped, pooled only with its own kind, weighed by the generation it read, and depended on by nothing. A new Pipeline page, Blame, draws the machines round a table, the queue in the middle, and every wait as an arrow: today it shows no arrow between two machines, three loops of GPU segments waiting their turn, one run handed back to the planner, and every machine's help going back to the queue. Built and checked here: the page and its data. Built the same morning at Paul's word and measured live (section XI): the CPU's own low water (A) and chains that ask for their next step when they return (B), with the planner's pass cut from 105 seconds to 43; the CPU went from 95.0% busy to 99.3%, and a transform's step from 7.6 minutes to about 2.3. Paul accepted them. Section XII sets each problem beside its solution; section XIII draws the data flow as it now runs, step by step. Then, at Paul's word, the held-out read in pieces (C) and the background job, both built and live (section XIV).

Index Terms: duty cycle, utilization, queueing, Little's law, polling server, critical path, work-conserving schedule, speculative execution, pooled estimates.

I. The question

Paul, with four screenshots of the Pipeline (the Board, the Timeline, the Flow and the Queued list): "Describe the hold-up on waiting. Is this caused by the backlog of CUDA computations? This one run is going very long ... Describe if we can add other steps to improve the utilization of all GPUs and CPUs by scheduling appropriate intervals or search depth ... The question mainly is: will this clean up with time? Is it because we had an unnatural backlog, or will this waiting always be happening? If so, is there a way to use the time, the wait time, as a backup research for something?" And then: "Also create a Blame tab ... the computers point at each other, saying who they are waiting on. On their other hand, it shows their backup help, where it can reach out back to the queue to assist in small jobs of improvement, piecemeal in returnable increments."

Then, for this report: "If we can't improve the bottleneck, can we use the extra compute time on something usefull if we guess right. Like a constant low priory background job that can be worked on piecemeal. Each of these additions needs to allow discrepencies of to consolidate their start times and generation weights. We want to prevent creating circular depdencies. We want to analyze the overall work duty cycle."

Four questions, then: how busy is the fleet (the duty cycle); what is the waiting (and whether it clears); can it be shortened (the bottleneck); and can the rest be used (the background job), without double counting, without stale evidence weighing as fresh, and without a cycle in the queue's graph.

II. Background: the fleet as a queueing system

The fleet is eleven lanes. Eight are CPU machines (saskatchewan, boburner, columbia, calber, whitewolf, macmini, jaguaribe, bosporus) that run treadcore's census: a first read of a trial (55 seeds, about 2.5 minutes), read-ons that add seeds until the trial is decided (27 to 42 seeds a step), held-out reads of 400 unseen seeds for a trial past the read-on bar, the families and the promotion's tests, the line's proof against main, the transform's SPSA steps, and the builds of each new commit. Three are GPU lanes (boburner-cuda, calber-cuda, whitewolf-cuda), one card each, that run crucible's islands: an evolving population that lives on its card, advanced a segment at a time (100 generations, about twelve minutes). The keeper (fleet-queue.py) hands each free lane the first queued job it can run. The planner (treadcore.py) decides what is queued, once a pass.

Four results from queueing theory frame what follows [2]-[5].

1. The utilization law, U = X·S: a lane's busy share is its throughput times the work a job takes. A schedule is work-conserving when no lane idles while a job it could run waits. At U near 1 there is nothing left to harvest; only the order of the work can change.
2. Little's law, L = λW: the jobs waiting (L) equal the rate they arrive (λ) times how long each waits (W), for any stable queue, whatever the order. A queue that holds the same length all day is not a backlog; it is a buffer, and its W is a choice.
3. The polling server: a server that looks at its inputs every P seconds makes an input that arrives at a random moment wait P/2 on average, plus the time the look takes. The planner is such a server, with P = plan_s = 300 s.
4. The critical path: a chain of N steps, each of which can begin only when the last has come back, takes N·(h + q + w): the handoff h to ask for the next step, its wait q in the queue, and its work w. Its floor is N·w, however many machines there are; this is Amdahl's law for a chain [5].

III. Method

Everything is read from the queue's own records, nothing sampled by hand. A job's work is its span from started to done, failed, stopped or requeued in events.jsonl; a lane's busy share in a window is the sum of its spans clipped to the window. A start with no ending (one, left by the keeper that went down on 31 October) is not counted. A job's asked time is its queue file's name, the keeper's clock in milliseconds when the planner's job was added (a high-priority job's name is 10^11 ms earlier, so it sorts first; the method adds it back). A handoff in a chain is the minutes from one job's return to the next one's asking; its queue wait the minutes from asking to starting. Chains are the transform's steps (rNNNN-kNN, consecutive) and a trial's reads (its first read wNNNN, then the read-ons and held-out reads whose titles name it). Two jobs asked together (a held-out read's two halves) are not a handoff. Stages are the planner's own (stage_of). Windows: twelve hours, a day and a week to 10:00 on 5 November.

IV. Results

A. The duty cycle

Table I. Busy share by day: the mean of the lanes that ran that day, and the lane-hours lost to jobs sent back, failed or stopped.

The week had an unnatural part, and it is over. The 29th to the 31st were outages: machines going down, ssh failing, a fleet shut off for four hours on the 31st, and 31 lane-hours of work run again. From the 1st the CPU lanes run 87% to 98%, the dips being the evening of the 3rd (a promotion's worktree blocking its bundle) and the 54 idle minutes of the 4th [1]. Since the handover of 4 November [1] nothing has been sent back.

Table II. The last twelve hours, a day and a week.

Table III. Where the CPU's busy time went, last twelve hours, by stage.

Five sixths of the CPU is the search: first reads and read-ons. A day's jobs: 1,519 first reads, 1,505 read-ons, 378 GPU segments, 177 transform steps, 131 builds, 120 held-out reads, 49 runs asked for, 26 families, 24 proofs and 3 promotions. The funnel: 1,520 first reads; 652 trials read on (42.9%); 75 reach a held-out read (11.5% of those); three become generations. Twenty-seven generations landed in the week, a median 2.2 hours apart.

B. What waits, and on what

Through the morning the queue held thirteen to eighteen jobs, and every wait was one of three kinds:

1. A GPU segment waiting its turn on its own card: fifteen, five an island. Crucible keeps ahead: 6 segments asked for each island, one running and five queued, each after the one before it, because a population lives on its card and its next segment can run nowhere else. The planner tops them up as they go. By Little's law, L = 5 and λ = one segment every twelve minutes give W = 60 minutes for the last of them: a buffer, by design, that will look exactly like this every day. It covers a keeper restart or a slow planner pass with an hour of work in hand, and it does not touch a CPU lane.
2. A build waiting its turn: when main moves, every machine builds the new commit, a job a machine ahead of its others. Today this was the 07:46 gap: main moved at 07:42 and every CPU lane stood 2.5 to 3.3 minutes while the line was replayed and the builds ran (20 to 74 seconds each).
3. A run handed back to the planner: a chain whose last job came back and whose next is not asked yet. At 10:05 that was r0187, the transform, between its steps 16 and 17.

No job waited on a job another machine was running. The red arrow of Fig. 1 has nothing to draw.

C. The chains: where the minutes go

Table IV. The handoffs, medians of the day.

The planner's handoff is the polling server's P/2 plus its pass: 2.5 minutes of waiting for the next pass and about 0.7 for the pass itself predicts 3.2; the measurement is 3.1 and 3.2. A trial's reads already go to the front of the queue (they are high priority), so their queue wait is about a minute; the transform's steps are the rotation's and are not, so theirs is 2.6.

"This one run is going very long": the transform. r0187, asked at 07:56, is thirty SPSA steps over the 942 numbers of a learned override, each step one census of 24 seeds on one lane. Its steps compute for 0.8 to 3.5 minutes and start about ten minutes apart (07:59, 08:10, 08:17, 08:28, ...). By the critical path, T = 30·(3.2 + 2.6 + 1.8) = 228 minutes, 3.8 hours; r0184 took 3.4 hours (04:26 to 07:52) and r0187 is on course for 3.9. Three quarters of the transform's time is waiting between its own steps, while every machine is busy with something else.

A trial's path, from its first read to its last held-out read, takes a median 36 minutes, of which 15 are spent between its own jobs (100 trials in the day reached a held-out read).

D. The CPU's last two percent

In twelve hours each CPU lane was idle 8 to 27 minutes, in gaps of about a minute (24 gaps over a minute; one shared gap of three minutes at the 07:46 build). The cause is the keeper's trigger. It calls the planner when the queue holds fewer than low_water jobs (7), every plan_s (300 s), or when a lane has stood idle with nothing it can start for starve_idle_s (30 s) and then starve_s (60 s). The queue's count includes the fifteen GPU segments, so it is never under seven, and the CPU queue, two jobs a machine (queue_per_machine, which Paul set to 2), drains in about six minutes of three-minute reads, before the next five-minute pass. The lane that finds it empty waits for the starved path: about a minute.

V. The round table (built)

The page Paul asked for is the Pipeline's sixth, Blame. The machines sit round the edge of the page as at a round table, CPU and GPU alike, each with what it runs now, its busy share over six hours and its help: the share of that time spent on the small returnable reads (first reads, read-ons, held-out pieces, the proof, the transform's steps, the GPU segments), which is what a free lane takes back from the queue when its own line has nothing for it. The queue and its planner are the table. Four marks:

• a red arrow from one machine to another: queued work dealt to the first that waits on a job the second is running, the only real blame;
• an amber arrow to the table: a run handed back to the planner, with the minutes it has waited;
• a loop beside a machine, with a count: jobs waiting their turn on it (amber when it is its build);
• a green arrow to the table: the machine's help, the hand it reaches back with.

A queued job belongs to the machine the Timeline's forecast deals it to, so the page and the Timeline cannot disagree; the planner writes blame into plan/pipeline.json at each pass (blame_view, a quarter of a second) and the page counts nothing. Two lines above the table answer the question before the picture does: the busy shares, and the handoffs of Table IV. board-check.py holds the four kinds of wait, the hand-back and the shares to a fixed queue; the six answers of its "About this screen" are in line.json with every other page's.

VI. Improving the bottleneck: four changes, estimated

Table V. Today and the estimate, each change alone and all four.

A. The low water counted by kind (keeper; a few lines; no risk). Count only the queued jobs a CPU lane can run against the CPU lanes, and the GPU buffer against the GPU lanes, so a CPU queue running low calls the planner at once. Theory: a work-conserving schedule leaves no lane idle while work could be asked; the gaps of IV-D are the only CPU idle there is. Estimate: most of the 2.3% recovered, about 4 lane-hours and 100 first reads a day. Cost: at most a few more plan passes an hour.

B. Chains advance when they return (keeper and planner; medium). When a job of a chain comes back (a transform step, a read-on, a held-out read, a proof), the keeper asks the planner within about fifteen seconds instead of at the next pass, and a chain's next step goes ahead of new first reads. Theory: the polling server's P/2 becomes the debounce; the queue wait becomes about one job's remainder. Estimate: h from 3.2 to about 0.9 minutes, q from 2.6 to about 0.5: the transform 30·(0.9 + 0.5 + 1.8) = 96 minutes, 1.6 hours (2.4 times faster); a trial's path about 11 minutes shorter. Its floor is the work, 54 minutes for the transform. Cost: plan passes rise from twelve an hour to thirty or forty with a one-minute debounce, each about forty seconds on the Mac's one core; triggering on the transform's steps and the held-out reads alone (about ten an hour) takes most of the gain for a tenth of the passes, and is the version to build first.

C. The held-out read in four pieces (planner; small). A held-out read is one job of 400 seeds, about twelve minutes on one lane, and a passing trial needs two. Four jobs of 100 seeds on four lanes, pooled seed by seed, give the same estimate (the paired difference's mean and error over the same 400 seeds) in about 3.5 minutes. Estimate: about 17 minutes off each generation's path; the same CPU.

D. Generations a day. A generation is limited by the search's luck against the bar (held-out z 1.55 today, easing after four dry hours), not by the machines. The week's 27 generations came a mean 6.2 hours apart (the median, 2.2 hours, is short because they come in bursts); A to C take about 30 minutes of latency off each, and A adds about 2% more trials. About 8% more generations, 4.2 a day for 3.9, with the CPU used no differently: the gain is mostly in how soon a found improvement reaches the game, not in how many are found.

E. The GPUs: nothing to take. At 99.8% and with a population on each card, the islands have no slack to harvest and no other work their cards could do between segments. The look-ahead of six could be three without losing a minute (a 36-minute buffer still covers a restart), which would only shorten the Queued list.

VII. The background job: using the time "if we guess right"

Paul's idea: a constant, lowest-priority job, worked piecemeal, that soaks up whatever time the line leaves. At 97.7% busy there is little time left to soak up, about 4.4 lane-hours a day, and A takes most of it. But a background job can also take a small fixed share from the first reads, the fleet's least valuable work at the margin, if what it computes is worth more. The design, so that it can be built safely:

What it computes: held-out reads ahead of time. A trial that has had three read-ons reaches a held-out read 28% of the time (74 of 269 in the day). A background piece reads 25 unseen seeds of such a trial's held-out set, the strongest by its read-on z first. If we guess right, the trial's held-out read is already part done when it is asked for, and the generation's path is shorter by the pieces in hand. If we guess wrong, nothing is lost but the CPU: the seeds are the trial's own held-out evidence, kept and pooled if it is ever asked for later. Each piece is returnable: it stands alone and adds to every other.

Consolidating start times and generation weights. Pieces start and end at any time, across promotions, so each is stamped and pooled only with its own kind:

1. The stamp. Every piece carries its question (the trial's weights, hashed), the defaults it played against (the commit's defaults, as same_defaults compares them), the GAME_VERSION, the SCORE_VERSION, the census binary's hash and its seed block. The stamp is taken when the piece is asked for, from the build it runs, never when it ends: a piece that starts under one generation and ends under the next belongs to the first.
2. The seeds. Each piece takes the next block of a trial's held-out range (60,000,000 up, 25 seeds a block, a counter in the planner's state), so no seed is counted twice and the foreground read takes the blocks after them.
3. The pool. Pieces pool with each other and with the foreground read only when their stamps match, by the seed: the paired difference's mean and standard error over every seed in hand, so order and start time do not matter.
4. The generation weight. A piece whose defaults are today's weighs fully; one whose defaults have moved weighs nothing toward today's question and is archived with its own (the same rule the proof's pool follows [6]). There is no partial weight: a trial read against other defaults is a different question.

No circular dependencies. A background piece's only dependency is its machine's build of its commit (after_host). No foreground job may name a background job in its after; the planner reads the background pool as data when it decides, which is a read, not a wait. A lint would prove it at every pass: the queue's after graph sorts topologically, and no edge enters a background job. Because a piece is small (about 2.5 minutes) and sorts last, a foreground job never waits more than one piece for a lane.

Its budget. One piece queued for each CPU lane at most, never more; or, once A has taken the idle, a fixed share of the CPU (5% is about 230 pieces a day) taken from the first reads. Whether it pays is measurable: the minutes it takes off each generation's path against the first reads it displaces (about 230 a day at 5%). Build it after B and C, and as an experiment with a stated rule: it stays if the median path from a trial's last read-on to its held-out verdict falls by more than two of its standard errors over a week.

Other background work considered and set aside: more seeds for the line's proof (the line reads +0.02 ± 0.08 against main over 5,600 seeds: a tie, which more seeds only confirm); the families read again (minutes a day); a drift sentinel reading main against the last generation (already the proof's job).

VIII. Discussion

Is the waiting caused by the backlog of CUDA computations? No. The fifteen waiting GPU segments wait only for their own cards, which never stop, and no CPU work waits on a GPU. Will it clean up with time? The GPU list will look like this forever, by design; the CPU queue, two jobs a machine, is never long. Was it an unnatural backlog? The week before the 1st of November was; this week is steady. Will waiting always happen? Yes, and it should: a queue with nothing in it is a lane about to idle. What can change is not how full the machines are but how long the few chains that make a generation wait between their own steps.

The answer to "can we use the wait time" is therefore mostly about time, not machines: the machines' spare time is two percent, and the chains' spare time is three quarters of the transform and two fifths of a trial's path. A, B and C take that back; the background job is a way to spend a little capacity to buy a little more latency, safely, once the cheaper changes have been made.

IX. What Paul decides

Which of A to C to build, and in what order (the recommendation: A, then B on the transform and the held-out reads only, then C); and whether to run the background job of VII as an experiment once they are in. The round table is built and live. Paul chose the same morning: "Build A, then B" (section XI), and, with them built and measured, "Accepted."; then "Build C, then the background job" (section XIV).

X. Addendum: the Values page had stopped

Reading the pipeline document for this report found the Values page empty since 4 November: a value added that day (size_warn_gb, read by growth.py, not the planner) has no attribute name, and the planner asked for it by name, which failed and dropped the whole table from pipeline.json. It now reads such a value from tuning.json or its default, and all 37 values are back.

XI. Addendum, the same morning: A and B built

Paul: "Build A, then B. Commit, push."

A, in the keeper. The low water counts only the queued jobs a CPU machine could take (cpu_queued); the GPU islands' buffer no longer holds it up. Twenty seconds at least between two passes it starts (low_min_s), so a planner with nothing to add is not asked each poll.

B, in the keeper and the planner. The planner marks a transform's step and a held-out read replan (chain_marks), and a transform's step goes ahead of new trials, as a held-out read already did. When a job marked so comes back, the keeper asks the planner fifteen seconds later (replan_after_s), the returns of those seconds gathered into one pass. A trial's read-ons are left to the regular pass: one pass each for their 1,500 returns a day would take most of the Mac's core.

Checked. A new check, tools/plan-check.sh, runs a real keeper with a stub planner that writes down each time it is asked. With eight GPU jobs queued and the CPU's queue empty, the keeper asked four times in twelve seconds, each low_min_s at most; the keeper before this asked none. A step marked replan came back and the planner was asked two seconds later (replan_after_s 2, plan_s an hour); the keeper before this did not ask. board-check.py holds which jobs carry the mark. The keeper's other checks pass: the queue's buttons, the breaker, the handover and the idle flag.

Live, from 10:26. The keeper took the new code at 10:26:23 and handed over its eleven running jobs; none ran again.

Table VI. Before and after, measured.

What the estimate missed: the pass. Section VI took a pass to be about 40 seconds, from the handoff's 3.2 minutes less the polling server's 2.5. The keeper now writes each pass's length into its events, and a pass took 80 to 106 seconds. Asked sooner, the planner was running most of the time and a step's handoff waited on a pass already under way. A profile of one pass (plan/profile-next makes the next pass write plan/profile.txt) found half of it in the file system: 27,000 fetched censuses globbed sixteen times a pass (44 seconds), and every finished job's record loaded five and seven times over by two functions (25 seconds). The planner is a fresh process each pass and a fetched census does not change, so each is now read once a pass (census_dirs, done_titles, and each census's commit and one-map verdict); the outputs are the same to the byte, checked against the code before on the live queue, and a commit's game version is kept in the state between passes. A pass went from 105 seconds to 43. At that length a step's handoff is about a minute, half of it the pass itself; the thirty steps of a transform 70 to 90 minutes (as their work runs 0.8 to 1.8 minutes a step), against 3.8 hours.

What stays open. C (the held-out read in pieces) and the background job of VII. The planner now passes about once every one and a half minutes, about half the Mac's one core; a pass under 43 seconds would shorten every handoff again, and its remaining cost is spread over many small reads and 450 git calls.

XII. The solutions, accepted

Paul: "Amend the last lab report with your improvements and your solutions. Accepted."

Table VII. Each problem this report found, its solution, and where it stands.

What the solutions share. None adds a machine or takes work from one. Each removes a wait between two pieces of work that were already there: a CPU lane waiting for a planner that did not know it was empty, a chain waiting for a pass that did not know it had returned, a pass waiting on the disk for what it had already read. That is the whole of the bottleneck this report found, because the machines were already full.

XIII. The data flow, as it runs now

Each step, with where it lives and what it takes:

1. A plan pass (the Mac; tools/plans/treadcore.py plan N, a fresh process each time, one at a time). It reads the planner's state (plan/state.json), the queue's folders and every fetched result, each once a pass now, and decides what the fleet asks next: new trials from the fit and the A/B, read-ons for undecided trials, held-out reads for trials past the read-on bar (four pieces each, and the reads whose pieces are all back merged into one census first), the transform's next step, families, a promotion, the line's proof, builds when main moved, and the GPU islands' segments, six ahead a card; and last, the background job's pieces ahead of time, within its budget, after a check that nothing would wait on them or in a circle. It marks a transform's step and a held-out read's piece replan, and a transform's step high priority. It prints the jobs as JSON and writes plan/pipeline.json. About 43 seconds.
2. The queue (the Mac; QUEUE/queue/). The keeper adds each job as a file named by when it was asked; a high-priority job's name sorts first, a background piece's last. Thirteen to eighteen wait at a time, fifteen of them the GPU islands' buffer.
3. A lane takes it (the Mac; a thread a machine in tools/fleet-queue.py). A free lane claims the first job it can run by renaming it into running/HOST/, sends the job's files (the line's bundle with a build or a promotion), and starts it over ssh in a process group of its own, its pid kept on the machine. Across a keeper's own restart the job runs on, handed over.
4. The machine runs it (eight CPU machines, three GPU cards). A census at the fleet's commit, after the machine's build of it, on 55 seeds (a first read), 27 to 42 (a read-on), 24 (a transform's step) or 100 (a piece of a held-out read); or a crucible segment of 100 generations. 2.5 minutes for a first read, 0.8 to 1.8 for a transform's step, 2 to 7 for a held-out piece (by the machine), about 12 for a segment.
5. It comes back (the Mac). The lane fetches the job's out folder into results/ID/, moves the job to done/ or failed/ with its host, exit code and seconds, and writes the event. A job marked replan sets the keeper's next pass for fifteen seconds on. The lane takes its next job at once.
6. The keeper asks again (the Mac; the keeper's loop). A pass begins when a returned chain's fifteen seconds are up (B), when the jobs a CPU machine could take fall under the low water, twenty seconds apart at least (A), every five minutes, when a run is asked for, or when a lane stands idle with nothing it can start. A pass about every 1.6 minutes, each read of a returned result one pass away. Then step 1.
7. A generation (out of step 1). A trial whose read-ons decide it, and whose two held-out reads pass the ratchet's bar (held-out z 1.55 today), with the families, the trade rule and the gauntlet's floors holding, is promoted: a job, on any CPU machine, that builds the line with the candidate's numbers, runs the brain's tests, the families and the gauntlet, and sends its commit home as a bundle. It goes on the line.
8. Main and the builds (out of step 1). The line is read against main every hour, 800 seeds a read, every read of one line against one main pooled; past z 3 auto mode pushes it to main. A reader never moves the line; the saved state sets its branch and bundle right. When main moves, every machine builds the new commit, the line's bundle with it, as jobs of steps 2 to 5; a failed build is asked again once.
9. What a person sees (the console). Each pass's pipeline.json is the Pipeline's six pages; the keeper's keeper.json and events are health, with each pass's length and the last time a chain asked; flags say what needs a person, among them an idle spell. The keeper's chores, each minute, keep the Results pages and the recordings' sweep going whether a window is open or not.

No step waits on a later one, and nothing waits on a job that waits on it: the only waits are a job on its machine's build (step 4 on step 8's builds), a GPU segment on the segment before it, and the planner's reading of what came back (step 6 to step 1), which is a read and never a lock. That is the property the background job of section VII must keep.

XIV. Addendum, the same day: C and the background job built

Paul: "Build C, then the background job. Commit, push."

C, in the planner. A held-out read is asked as four jobs of 100 seeds (held_read, held_piece), on any CPU machine, each its quarter of the read's seeds: piece i plays seeds SEED0 + 100(i − 1) to SEED0 + 100i − 1. When all four are back, the next pass (B asks for it fifteen seconds after each piece returns) copies the four pieces' per-seed logs into one folder under the read's own id, writes its run.env as the whole read's, and pools them with treadcore's scripts/pool.sh, which was written for a census split over jobs (held_merge, merge_pieces). A seed plays the map of its own number, n mod k, so the pieces play exactly the games the one job would have, and the merged read is the same census. Tested on a real read: h0796's 400 seeds dealt out as four pieces and merged gave its pooled.txt back to the character, −0.37 ± 0.41 over 400 seeds. Everything that reads a held-out read (whether it passes, the candidate's evidence, the second read, the families, the found flags, the trace) reads the merged census as it read the one job; the pieces are not reads of their own, so the fit's ledger does not count a seed twice. A piece that fails is asked again, twice at most, then the read is dropped and a flag says so; a piece read on other defaults than the newest of its read weighs nothing and is asked again.

The background job, in the planner and the keeper. Each pass, after the line's own work, the planner asks held-out pieces ahead of time for the trials still reading on with two read-ons or more and a pooled z of 1.5 or more, strongest first (ahead_jobs): the pieces of the trial's first held-out read, its id and seeds reserved then. They go last in the queue (priority "low"; the keeper's low water does not count them), one queued for each CPU machine at most and 150 a day. Each is stamped when asked (the trial's weights, the commit, the game, the z it had). If the trial is later asked for its read, the read takes the reserved id and seeds, asks only the pieces not already asked, and the ones still queued go to the front, no longer background (raise_queued: taken from the lanes' sight before it is rewritten, so a piece a machine has claimed is never queued twice). A trial dropped or gone for three days is forgotten; its pieces' results stay. Every pass, before the jobs are printed, graph_lint sorts the queue's after edges with the new jobs' topologically and refuses any new job that would wait on a background job or in a circle.

Checked. board-check.py: the four pieces and their seeds; a failed piece asked again as r1; a piece on moved defaults asked again; the merge's files, run.env and pool.sh's 400 seeds; the merged read in held_pairs and the pieces out of the ledger; three failures dropping a read; the background job's choice, its one-a-lane and daily budgets, the reserved read taken later with its queued piece raised; a job after a background job and a circle both refused. plan-check.sh: a background piece sorts after the standing work and the low water does not count it. The keeper's other checks pass.

Live, from 11:24. The keeper took the new code by handover (eleven jobs, none run again). The first pass asked two background pieces; the first split read, h0801 for w10893, was asked at 11:34:05:

Table VIII. The first held-out read in pieces.

What the estimate missed. Section VI took a piece to run as fast as a whole read's machine runs a quarter of one, and estimated 3.5 minutes. A whole read was held to the three fast machines (long_needs); a piece goes to any CPU machine, and the slowest of the four sets the read's time: jaguaribe took 6.9 minutes for the 100 seeds macmini read in 2.0. From asked to merged, 7.1 minutes, against a whole read's median 9.6 minutes of work alone (228 reads in two days, 7.7 to 13.3 between the quartiles), before its queue wait. A next step, not built: pieces sized to each machine's speed, or pieces that prefer the fast machines while one is free.

The background job, honestly. It asked ten pieces in its first twenty minutes, for four trials, and filled its one-a-machine allowance; with the CPU at 99%, none had run by 11:41. That is what it was built to do: take only the time no one else wants. Its pieces earn their keep the other way too: a trial asked for its read finds its pieces already queued and moves them to the front, so the read's ids, seeds and stamps were settled before it was needed. Whether it shortens a generation's path is the experiment of section VII: the median from a trial's last read-on to its held-out verdict, over a week, against this week's.

XV. Addendum, the same afternoon: the Timeline zoomed, its milestones, and each machine's queue at the round table

Paul, in four asks: "In the queue, can we see arrows of how the job moves from one to the other? ... On highlight of a cell, highlight all the chain elements associated"; then "When I highlight, I don't see any combination of gray. Is the CUDA block used? Is there a synchronization of CUDA to game rules? ... Show separation of major milestones of generation, 'cause I don't understand when the CUDA will return. Also, maybe we can zoom the timeline in and out: months, weeks, days, day, hours, hour"; then "Describe why so many CUDA are a backlog for the GPU island. Is there an exponential growth? ... Is it limited?"; and "Add more accordions to Blame first, to show the job dependency on hover. I want to see job arrows amongst the Blame circle, with jobs shown, and a filtered job queue for each machine." All built, committed in orrery 0fcffbc and 8c8bb46. Every figure of this section is the console's own --frame render of the live pipeline.json of 13:42, but Fig. 16, which says how it was made.

A. Following a chain

The planner names the chain each job is a link of (chain_of): a trial's first read, read-ons, held-out reads and their pieces and its promotion are the trial's; a run's steps are the run's; an island's segments are the island's; a commit's builds are the commit's; a proof is its line's. The Timeline carries every job alone beside its merged bars. Point at a block and its chain lights: the rest of the hours step back, every job of the chain is framed in amber where it ran, and an arrow runs from the end of the job each link followed to its own start. A click keeps the chain lit.

B. Why the GPU rows join no chain, and when the GPU hands the game anything

No trial uses a GPU. The trials are the CPU machines' search of the brain's numbers; the GPU islands are crucible, a line of its own, evolving nets in a duel arena. An island's chain is its own segments on its own row, so a pointer on a grey block lights one row and nothing crosses it. The islands meet each other (each segment imports the others' newest champions) but meet the game by one road only:

1. A milestone. A segment's champion is examined against the ladder's newest milestone; it becomes the next only when it beats it by two standard errors with the ladder's mean not below zero (crucible_ladder_home).
2. The bridge read. The planner then reads the milestone as the Gladiator (style 17) on a fast CPU machine against the shipped one, 48 seeds each on the same seeds.
3. The ship. If it reads better by two errors, tools/bridge.py commits it into treadcore as nets/gladiator.bin (in auto mode by itself).

The last time that road was travelled: M0015 was made at 05:42 on 4 November and shipped at 10:31 (treadcore LR 138, 21fdfbd). In the 32 hours since, 502 segments have been examined against M0015 and none has passed it. Against M0015 almost every duel is drawn (about 98% of calber's), so a margin moves slowly. The page now says, on each GPU row, what its next milestone must beat and the planner's guess from the trend of its exams (cuda_return): the median margin of each hour of the last day, fitted to a line, and the hours until it would cross +2 at that climb.

Table IX. When each island would next hand the game a net, at 13:42: a guess from a trend.

Margins in standard errors of the exam's win-less-loss share; a milestone needs +2. A guess, said as one: a climb can stall, and the best of six hours (calber's −1.5) is nearer than the median. The bridge read that follows a milestone is about ten minutes of a CPU machine, and is now a link of its island's chain (chain_extra, after the segment that made the milestone), so the arrow from the grey row to a CPU row will be drawn the first time there is one.

C. The islands and the game's rules: a side channel

Synchronised, in part. Each segment is built with sixteen constants read from treadcore's rules.rs at the fleet's commit (speed, acceleration, braking, turning, gun range, shell speed, reload, armour, damage, hit radius, shells, vision, and the pillboxes' range, damage, reload and health: world_rules, world_flags); a change rebuilds the island at its next segment and raises a flag. The islands have played treadcore's rules since their first world segment at 23:03 on 3 November, and rules.rs has not changed since game 8 (19:19, 2 November), so no move has happened to show. On the Timeline the channel is a dashed grey line through the GPU rows only. Not synchronised: the economy of games 5 to 8 (ore, oil, workers, supply, trade) and the score of 4 November (score 1) are not in the islands' world, where a net is fitter for winning duels. And a segment's build flags are fixed when it is queued, so a change of rules reaches an island after the up to five segments already queued: an hour, at today's lengths.

D. The GPU's queue is a buffer, and it does not grow

Section IV-B gave the reason; Paul asked again whether it grows. It cannot: each island keeps exactly ahead = 6 segments, one running and five queued, each after the one before it, and every pass tops the island up to six and no further (crucible_jobs). At 13:42 that was 18 of the 29 jobs on the round table, six an island. It looks long because six was chosen on 28 October, when a hybrid segment took about 2.6 minutes and the planner passed every five: six were a quarter of an hour in hand. Since the world build a segment is 100 generations of a world about ten times the cost, 8.6 to 13.4 minutes, so six are 52 to 80 minutes of forecast, where a pass now comes every thirty seconds or so. It costs no GPU time; it costs the lag of section C, and a picture that looks like a backlog. A next step, not built: the buffer sized in minutes, about twenty, two segments today.

E. The Timeline zoomed, and the milestones drawn through it

Six zooms: the hour and the six hours draw each job as before; a day, a week, four weeks and three months draw bins of the event log, five minutes, thirty, two hours and six hours wide, each column a machine's bin filled from the bottom by the share of it each stage took, the rest idle. The planner keeps the bins as the log grows (timeline_bins, plan/timeline-bins.json: five-minute bins for two weeks, hours for always, read on from the byte the last pass stopped at, about 0.6 s a pass), on the local clock, and hands the page each change of summer time (clock_shifts): the day of 1 November had 25 hours, and the log begins at 09:34 on 28 October, summer time, which the page's own arithmetic had said was 08:34. Before the log begins the rows are hatched.

Lines with tags mark what happened, three months of it (timeline_marks): a generation on the line (29), the line on main (13 landings, from treadcore's main), a candidate the tests stopped (22), the game's rules or score moving (8: games 2 to 8, and score 1), a GPU milestone (15), a net shipped into the game (1, the Gladiator) and the islands' rules (1). Where tags would collide, main, the game and a ship are placed first. pipeline.json is now written compact: with the bins it was 889 KB indented, 327 KB without.

F. Each machine's queue at the round table

The planner now writes each machine's queue with the round table (blame_view's queues): what it runs, then what the Timeline's forecast deals it, in order, each job with the job it waits on and where that job is: the one ahead of it on the same machine, its build of the newest commit, or a job another machine has, the only wait between two machines. A job no machine that is up can take is listed apart. On the page each machine is an accordion beside the table; a click on its header folds it. Point at a machine, at its seat or in its queue, and its jobs are laid out at the table between its seat and the middle, in the order they will run: a grey bracket from each to the job ahead, amber to its build, a red arrow to a job another machine has, drawn as a chip at that machine's seat, and red from any job elsewhere that waits on one of these. The rest of the table steps back. Point at one job and its chip and arrow light. A click keeps the machine or job while the pointer goes to the chips, and what it is pointing at is written in a pane under the queues, where a tooltip would have covered the arrows it explains.

At 13:42 no job waited on another machine: every wait was a machine's own turn, two or three on each CPU machine and five on each GPU island (Fig. 13).

G. Checked

board-check.py: each kind of job named to its chain; the event log's bins read on from a half-written line the same as read whole, a failed job's time as failed; a day's bins on the five minutes; the 25-hour day's ticks and its shift; the return's climb, median, best and hours from a lineage made to climb one error an hour; a bridge read linked after its milestone's segment; each machine's filtered queue and the job each waits on. The console's tests (401): a chain's links across machines and a link the planner names before the clock; a zoom's bins, its dates in summer time and its CUDA boxes read; each machine's queue and what each job says it waits on.

XVI. Addendum, the same evening: the fleet's watts, the Lineup and the Hit Parade

Paul, in three asks, his words spell-checked: "Use the assessment of CPU power, and general Gold-efficiency power supplies; use GPU power and CPU estimates. Perhaps there is a sensor, and perhaps we can have a general guess at full-tilt CPU, 'cause things like the macmini are minimal watts, where the 48-core server CPU is power hungry. Can we get general estimates, and also have each computer report its power-estimate participation to the graph as it hands it off? Perhaps tracking this, to give this a general cost per kWh. Build it into an underlay graph underneath the timeline, showing graph color from blue to white to purple to yellow to red." Then: "Add another visualization, a tab. It has all the same elements as the Timeline does, but it rearranges each chain into its own Gantt chart, with the blocks of the chain reassembled on each row, with machines' names in the job block (also showing job info). But this would unscramble the net trace of arrows, to have each dead end and each success clearly shown, and forks looking like nested accordions already open, showing arrows and sub-rows." And: "Generate a graph, a rate of type per machine, and keep the statistic in a chart: a new screen; it's like a success counter matrix, a wall of success. Approved by Von Glurt. Take pictures, add to the lab report addendum. Spell-check my quotes, and in the lab report show this history in its new form. Give it a nifty 1950s-style name for the tab; I'm at a loss what to call it. It's a timeline, but a different reorder of a timeline."

Two new tabs, named for the television of the 1950s: the Lineup (after The Lineup, 1954: every chain stands in its own row) and the Hit Parade (after Your Hit Parade, the decade's countdown of hits). Every figure of this section is the console's --frame render of the live pipeline.json of 14:56; none is constructed.

A. The fleet's watts, and what they cost

What is measured, and what is not. Every five minutes the keeper reads each box's CPU use (two reads of /proc/stat two seconds apart; top on the Mac) and each GPU's draw (nvidia-smi), with which of its lanes was running a job (tools/power.py sample, one of the keeper's chores). The CPUs' own watts are not measured: five boxes have an energy counter (Intel's and AMD's RAPL) and all five keep it for root; the two Xeon servers, of 2010, have none; and no box has a meter at the wall. So a CPU's watts are its model's (tools/plans/power.json, from its maker's figures) at the use measured. A box at the wall = (the rest of it + its CPU at idle + the CPU's use × its full tilt less idle + its GPU) ÷ its supply's efficiency, 0.88 for 80 Plus Gold (at least 87% at a fifth of its rating, 90% at half, 87% at full); the Mac mini by Apple's own figures at the wall. Until a box has six samples the model's guess stands (90% of its cores busy, a card at 60% of its limit): one two-second read had caught jaguaribe at 9%, in the last seeds of a job.

Table X. Each box at 14:56, at the wall: its full tilt, now, and its cores' busy share while it runs a job, measured.

The last 24 hours drew 30.7 kWh, $5.22 at $0.17 a kWh: about $157 a month. The price is the US average residential of 2025, a guess until Paul sets his own (Settings, Long run, power_usd_kwh).

Two findings. First, a busy machine is not a machine at full tilt: a census job runs its seeds in parallel and ends in a tail where the last few run alone, so the two 24-thread Xeon servers run a third of their threads while they work (33% and 29%), the 24-thread i7-13700F and the 16-thread Ryzen half, and calber and whitewolf nine tenths. Second, energy a job is lopsided: today the Mac mini drew 0.72 Wh a job, columbia 1.68, boburner 4.42, saskatchewan 4.72, calber 7.44, whitewolf 10.4, jaguaribe 11.2 and bosporus 13.9; a GPU segment 18 to 23 Wh. Jobs are not all the same size (the fast machines take the long held-out reads), so the ratio is a strong hint, not a measurement: sixteen to nineteen times the Mac mini's energy for a Xeon server's job. Today's earlier jobs were charged at the model's first guess, before the six samples; the figures settle as the samples go on.

Each job reports its part as it is handed off. The planner charges a job its lane's watts busy for the minutes it ran (a CPU lane its box's rest and CPU, a GPU lane its card), and keeps the day's total by lane and kind of job. On the Timeline each row's foot is its own watts, its name says its watts now, and under the rows the fleet's watts stand a column a bin, coloured as Paul asked from blue (none) through white, purple and yellow to red (full tilt). A machine off draws nothing: from the keeper finding it unreachable or powering it off until it is back, its time counts zero, and the fleet's shutdown of 31 October shows as a fall to 360 W (Fig. 18). Following a chain on the Timeline adds up its watt-hours; the round table's queues and pane say each machine's.

Not done, and Paul's to decide: the CPUs' own counters can be read by a rule as root on the five machines that have them (a change to those machines); the two Xeons can be measured only by a meter at the wall.

B. The Lineup: every chain in its own rows

The planner says how each chain ended or where it stands (chain_meta). A trial is won when it was promoted (and says when it is on main); a dead end when its promotion failed, when its held-out read did not pass, or when it was dropped after its reads; going when it is reading, in its held-out read, or a candidate waiting its promotion. A run is won when done, a dead end when failed; a build is won when every machine built it; a clone round is won when it shipped. The page stands each chain in its own rows: a head row with how it ended, and under it its jobs where they ran, each block its machine and its job; a fork (jobs of the chain that ran at once: a held-out read's pieces, a record run's shards) stands open on sub-rows of its own, as an accordion; an arrow runs from each job to the next. A red wall marks where a chain died, a green tag where it won, an amber one where it goes on, with hollow blocks for what the forecast deals it next. Three zooms: the hour and the six hours share the Timeline's clock, its milestones and the fleet's watts; each its own stretches every chain across the width, its own Gantt chart, so every block has room for its machine and job. Filters choose the kind (trials, runs, builds, islands, all) and the ending (going, won, dead ends).

The six hours to 14:56, in the new form.

Table XI. Every chain of the six hours, by how it ended.

How the 364 trials died: 343 dropped after their reads, 200 of them after one read (55 seeds), 51 after two, 40 after three, 22 after four and 30 after five to nine; and 21 whose held-out read did not pass. That is the search doing its work: a first read decides most trials by itself, and a trial that reaches a held-out read passes about one time in eleven (one won and one candidate against 21 that did not).

C. The Hit Parade: a wall of counters

The planner keeps a tally of every job as it is handed off, a day, a machine and a kind of job at a time: done, not done (failed, stopped or sent back), its minutes and its watt-hours (parade_view; the tally lives with the event log's bins and is kept as long as they are). The page is a wall: every machine a row, every kind of job a column, each counter the jobs done large, the share that came back done (green at 99% and up, amber from 90%, red below), its rate an hour and its watt-hours a job, and a lamp along its top as bright as its count against its column's most. Beside the wall each machine's rate by kind as one bar; under it a bar a day since the log began, done in green with the not done in red; over it the scoreboard, and the Von Glurt seal: approved.

Table XII. The scoreboard.

Today the Mac mini did the most: 607 jobs, 40.6 an hour, then saskatchewan 433 and boburner 330. The wall's weakest counters, since the log began, are the promotions on the Mac (25 done of 39, 64%: the faults of 3 and 4 November, sections X and XI), the held-out reads on whitewolf (22 of 28, 79%), columbia (34 of 39, 87%) and calber (49 of 54, 91%): a long read on a slower machine is the one most often stopped or sent back.

D. Checked

board-check.py: the model's watts (saskatchewan at full tilt, the Mac mini's 4 W idle), the probe read on Linux and on the Mac, the model's guess under six samples and the samples' mean at six, a GPU lane its card; the event log's off machine counting nothing, its GPU lane with it, and a job's watt-hours on its day; each trial's end said (promoted, its promotion failed, dropped, its held-out read not passing, a candidate, still reading with its forecast); the tally's windows and hours. The console's tests (403): the watts read and coloured blue to red; a chain's end and its forecast read; the tally read; both pieces of a fork following the same read-on. Live: the keeper samples every five minutes; the planner writes the watts, the chains and the tally at every pass.

References

[1] orrery, Fifty-Four Idle Minutes: the Line Replayed Twice, Three Guards, and Restarting Orrery Without Losing Work, lab report, 4 November 2026, sections X and XI.

[2] J. D. C. Little, "A proof for the queuing formula L = λW", Operations Research, vol. 9, no. 3, pp. 383–387, 1961.

[3] L. Kleinrock, Queueing Systems, Volume I: Theory. New York: Wiley, 1975.

[4] H. Takagi, Analysis of Polling Systems. Cambridge, MA: MIT Press, 1986.

[5] G. M. Amdahl, "Validity of the single processor approach to achieving large scale computing capabilities", in Proc. AFIPS Spring Joint Computer Conference, 1967, pp. 483–485.

[6] orrery commit 228c30c, "The line's proofs pooled", 4 November 2026.

[7] The queue's events.jsonl, done/ and plan/pipeline.json, 29 October to 5 November 2026.