Many Versions of One Brain: What the A/B Found, and Whole-Number Choices of Rewritten Functions

Abstract

Paul asked, before anything was published, for a check that the A/B testing and enum matching were finished as he described them, and then for a report on expanding the method: several functions that do the same job, written differently, some reading new senses, chosen by a weight, with only the chosen one running.

• The enum is sound. Settings::choice makes every choice a whole number in range, each of the three choices dispatches to exactly one version, and no search can blend them.
• The A/B was not finished, and the fault was serious. Every switch's A/B crashed the families check, which recorded the crash as a floor broken. No switch could ever have been promoted. The strongest result in the fleet, pings off at +1.99 a seat on two held-out reads, was being refused for it. Mended on both sides.
• The mended gate then caught a second, worse fault (§III-B): promoted as it stood, pings off would have put a brain on main different from the one measured. Mended, and pings off is now the default, c015965.
• The method's own harvest is thin so far. Of the three choices, none has beaten the defaults. ladder = hunt_first plays identical games to the default. All six model slots read lose by about 20 a seat.
• There are no rewritten functions yet. ladder reorders, roles deals, model swaps the whole brain. What Paul describes is a new kind of version, and §V sets out how to add it.

I. Objective

To answer two questions from the code and the fleet's own records rather than from the design pages: is the A/B with whole-number choices finished, and how should it grow?

II. What Was Asked For, Checked

III. The Fault

pings off was read three times and passed both held-out reads each time it reached them, and was never promoted. The planner's record said:

grove/w1656  caps=baseline, -pings  UNDER  seconds 0  families []

A floor broken takes minutes and names the family. This took no time and named none. The job's log:

panicked at scenario.rs:1165: not a number: baseline,
test result: FAILED. 0 passed; 1 failed; ... finished in 0.00s

weights_set wrote the trial as caps=baseline, -pings; the test split that on its space and read baseline, as a number. It exits non-zero both for a floor broken and for a panic, and the planner read both as UNDER. Three candidates had it: -escort, and pings off twice. Number trials and choices were never affected: 112 candidates held, and none of them contained a switch.

Three repairs:

• set_weights reads a caps line, sets it on every Commander and leaves a RuleBot alone, and a misspelt switch stops the read. Test: a_switch_is_set_on_every_commander.
• weights_set writes one word.
• A read with no family in it is ERROR, not UNDER. It is read again on the next commit, and it does not count as a verdict on the candidate.

The third matters most. The first two mend this crash; the third is why the next one will be seen.

B. The second fault: a switch promoted but not applied

The first attempt at promotion, r0061, failed a brain test, and the test was right. set_settings turned a switch off only when it saw it change from on to off. A dealt bot is made by new, then given its role's fields (c.pings = true), then its settings. With pings off in the baseline there is no change to see, so the bot goes on pinging. Every read of pings off started from an on baseline, where the change was seen. Promoted, the same brain would have been a different one. Nothing would have reported it: the next generation would have measured against a brain still pinging while its weights said otherwise.

Switches now go by state, not by change, and keep what the role gave them (a_switch_off_before_the_deal_is_off_after_it). Two tests assumed pings were on by default and now ask for them by name, by the rule in docs/brains.md §2. The suite passes with pings on and with pings off as the default. The second attempt, r0062, passed its tests and families on a branch and was pushed.

No switch had ever been promoted, so nothing had ever walked this path. The first fault hid the second.

IV. Results

Each is a pooled read against the defaults in seats a seat; the error is one standard error.

A. Choices

ladder = hunt_first is a dead version, and the cause is not yet known. Exact equality over 165 seeds on three commits cannot be chance. It does differ in the unit test, so the dispatch works; in play the reorder never changes a decision. w_posture at 0.95 gives the fixed order only a twentieth of the score, which is a likely part of why all of ladder reads near nothing, and it does not by itself explain exact equality. A census ought to flag a read of exactly 0.00 and stop asking again: the fleet has spent three reads on it.

roles has only been read on an older brain: no read on today's defaults yet.

B. Switches

Read that first line carefully before believing it. A team that stops calling enemies and answering pings gains two seats. Either the pings are costing more than they give, which is a real finding, or the score misses what they are for. The census cannot tell these apart, and the families exist to catch the second.

What promoting it changed. With pings off, the team test read "0 pings, 0 seats rallied". As the code reads, bots no longer answer a call, a person's included. A census of bots rewards this by +2 a seat; whether a person playing with bots wants it is not something the census can answer. It is one line to reverse.

C. The families check on pings off

With the fault mended, run for real on this Mac: all nine families hold. The numbers are identical to today's defaults, which raised the question of whether the switch was applied at all. So the same path was run with dodge off, and six families fell under their floors (a duel in the open from 79.8% to 40.8%, a rolling tank under a shell from 99.8% to 89.9%). The switch is applied and the gate is real.

So the families say nothing about pings: no family exercises them. They show that dropping pings breaks nothing else. The second reading above, that the score misses what pings are for, is not ruled out. Pings are for a person's team as well as for bots, and a census of bots would not see what a person loses.

V. Expanding the Method

A. The new kind of version

The three choices so far change a parameter of a decision. Paul describes something different: the same decision, implemented again by a different idea.

// weights.rs
TargetPick "target_pick" 0.0, "tactics: who to shoot (choice: nearest_scored | focus | threat_ranked | lead_time)";

// commander.rs: the one place they differ
let target = match self.settings.choice(W::TargetPick) {
    1 => self.pick_focus(obs),
    2 => self.pick_threat_ranked(obs, &self.senses_threat(obs)),
    3 => self.pick_lead_time(obs, &self.senses_motion(obs)),
    _ => self.pick(obs), // as built
};

B. The rules

1. A whole number, always. The choice is read through Settings::choice and nowhere else. A match on it has a _ arm for version 0. Weights inside a version stay ordinary numbers and are tuned as such.
2. Only the chosen one runs. The match calls one function. Its senses are computed inside that arm, never ahead of it: a version that is not chosen costs no time and reads nothing. This is also what keeps A/B honest: if versions shared a precomputed sense, adding a version could change the timing of the others.
3. Version 0 stays, unchanged. Written beside, not over. A test pins that 0 is the brain as it was.
4. Same inputs and outputs at the dispatch. One return type, so a version cannot reach into another.
5. Deterministic. The same seed plays the same game; a census's paired reads depend on it.

C. More senses for a new function

A sense for a rewritten function differs from a sense for the learned tank (docs/brains.md §4), and it is cheaper to judge: it needs no ablation, because the version that reads it is itself the experiment. Good first candidates, reading what the sim already holds:

A sense from outside what a player could see would make the brain cheat. Each must be drawn from what obs already gives that seat.

D. A/B, A/B/C/D, up to eight

Today each version is compared with the default, alone. That extends to many without new machinery, at a cost that should be stated:

• Cost. About 70 seeds a version a generation. Eight versions of five choices is about 2,800 seeds a generation.
• False passes grow with versions. Eight tries at a 2% false-pass read give one in seven a false pass at the first gate. The held-out reads and the sequential bars absorb this, and they are why it is safe to have many.
• A tournament when two pass. If versions 2 and 3 both beat 0, the last step should read them against each other, on the same seeds, and promote the winner. Today the fleet would promote one, then re-read the other on the new defaults, which gets there more slowly. A later improvement, not a prerequisite.
• Drop dead versions. Exact zero means stop asking (§IV-A).
• Interactions are found by promotion. Each version is read against today's defaults, so a promoted version changes what the others are read against. The fleet re-reads on new defaults already, and this is why a version dropped once may be worth asking again later.

E. Order of work

1. ~~Get a clean promotion through for a switch, which proves the mended gate.~~ Done, c015965.
2. Find out why hunt_first is dead, and flag exact-zero reads.
3. The first rewritten function: target_pick. It is one decision, already a scorer, cheap to read, and focus_blend shows that targeting moves results. Two versions and one new sense.
4. Then waypoint, and think_every/think_order as the backlog has them.
5. A guard in tune and spsa: refuse a choice by name, so a hand-started run cannot waste reads on it.
6. A model in the game and the families, not only the census, before a model version can mean anything beyond the census.

VI. Training Changed

Train and school runs now start from the strongest slot read on today's defaults (best_slot) unless told otherwise, which is the league's loop docs/brains.md §6 asks for; each generation was starting over before. On the last defaults that was slot 4, g5-r0039-round40, at −18.9. Reads on the new defaults start again, and until a slot is read the newest generation is used. Starting there does not make it good: every slot is far behind the rules, and nothing yet shows that a warm start closes that gap. The first run with it is the first measurement. A running school, r0059, keeps the start it had. It has not yet started a run.

VII. Discussion

The check was worth asking for. Nothing on the design page was false in what it said was built, and yet the most valuable path through it could not complete. The test that would have shown this, a switch reaching the families, did not exist, because each part had been tested on its own.

The lesson for §V: the first rewritten function should go all the way through to a promotion attempt before a second is written, so a fault like this one shows up at the cost of one version rather than eight.

Still open

• Why ladder = hunt_first plays identical games.
• roles on today's defaults.
• A family of a team answering a call, and a decision on whether bots should answer a person's ping whatever the switch says.
• Whether pings off is a real gain or a gap in what the score sees. A family of a team answering a call would test it; none exists.
• All of §V is a design. No rewritten function exists.