The formula, not the worked example: storing skills that transfer
The problem
Knowing that 7 × 8 is 56 and knowing how to multiply are two different things. The first covers exactly one question. The second makes answers you have never worked out before, including ones nobody has ever asked.
Most software memory is the first kind. An assistant works out how to bump a library version in a Rust project — edit the manifest, run the installer, run the tests — and the whole transcript gets filed away. Next time a Rust project needs the same thing, the transcript comes back and nothing has to be re-derived. This is cheap and it works, right up until the next job is a Python project. Different words, different commands, no match, and the assistant works the whole thing out again from nothing, for the fourth time, even though the shape of the job has not changed once.
What you actually want to keep is the shape: the package name goes here, the version goes there, the tests run afterwards. Normally that means training something — collect a few thousand examples, fit a model, hope it generalises. This walkthrough gets it from five examples and some arithmetic, in about forty lines. It also shows you the two places it breaks, because both are close enough to matter.
You need a running HeatherDB and numpy.
The one idea you need
Everything here is a long list of numbers — a thousand of them — standing in for a thing. A word, a command, a whole situation: each one gets its own list. Two lists made for two different things point in unrelated directions, which in a thousand dimensions means they are as good as strangers.
Three operations do all the work, and none of them is complicated.
Gluing. Take two lists and combine them into a third that means these two
go together. Glue the list for "installer" to the list for "pip install" and
you get a single list meaning the installer is pip install. The engine calls
this bind.
Piling. Add several lists into one heap. The heap resembles every list that
went into it, a bit, and nothing else at all. Pile up "the installer is pip
install", "the tester is pytest" and "the version marker is ==" and the heap is
a description of a Python project. The engine calls this bundle.
Asking. Hand a glued pair one of its halves and it gives you back the
other, slightly smudged. Ask the Python heap "what is the installer?" and you
get something close to "pip install". This works on heaps too, which is the
useful part. The engine calls this unbind.
The smudging matters, so it needs a name: confidence is how strongly the answer that comes back resembles the real thing it should be. A confident answer is a clean one. A low-confidence answer is the memory telling you it is guessing, and later on that turns out to be the most valuable number in the whole exercise.
Here is the move the rest of the post rests on. Take one worked example — the description of a Python project, and the recipe that fixed it — and ask the recipe what the description was. The word "pytest" appears on both sides: it is in the description of the project and it is in step 2 of the recipe. Asking cancels it out, the way a common factor above and below a fraction line disappears. What survives is not a Python anything. It is the wiring: whatever the tester was, it goes in step 2.
Do that to five examples and pile the five results together. Each example's own words have already cancelled themselves out, so what reinforces is only what they share — the wiring. The heap is a formula with empty slots.
Step 1 — what a stored example gets you
Six ecosystems, one job: bump a dependency. Hold out Elixir, keep the other five. First, the ordinary approach — find the closest job you have already solved and replay it.
best = max(TRAIN, key=lambda e: cos(situation(e), situation(HELD)))
STEP 1 — the closest thing you have already solved
nearest stored job to elixir: node, and only 0.046 alike
replaying it gives: ['npm-install', 'npm-test', 'caret']
the right answer is: ['mix-deps-get', 'mix-test', 'tildegt']
Nothing is close: 0.046 out of a possible 1.0, which is the number you get for two unrelated things. And it could not have gone any other way — the best a stored example can hand you is some other ecosystem's commands. Every slot is wrong.
Step 2 — what the formula gets you
Same five examples, but pile up their wiring instead of their contents, then apply the pile to the Elixir description.
shape = pile([ask(solution(e), situation(e)) for e in TRAIN]) # five wirings, added up
answer = pair(shape, situation(HELD)) # run it forwards
STEP 2 — the shape, derived from five jobs, applied to a sixth
step 1 -> mix-deps-get confidence 0.209 correct
step 2 -> mix-test confidence 0.174 correct
step 3 -> tildegt confidence 0.242 correct
All three steps, filled with Elixir's own commands, from a formula built
without a single Elixir example in it. The formula never knew what
mix-deps-get was. It knew that whatever was named as the installer belongs in
step 1, and the Elixir description supplied the rest.
Step 3 — how many examples you need
STEP 3 — how many worked examples you need
examples steps recovered on a job never seen
1 83%
2 96%
4 100%
8 100%
16 100%
23 100%
Four. From four it is perfect, and staying perfect out to twenty-three costs nothing — extra examples only cancel their own specifics harder, so they sharpen the shared wiring rather than crowding it. This is the opposite of how a cache behaves, and it is worth pausing on: you do not need a corpus, you need a handful, and adding more is free.
Step 4 — the two things that break it
The first limit is how long a recipe can be. Sixteen examples throughout, only the number of steps changing:
STEP 4a — the first limit: how long a recipe can be
steps in recipe recovered (16 worked examples)
2 100%
3 100%
4 100%
5 98%
6 94%
8 61%
Perfect to four steps, then a slide, and by eight the formula is losing four answers in ten. The cause is crowding: everything is packed into the same thousand numbers, and the mess each slot makes for the others grows faster than the slots do. The rule of thumb is the square root of a thirty-second of however many numbers you have: about five clean slots for a thousand numbers, about eleven for four thousand. Split a long recipe into two short ones rather than paying for a bigger memory.
The second limit is how much a single pile can hold, which is a different number. Here is a plain pile of unrelated facts — keys glued to values, no shared shape at all — looked up one at a time:
STEP 4b — the second limit: a plain pile of unrelated facts
facts in pile looked up right confidence
8 100% 0.334
16 100% 0.237
32 100% 0.170
48 96% 0.141
64 86% 0.123
Perfect to thirty-two, fraying by forty-eight. Watch the confidence column rather than the accuracy column: it is halving steadily long before any answer goes wrong, which means the pile tells you it is getting crowded well before it starts lying. The rule of thumb is a thirty-second of however many numbers you are using — thirty-two facts in a pile of a thousand numbers.
These two limits pull in opposite directions and that is the whole design lesson. A pile of facts fills up. A pile of wirings does not, because the facts cancelled before they went in.
Step 5 — when two examples disagree
The failure worth designing against is not a wrong answer. It is a confident wrong answer. Feed the formula four examples wired one way and four wired the opposite way, then ask it about a job it has not seen:
STEP 5 — when two examples disagree
eight examples that agree:
step 1 confidence 0.163 correct
step 2 confidence 0.232 correct
step 3 confidence 0.187 correct
four saying one thing, four saying the opposite:
step 1 confidence 0.165 WRONG
step 2 confidence 0.160 WRONG
step 3 confidence 0.106 correct
Two of the three answers are now wrong, so on its own the answer is worthless. The confidence gives it away: the average falls from 0.194 to 0.144, and the step that collapses hardest — 0.106 — is one of the broken ones. Contradiction does not produce a confident lie. It produces a visible shrug, because two opposed wirings partly cancel each other when you add them, and what is left is weaker than either.
That gives you the rule for running this in production. Set a floor on confidence from a handful of cases you know are clean, and treat anything under it as this family has examples that disagree, do not use it rather than as an answer. A memory that says nothing is a memory you can build on.
The whole thing
import base64, hashlib, json, urllib.request
import numpy as np
BASE = "http://localhost:6380"
AUTH = base64.b64encode(b"admin:quickstart-password").decode()
D = 1024
def vec(op, body):
"""Arithmetic the engine does on lists you hand it. Nothing is stored."""
req = urllib.request.Request(f"{BASE}/vec/{op}", method="POST",
data=json.dumps(body).encode(),
headers={"Content-Type": "application/json", "Authorization": "Basic " + AUTH})
return np.array(json.load(urllib.request.urlopen(req))["result"])
L = lambda v: [float(x) for x in v]
pair = lambda a, b: vec("bind", {"a": L(a), "b": L(b)}) # glue two into one
pile = lambda vs: vec("bundle", {"terms": [{"vector": L(v)} for v in vs]})
ask = lambda c, k: vec("unbind", {"a": L(c), "b": L(k)}) # what was glued to k?
cos = lambda a, b: float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
def sym(name):
"""A stable list of numbers for a name. Two different names come out as
unalike as two random directions in 1024 dimensions, which is very."""
s = int.from_bytes(hashlib.sha256(name.encode()).digest()[:8], "big")
v = np.random.default_rng(s).standard_normal(D)
return v / np.linalg.norm(v)
ROLES = ["installer", "tester", "separator"]
ECO = {
"rust": {"installer": "cargo-add", "tester": "cargo-test", "separator": "at"},
"python": {"installer": "pip-install", "tester": "pytest", "separator": "eqeq"},
"node": {"installer": "npm-install", "tester": "npm-test", "separator": "caret"},
"go": {"installer": "go-get", "tester": "go-test", "separator": "atsign"},
"ruby": {"installer": "bundle-update", "tester": "rspec", "separator": "tilde"},
"elixir": {"installer": "mix-deps-get", "tester": "mix-test", "separator": "tildegt"},
}
ROLE = {r: sym("role:" + r) for r in ROLES} # "the installer is …"
STEP = {i: sym("slot:%d" % i) for i in range(3)} # "step 1 of the recipe is …"
# A job description and its recipe. Same words on both sides, filed under
# different labels — that overlap is what cancels later.
def situation(e): return pile([pair(ROLE[r], sym("tok:" + ECO[e][r])) for r in ROLES])
def solution(e): return pile([pair(STEP[i], sym("tok:" + ECO[e][ROLES[i]])) for i in range(3)])
# An answer comes back smudged, so it is matched against every word we know.
WORDS = sorted({t for v in ECO.values() for t in v.values()})
BOOK = [sym("tok:" + w) for w in WORDS]
def nearest(v):
v = v / np.linalg.norm(v)
s = [np.dot(c, v) for c in BOOK]
i = int(np.argmax(s))
return WORDS[i], float(s[i])
HELD, TRAIN = "elixir", [e for e in ECO if e != "elixir"]
print("STEP 1 — the closest thing you have already solved")
best = max(TRAIN, key=lambda e: cos(situation(e), situation(HELD)))
print(" nearest stored job to %s: %s, and only %.3f alike" %
(HELD, best, cos(situation(best), situation(HELD))))
print(" replaying it gives:", [nearest(ask(solution(best), STEP[i]))[0] for i in range(3)])
print(" the right answer is:", [ECO[HELD][r] for r in ROLES])
# Ask each recipe what its own job description was. The shared words cancel and
# the wiring survives; five wirings added together are the formula.
print("\nSTEP 2 — the shape, derived from five jobs, applied to a sixth")
shape = pile([ask(solution(e), situation(e)) for e in TRAIN])
answer = pair(shape, situation(HELD))
for i in range(3):
word, conf = nearest(ask(answer, STEP[i]))
print(" step %d -> %-14s confidence %.3f %s" %
(i + 1, word, conf, "correct" if word == ECO[HELD][ROLES[i]] else "WRONG"))
What to change for your own jobs
Real text instead of made-up tokens. Swap sym for a sentence embedder and
project its output to your chosen length. Nothing else changes. The cancellation
does not care where the lists came from, only that the same thing produces the
same list on both sides of an example.
Your own roles and steps. ROLES is whatever your job descriptions can be
broken into, and STEP is however many steps your recipe has. Keep the recipe
to four or five steps for a thousand numbers; if it is longer, cut it into
stages and give each stage its own formula.
Grouping. Everything above assumed you knew which five examples belonged together. In practice you group by the job descriptions, which sit visibly closer to each other within a family than across families — that clustering works long before any wiring has been worked out.
When to use this and when not to
Use it when the jobs rhyme. Every example must have the same wiring. Five dependency bumps in five ecosystems, yes. Five unrelated support tickets, no — there is no shared shape to survive the cancellation, and you will get a heap of noise with low confidence, which is at least honest about itself.
Keep recipes short. Four steps is comfortable, eight is not, and the fix is more numbers or fewer steps, not more examples.
Read the confidence, always. It is the only thing separating "here is the answer" from "these examples disagree and I am averaging nonsense". A pipeline that ignores it will work beautifully in testing and quietly produce blended garbage the first time two of your examples contradict each other.
More depth: mnemonic, fidelity, and Store a procedure.