The path before the walking: planning by rollout
The problem
You are meeting someone across town. You do not set off in a promising direction and hope. You picture two or three routes, notice that one of them ends up at the wrong side of the river, and take a different one. You never walked the bad route; you walked it in your head and threw it away.
Software that does this normally learns it the expensive way. You run the thing in its world thousands of times, score every run, and train a set of reflexes that maps "here is the situation" straight onto "do this". The reflexes work, and they have two costs worth naming. They are silent about why — a trained reflex cannot tell you what it expected to happen. And they are welded to the goal they were trained on: a new destination means training again.
If you already have something that answers what would happen if I did this — and the previous post built one out of a memory — you do not need reflexes at all. You can picture the routes. This walkthrough does that: enumerate a handful of short move-sequences, imagine each one, keep the one that ends up nearest the goal, take its first move, and repeat. Then it does the more interesting half — getting the thing to notice when the destination lies outside anything it has ever seen, and say so, instead of producing a confident route to nowhere.
You need a running HeatherDB and numpy. Everything from the previous post is reproduced here, so this stands alone.
How picturing a route works here
The memory answers one question: given a square and a move, which square do you land on. It answers by fetching the square's note, unfolding the move out of it, and then snapping the blurry result onto the nearest square it actually knows about.
That snap has a consequence worth exploiting. Because every answer is an exact square rather than a fuzzy point, imagining three moves in a row is just asking the same question three times, and asking about the same square-and-move twice gives the same answer. So you can cache. In a 25-square room with four moves there are only a hundred distinct questions, and a planner that searches thousands of routes still only ever asks those hundred.
A plan, then, is a list of moves. Score it by imagining it and looking at where you land. Take all 64 possible three-move sequences, imagine each, and keep the best. Then throw away everything except its first move and plan again from the new square — which sounds wasteful and is what stops a stale plan from being followed off a cliff.
Every prediction also comes with a second number: how strongly the memory recognised its own answer, the overlap between the blurry result and the square it snapped to. Call it fidelity. On a move the memory has seen, fidelity is healthy. On a move into territory it has never visited, it collapses. That number is the whole of the second half of this post.
Step 1 — load a model of the room
Same 5×5 room, same wandering, same two collections: dynA holding one note
per square, stA holding the squares an answer is allowed to be.
known_all = install(learn(set(range(N * N))), "A")
print("model A knows", len(known_all), "of 25 squares")
model A knows 25 of 25 squares
Note what install does with squares that have no note: it leaves them out of
both collections entirely. A square the model never acted in is not a square it
can return.
Step 2 — score a plan by imagining it
def plan(tag, s, goal):
best = None
for seq in SEQS: # all 64 three-move sequences
cur, worst, cost = s, 1.0, 0
for a in seq:
cur, f = predict(tag, cur, a)
worst = min(worst, f) # the weakest recognition anywhere in the plan
cost += manh(cur, goal) # summed over the whole imagined walk, so
score = (-cost, worst) # arriving sooner beats arriving eventually
if best is None or score > best[0]: best = (score, seq, cur)
return best[1], best[2], best[0][1]
One line there is worth more than it looks. Scoring only the final imagined square looks natural and quietly breaks the planner: sitting still for two moves and then arriving scores exactly the same as arriving immediately, so the agent dithers next to its goal forever. Adding up the distance across the whole imagined walk breaks that tie in favour of getting there.
Step 3 — walk the room
Plan, take one move, plan again.
walk 0 -> 24: [0, 5, 10, 11, 16, 21, 22, 23, 24]
weakest recognition in each chosen plan: [0.43, 0.43, 0.43, 0.49, 0.46, 0.46, 0.46, 0.47]
20 random start/goal pairs: reached 20/20, mean 3.5 moves, shortest possible mean 3.5
Corner to opposite corner in eight moves, which is the shortest route there is. Over twenty random start-and-goal pairs it arrives every time, and its average of 3.5 moves is exactly the average shortest distance between those pairs — it is not merely arriving, it is taking the short way. No reflexes were trained. The only thing that was ever learned is what one move does.
The fidelity numbers hold steady in the 0.43–0.49 band the whole way. Remember that band.
Step 4 — a goal it has never seen
Now build a second model that only ever wandered the left three columns of the room, and never once observed a move into the right two.
left = {i for i in range(N * N) if i % N <= 2}
known_left = install(learn(left), "B")
model B knows 15 of 25 squares (the left three columns)
recognition on moves it has seen : mean 0.46, worst 0.34 (n=55)
recognition on moves it has not : mean 0.20, best 0.33 (n=45)
'go east' from squares 0..4: [0.34, 0.42, 0.04, 0.12, 0.11]
Those two lines separate cleanly and with nothing overlapping: every move the model has experience of reads at least 0.34, and every move it does not reads at most 0.33. The last line walks a single row from the left wall to the right: 0.34, 0.42, then 0.04 as soon as "go east" would step off the edge of what it knows, and 0.11-ish for the squares beyond that were never visited at all.
You can also just ask whether the destination is a place the memory has ever been, before planning anything:
is the goal a square the memory has ever been to?
model A: square 12 -> 1.00 square 24 -> 1.00
model B: square 12 -> 1.00 square 24 -> 0.06
A perfect match means the memory holds that square. 0.06 is the answer you get from a memory that has nothing resembling it — noise, essentially. That single number is the difference between a reachable goal and a fictional one, and it costs one call before you plan at all.
Wire the fidelity band in as a floor and let all three cases run:
model A, goal 24: [0, 5, 10, 11, 16, 21, 22, 23, 24] -> arrived
model B, goal 12: [0, 5, 6, 7, 12] -> arrived
model B, goal 24: [0, 5, 10, 11, 16, 21, 22, 22, 22, 22, 22, 22, 22] -> out of moves, still short of the goal
The half-blind model still navigates the part of the room it knows, in four moves. Asked for a square in the unseen half, it walks to square 22 — the nearest point in its own experience — and then stops moving. It never proposes a step off the edge, because every such step scores below the floor, so it parks at the boundary of what it knows and stays there. Combined with the 0.06 recognition on the goal, the agent has everything it needs to report "I cannot get there and here is why", rather than producing a plan-shaped object that means nothing.
The whole thing
import base64, itertools, json, urllib.request
import numpy as np
BASE, D = "http://localhost:6398", 1024
AUTH = base64.b64encode(b"admin:blog-demo").decode()
def call(path, body=None, method="POST"):
data = json.dumps(body).encode() if body is not None else None
req = urllib.request.Request(BASE + path, data=data, method=method,
headers={"Content-Type": "application/json", "Authorization": "Basic " + AUTH})
with urllib.request.urlopen(req, timeout=120) as r:
return json.loads(r.read() or b"null")
def nrm(v): return v / (np.linalg.norm(v) + 1e-12)
def vec(op, b): return np.array(call("/vec/" + op, b)["result"])
def blend(a, b): # bind: fold two patterns into one, reversibly
return vec("bind", {"a": list(map(float, a)), "b": list(map(float, b)),
"normalize": False})
def unblend(c, k): # unbind: take a known pattern back out of the fold
return vec("unbind", {"a": list(map(float, c)), "b": list(map(float, k))})
# The room: 25 squares in a 5x5 grid, four moves, walls hold you in place.
N, ACTS = 5, ["N", "S", "E", "W"]
DELTA = {"N": (-1, 0), "S": (1, 0), "E": (0, 1), "W": (0, -1)}
def world_step(s, a):
r, c = divmod(s, N); dr, dc = DELTA[a]; nr, nc = r + dr, c + dc
return nr * N + nc if 0 <= nr < N and 0 <= nc < N else s
rng = np.random.default_rng(0)
V = np.stack([nrm(rng.standard_normal(D)) for _ in range(N * N)]) # one pattern per square
role = {a: nrm(rng.standard_normal(D)) for a in ACTS} # one per move
def learn(allowed, steps=2000, seed=1):
"""Wander and fold each 'I did this here and ended up there' into the
square's own note. Transitions touching a forbidden square are skipped,
which is how the half-blind model gets made."""
r = np.random.default_rng(seed)
M = np.zeros((N * N, D)); s = min(allowed)
for _ in range(steps):
a = ACTS[int(r.integers(4))]
s2 = world_step(s, a)
if s in allowed and s2 in allowed:
M[s] += blend(role[a], V[s2]); s = s2
return M
def install(M, tag):
"""Two collections. `dyn` answers 'what happens here'; `st` is the list of
squares an answer is allowed to be. Squares never visited go in neither."""
known = [i for i in range(N * N) if np.linalg.norm(M[i]) > 0]
call(f"/collections/dyn{tag}/bulk_load",
{"addresses": [V[i].tolist() for i in known],
"counters": [nrm(M[i]).tolist() for i in known]})
call(f"/collections/st{tag}/bulk_load",
{"addresses": [V[i].tolist() for i in known],
"counters": [V[i].tolist() for i in known]})
return known
cache = {}
def predict(tag, s, a):
"""One imagined move. Returns the square it lands on and how strongly the
memory recognised the answer — the second number is the useful one."""
if (tag, s, a) in cache: return cache[(tag, s, a)]
m = np.asarray(call(f"/collections/dyn{tag}/read",
{"query": V[s].tolist(), "strategy": "fast"})["result"])
q = nrm(unblend(m, role[a])) # blurry answer
y = nrm(np.asarray(call(f"/collections/st{tag}/read",
{"query": q.tolist(), "strategy": "iterative"})["result"]))
out = (int(np.argmax(V @ y)), float(q @ y)) # snapped answer, and the match
cache[(tag, s, a)] = out # answers are squares, so they cache
return out
SEQS = [list(p) for p in itertools.product(ACTS, repeat=3)] # all 64 three-move plans
def manh(a, b):
(r1, c1), (r2, c2) = divmod(a, N), divmod(b, N)
return abs(r1 - r2) + abs(c1 - c2)
def plan(tag, s, goal):
best = None
for seq in SEQS:
cur, worst, cost = s, 1.0, 0
for a in seq:
cur, f = predict(tag, cur, a)
worst = min(worst, f) # the weakest recognition anywhere in the plan
cost += manh(cur, goal) # summed over the whole imagined walk, so
score = (-cost, worst) # arriving sooner beats arriving eventually
if best is None or score > best[0]: best = (score, seq, cur)
return best[1], best[2], best[0][1]
known_all = install(learn(set(range(N * N))), "A")
print("model A knows", len(known_all), "of 25 squares")
s, path, worsts = 0, [0], []
for _ in range(12):
if s == 24: break
seq, end, worst = plan("A", s, 24)
worsts.append(round(worst, 2))
s = world_step(s, seq[0]); path.append(s) # take one move, then replan
print("walk 0 -> 24:", path)
print("weakest recognition in each chosen plan:", worsts)
r = np.random.default_rng(3)
wins, lens, shortest = 0, [], []
for _ in range(20):
st, gl = (int(x) for x in r.choice(25, size=2, replace=False))
shortest.append(manh(st, gl)) # the baseline: the short way
s, n = st, 0
while s != gl and n < 20:
seq, _, _ = plan("A", s, gl); s = world_step(s, seq[0]); n += 1
wins += s == gl; lens.append(n)
print(f"20 random start/goal pairs: reached {wins}/20, mean {np.mean(lens):.1f} moves, "
f"shortest possible mean {np.mean(shortest):.1f}")
# The half-blind model: it only ever saw the left three columns.
left = {i for i in range(N * N) if i % N <= 2}
known_left = install(learn(left), "B")
print("\nmodel B knows", len(known_left), "of 25 squares (the left three columns)")
inside = [predict("B", s, a)[1] for s in known_left for a in ACTS
if world_step(s, a) in left]
outside = [predict("B", s, a)[1] for s in range(25) for a in ACTS
if s not in left or world_step(s, a) not in left]
print(f" recognition on moves it has seen : mean {np.mean(inside):.2f}, "
f"worst {np.min(inside):.2f} (n={len(inside)})")
print(f" recognition on moves it has not : mean {np.mean(outside):.2f}, "
f"best {np.max(outside):.2f} (n={len(outside)})")
print(" 'go east' from squares 0..4:", [round(predict("B", s, "E")[1], 2) for s in range(5)])
def recognises(tag, s):
a = call(f"/collections/st{tag}/analyze",
{"query": V[s].tolist(), "strategy": "fast"})["activated_locations"]
return a[0]["similarity"] if a else 0.0
print("\nis the goal a square the memory has ever been to?")
for tag in ("A", "B"):
print(f" model {tag}: square 12 -> {recognises(tag,12):.2f} "
f"square 24 -> {recognises(tag,24):.2f}")
FLOOR = 0.34 # the worst recognition seen on a move the model actually knows
def guarded_walk(tag, s, goal, budget=12):
path = [s]
for _ in range(budget):
if s == goal: return path, "arrived"
seq, end, worst = plan(tag, s, goal)
if worst < FLOOR:
return path, f"refused: weakest step reads {worst:.2f}, below {FLOOR}"
s = world_step(s, seq[0]); path.append(s)
return path, "out of moves, still short of the goal"
print()
for tag, g in (("A", 24), ("B", 12), ("B", 24)):
p, why = guarded_walk(tag, 0, g)
print(f"model {tag}, goal {g}: {p} -> {why}")
What to change for your own problem
The scoring function is the only place your goal lives. Here it is distance-to-a-square. Swap in "how much did this cost", "did this end in a state I am allowed to be in", "how many of my conditions does this satisfy" — the planner does not change, and neither does the memory. That is the practical difference from trained reflexes: a new goal is a new line of scoring, not a new training run.
Set the floor from your own measurement, not from this post. Run the two lines that print recognition inside and outside known ground on your own data and take the number where they separate. Here that was 0.34; on a bigger or more crowded memory it will be different, and if the two ranges overlap then your fidelity is not yet a reliable guard and you should say so rather than guess a threshold.
Sixty-four sequences at three moves is a choice, not a law. Longer plans search more but each extra move is another imagined step to be wrong in. Since the answers cache, going wider is usually cheaper than going deeper.
When this works
When the memory covers the ground you intend to plan over. A plan is only meaningful inside experience. The good news is that this is measurable rather than a matter of faith — that is what step 4 is.
When you re-plan every move. Taking a whole three-move sequence and executing it blind means acting on a picture that is two moves stale.
When something has explored first. The half-blind model is not a pathological case, it is the normal state of any agent that has not been everywhere yet, and planning consumes exactly the coverage that exploring produces. The next post is the other half: an agent that uses the same recognition number, with the sign flipped, to decide where to go next when nobody has told it what it wants.
More depth: plan, rollout, fidelity and Build a world model.