HeatherDB

Search documentation

Loading the documentation index…

Download
← Research

Sleeping on it: what idle replay does to a memory

agentsmemory

The problem

Practise an instrument for an hour and you get better. Sleep, come back the next morning without touching it, and you are better again. Nothing was added in between. Something was sorted out.

You have a store that has been taking in data for a week — sensor readings, support tickets, page views, whatever recurs in your business. It is bigger than it feels like it should be. Most of what came in was the same handful of situations over and over, with small variations, and you would like to know whether the store has noticed that or has just kept everything. This matters for two reasons: a store that has noticed is smaller and faster, and a store that has noticed has told you something about your data that you did not have to go looking for.

The usual way to find that out is to take the data somewhere else — export it, cluster it, pick a number of clusters, look at the result, and if you got the number wrong, do it again. This walkthrough does not export anything. You ask the store to go back over what it is already holding, and you read off how much of it turned out to be the same thing twice.

You need a running HeatherDB and numpy.

What "going back over it" actually means

The store keeps entries. Each new reading either lands on an entry that is already close to it — reinforcing it, nudging it slightly — or, if nothing is close enough, starts a new entry of its own.

That makes the shape of the store depend on the order things arrived in. The first few readings claimed entries in a wide-open space and got them cheaply. Later readings arrived into a crowded neighbourhood and made do with whatever was nearest. Nothing ever goes back and fixes this, because a write only ever touches the handful of entries it landed near.

Going back over it is exactly what it sounds like: every entry the store holds is presented to the store again, as though it were arriving for the first time. Three things fall out of that one move. Entries drift toward the middle of what they actually represent, so they get sharper. Splits that could not happen under the old crowding get their chance. And entries that have quietly drifted into the same place get folded into one.

The last of those is the measurement. The number of entries is not a setting anyone chose — it is an outcome. If it falls, the store found repetition it had not previously spotted. If it stops falling, the store has extracted everything currently available in the data. That number is what this walkthrough watches.

Step 1 — put in a week's worth

Six situations that genuinely recur, each seen two hundred times with a different wobble, plus three hundred one-off readings that mean nothing. No labels anywhere; the store is not told there are six of anything.

for i in range(1200):
    k = i % 6
    rows.append(unit(PATTERNS[k] + 0.4 * unit(rng.standard_normal(D))))
for i in range(300):
    rows.append(unit(rng.standard_normal(D)))     # noise, belongs to nothing

Grading is six probes the store has never seen, one drawn near each real situation. A probe is scored by how closely what comes back matches the situation it came from — 1.0 would be perfect.

after ingest:  1500 readings stored as 380 entries, recall 0.9954

Fifteen hundred readings, three hundred and eighty entries. It has already compressed roughly four to one on the way in, and it answers probes almost perfectly.

Step 2 — copy the database before you touch it

Going back over the data rewrites it in place. There is no dry run and no undo, so take a copy first. This is safe with the engine running and serving traffic.

heather --data-dir /tmp/heather-blog snapshot create --db fleet

✓ snapshot: /tmp/heather-blog/snapshots/fleet-1788791995

Do this even the first time, and especially the first time. Everything in the following steps is a write.

Step 3 — go back over it, five times

call("POST", f"/db/{DB}/dream?collections={COL}")

Note the ?collections= on the end. Without it every collection in the database gets reworked, including ones you were not thinking about. Name what you mean.

dream 1:  380 ->  368 entries, folded  20, recall 0.9943
dream 2:  368 ->  347 entries, folded  38, recall 0.9919
dream 3:  347 ->  334 entries, folded  39, recall 0.9902
dream 4:  334 ->  322 entries, folded  47, recall 0.9871
dream 5:  322 ->  317 entries, folded  64, recall 0.9855

Three hundred and eighty entries down to three hundred and seventeen, a seventeen per cent reduction, while the ability to answer probes goes from 0.9954 to 0.9855 — a loss of one per cent. The store is saying the same thing with fewer entries, which is the definition of having found structure.

Two details in that table are worth reading properly. The folded column counts far more foldings than the entry count drops, because the pass both folds entries together and splits others apart; the net is what matters. And the drop tails off — 12, 21, 13, 12, 5 — which is the store running out of repetition to find. When it reaches zero and stays there, you are done, and no amount of further passes will help.

Step 4 — check the report, not the request

There is a second thing the pass can do, asked for with ?mode=ladder. Before using it, know this: an unrecognised mode does not fail. It quietly runs the ordinary pass instead.

call("POST", f"/db/{DB}/dream?mode=Ladder2&collections={COL}")

?mode=Ladder2 -> touched 'week1', thresholds 0.00/0.00  (a replay pass, not a ladder)

A capital L and a stray digit, no error, and a database that has been rewritten in a way you did not ask for. The report is where the truth is: a plain pass reports the collection you named and thresholds of zero, and there is nothing else to distinguish it by. Read the response every time.

Worth knowing alongside that: the pass runs on request whether or not the database has this feature switched on in its configuration. A manual trigger does not check the opt-in. The database in this walkthrough never enabled anything, and every call above worked.

Step 5 — the ladder

Spelled correctly, ?mode=ladder does something different in kind. Rather than tidying up how cases are stored, it changes what is stored.

Read each entry as an episode: the entry itself is a situation, and what the store learned alongside it is what happened. Ask what happened what its situation was, and the parts that appear in both cancel out. What is left is not this case — it is the rule the case was an instance of. Do that for every episode, file the rules by the situations they came from, and you have a second collection whose contents are rules rather than cases.

call("POST", f"/db/{DB}/dream?mode=ladder&levels=2&collections={COL}")

?mode=ladder  -> wrote 'week1__L1', 307 entries, thresholds 0.57/0.99
?mode=ladder  -> wrote 'week1__L2', 306 entries, thresholds 0.57/0.00
collections now: ['week1', 'week1__L1', 'week1__L2']

The output is real collections on disk, marked __L1 and __L2. The second rung is the first rung fed back through the same machinery — rules about rules — which works because the output of a rung has the same form as its input. Collections carrying the __L marker are skipped as inputs, so the ladder cannot climb itself.

The two thresholds are the part to look at. Nobody set them. Each rung works out its own boundaries from its own contents: how alike two situations must be to count as the same context (0.57 here), and how well two rules must agree to be treated as the same rule (0.99 on the first rung). The second is what stops the obvious failure — two cases that look alike but did opposite things averaging into a rule that is true of neither. Looking alike only nominates candidates; agreeing is what joins them. Disagreement starts a new family instead, which means an unfamiliar rule is recognised as new even when the situation is familiar.

The whole thing

import base64, json, subprocess, urllib.request
import numpy as np

BASE, DB, COL, D = "http://localhost:6380", "fleet", "week1", 1024
AUTH = base64.b64encode(b"admin:quickstart-password").decode()

def call(method, path, body=None):
    req = urllib.request.Request(BASE + path, method=method,
        data=None if body is None else json.dumps(body).encode(),
        headers={"Content-Type": "application/json", "Authorization": "Basic " + AUTH})
    return json.load(urllib.request.urlopen(req))

unit = lambda v: v / np.linalg.norm(v)
cos  = lambda a, b: float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

call("POST", "/db", {"name": DB, "dimension": D})
call("POST", f"/db/{DB}/collections", {"name": COL})

# Six situations that genuinely recur, each seen 200 times with a different
# wobble, plus 300 one-off readings that mean nothing. This is a week of
# ingestion in miniature: mostly repetition, some noise, nothing labelled.
rng = np.random.default_rng(11)
PATTERNS = [unit(rng.standard_normal(D)) for _ in range(6)]
rows, tags = [], []
for i in range(1200):
    k = i % 6
    rows.append(unit(PATTERNS[k] + 0.4 * unit(rng.standard_normal(D)))); tags.append({"pattern": k})
for i in range(300):
    rows.append(unit(rng.standard_normal(D))); tags.append({"pattern": -1})
for s in range(0, len(rows), 50):
    call("POST", f"/db/{DB}/collections/{COL}/write",
         {"vectors": [[float(x) for x in v] for v in rows[s:s+50]],
          "metadata": tags[s:s+50]})

# Six probes the store has never seen, one per real situation. A probe is graded
# by how closely what comes back matches the situation it was drawn from.
probe = np.random.default_rng(99)
PROBES = [unit(PATTERNS[k] + 0.4 * unit(probe.standard_normal(D))) for k in range(6)]
def recall():
    out = []
    for k, q in enumerate(PROBES):
        r = call("POST", f"/db/{DB}/collections/{COL}/read", {"query": [float(x) for x in q]})
        out.append(cos(np.array(r["result"]), PATTERNS[k]))
    return float(np.mean(out))
entries = lambda: call("GET", f"/db/{DB}/collections/{COL}/stats")["num_locations"]

print(f"after ingest:  1500 readings stored as {entries()} entries, recall {recall():.4f}")

# Copy the database before changing it. Safe with the engine running.
print(subprocess.run(["heather", "--data-dir", "/tmp/heather-blog",
                      "snapshot", "create", "--db", DB],
                     capture_output=True, text=True).stdout.strip())

# Name the collection. Without ?collections= every collection gets reworked.
for i in range(5):
    r = call("POST", f"/db/{DB}/dream?collections={COL}")["collections"][0]
    print(f"dream {i+1}: {r['locations_before']:4d} -> {r['locations_after']:4d} entries, "
          f"folded {r['merged']:3d}, recall {recall():.4f}")

# A mode the engine does not recognise falls back to the ordinary pass without
# saying so. The report is the only place the difference is visible.
r = call("POST", f"/db/{DB}/dream?mode=Ladder2&collections={COL}")["collections"][0]
print(f"\n?mode=Ladder2 -> touched {r['collection']!r}, thresholds "
      f"{r['tau_split']:.2f}/{r['tau_cohere']:.2f}  (a replay pass, not a ladder)")

rep = call("POST", f"/db/{DB}/dream?mode=ladder&levels=2&collections={COL}")
for r in rep["collections"]:
    print(f"?mode=ladder  -> wrote {r['collection']!r}, {r['locations_after']} entries, "
          f"thresholds {r['tau_split']:.2f}/{r['tau_cohere']:.2f}")
print("collections now:", call("GET", f"/db/{DB}/collections")["collections"])

What to change for your own store

Run it on a copy first. Point the script at a restored snapshot rather than the live database, watch the entry count over five passes, and only then decide whether it is worth doing for real.

Let it happen on its own. The engine will do the ordinary pass unprompted when things are quiet, if you switch it on per database:

[dream]
enabled = true
idle_secs = 60
passes = 1
collections = []       # empty means every collection

It checks every fifteen seconds and runs only when the whole server has been quiet for idle_secs and the collection has changed since last time. That last condition is what stops a settled collection being worked over forever. The first has a consequence: a server that is never idle never does this on its own, so a busy production database needs the manual trigger on a schedule of your choosing.

Watch the entry count as a signal in its own right. A collection whose count keeps dropping every night is still discovering things about your data. One that has gone flat has found everything currently there, and a sudden resumption means something new started arriving.

When to use this and when not to

Your data must actually repeat. The seventeen per cent here came from six situations seen two hundred times each. Data where nothing recurs has nothing to fold, and you will spend the write for no reduction.

It costs a full rewrite. One write per stored entry plus a folding pass. That is real work on a large collection, which is why it is tied to idleness rather than run continuously.

One ordinary pass is the settled behaviour. Everything past that is frontier. Multiple passes grow coarser groupings on top of the fine ones, and the ladder rewrites what a collection means rather than how it is arranged. Both are worth knowing about before you need them; neither belongs on a production database you have not measured on a copy first.

More depth: consolidation, Use dreaming, and Consolidation.