Competitive learning evaluation protocol

Registered before running the experiment, 5 November 2026. This is a synthetic learning benchmark, not an individual-record recall benchmark or a standard industry leaderboard. Production engine algorithms and defaults are unchanged.

Question and target

Does competitive writing learn a useful shared pattern from noisy observations, such that a fresh noisy example can be mapped closer to its clean latent prototype? The clean prototypes are never written to HeatherDB. Labels are available to the evaluator only, except for a clearly labelled supervised centroid reference. Held-out examples never enter the training stream.

Fixed design

• Dimension 128, eight orthogonal latent prototypes generated by QR from a seeded Gaussian matrix. Data seeds: 11, 29, 47, 71, 101.
• Each observation is the unit-normalized sum of its prototype and isotropic unit noise, scaled by the declared noise amplitude. Every training and test noise vector is independently generated.
• Stationary balanced: noise amplitude 0.75; hard-noise condition: 1.5; imbalanced: noise 0.75 with class probabilities 0.65, then 0.05 for each of seven remaining classes. Balanced streams shuffle equal-class blocks.
• Learning checkpoints: 64, 256 and 1,024 writes. Test panels are balanced, with 16 fresh noisy observations per class (128 in total), held fixed across checkpoints to measure learning curves. No test-score hyperparameter tuning.
• Drift: after 256 balanced observations, rotate each latent prototype 45° into its own orthogonal direction. Evaluate after 0, 64, 256 and 1,024 new observations. Measure new-pattern prediction and old-pattern retention.
• Engine: the audited 1.0.0 candidate; f64, synchronous durability, data-seeded locations, competitive writes at shipped defaults (k=20, write beta=5). A predeclared k=1 competitive variant diagnoses winner-only learning. It does not replace or redefine the default result.

Inputs are seeded. The public database API does not expose the engine's overload-split RNG seed, so the run is not bitwise deterministic. Save actual learned codebooks and score arrays; report this source of variation.

Baselines and budgets

Use the identical training prefix for all methods:

1. No-op input: return the noisy observation unchanged.
2. Global mean: one streaming aggregate, with no class labels.
3. Plain storage: retain all observations, no competitive learning.
4. Reservoir sampled at the learned location count, separately for each competitive variant. Also include a reservoir at twice the default count, since a competitive location has an address and a counter.
5. Online k-means at the same learned location count, initialized from the first K observations and updated by nearest-centroid running means.
6. Online k-means at eight centroids, and a recency-weighted eight-centroid version (step 0.05). Eight is the known generator complexity: mark these as informed-complexity baselines, not automatically inferred cluster counts.
7. A labelled training-centroid reference, explicitly supervised and excluded from claims about unsupervised performance.
8. Offline eight-centroid k-means++ (five initializations, maximum 100 iterations), an informed-complexity baseline to avoid attributing an online initialization disadvantage to competitive learning.

Labels and clean prototypes must not affect competitive writes, reservoir selection, online k-means or temperature selection. Location counts and logical key/value float payloads are reported; count matching is not an RSS or on-disk byte-budget match. Graphs, caches and database overhead are outside that payload.

Isolate learning from the read mechanism

Export the addresses, counters and write counts produced by actual database writes. Score all learned representations through the same exact top-20 decoder at fixed attention scales 5 and 50, with logits scale * dot(query, address) + log(count) and values counter / count. Centroid and sample baselines use equivalent key/value/count representations. Use the same decoder for both write variants, including the k=1 ablation.

Primary score: cosine of the scale-50 output to the held-out observation's clean latent prototype. Report scale 5 too, plus nearest-address and nearest-counter fidelity, macro/per-class scores, prototype coverage, codebook size and old-pattern retention. These diagnostics distinguish address learning, counter contamination and decoding effects.

Spot-check the real HTTP attention and iterative-read endpoints separately. HTTP attention uses the database's configured k and approximate activation, whereas the controlled decoder uses exact common top-20 activation. Report their differences; do not attribute graph/read effects to write learning.

Criteria and analysis

Primary condition: stationary balanced noise 0.75 at 1,024 writes, shipped competitive defaults. Two primary comparisons are default competitive versus the noisy input, and versus its count-matched reservoir. Use paired seed-level means, not held-out queries as independent replicates. Report 95% t intervals and 97.5% intervals for these two comparisons (Bonferroni family coverage).

• A gain over the noisy input supports denoising on this task.
• A gain over the count-matched reservoir supports added value beyond retaining a comparably sized sample. The twice-count reservoir is an additional budget sensitivity check.
• Improvement from 64 to 1,024 observations supports a learning curve.
• Reduced prototype error after drift supports adaptation; report any loss of old-pattern performance alongside it.
• Useful compression additionally requires fewer learned locations than observations. Neither lower location count nor better training fit alone establishes learning.

Other conditions, ablations, baselines and diagnostics are secondary. Report negative results and any advantage of simpler baselines. No algorithm changes, test-driven parameter search, general reasoning claims or semantic-quality claims are part of this experiment.

Discovery-engine calibration

Discovery here means proposing unknown shared directions from actual learned codebooks, without class labels, followed by independent validation. It does not mean discovering a new scientific law in a synthetic generator whose structure is already known to the evaluator.

• Separate streams contain five or nine hidden groups, noise amplitude 0.75, and 30% isotropic background. The proposer is not given the group count. Training and independent validation each have 1,024 observations. A matched structure-free isotropic stream is the null. Use the same five data seeds.
• Candidate directions are normalized learned counter averages with effective write count at least four. Rank by average-counter norm times square root of count, deduplicate at cosine 0.95 and cap at 64 proposals, all before looking at validation data. Also report learned-address candidates as a diagnostic.
• Compare default and k=1 competitive writes with local top-20 averages of a count-matched reservoir, and HDBSCAN (min_cluster_size=16, min_samples=8) which does not receive a group count. A labelled centroid reference checks representational adequacy but is not an unsupervised contender. A global-mean proposal checks whether a validated aggregate adds anything beyond a single population direction.
• A proposal needs at least eight independent validation observations with cosine above 0.5 and a one-sided spherical-cap/binomial p-value below 0.05 divided by the number of proposed directions. The known isotropic distribution defines this synthetic null, not a distribution-free real-data test. No validation-derived tuning or proposal re-ranking.
• Only afterwards score hidden-group coverage and matched-candidate fraction at prototype cosine 0.9, with one group counted once. A validated aggregate that does not match an individual planted group is reported as unmatched, not automatically declared meaningless. Report all proposal counts and null validations, not just successful candidates.

This validation layer supplies rejection/abstention; the native memory itself does not automatically abstain. A discovery-engine claim needs proposal utility over simple baselines, held-out support and controlled null behaviour together.