Skip to content
Blog

Architecture

Building memory an agent can trust

What we learned moving from giant markdown files to scoped records, selective reads and evidence-based maintenance.

We saved a rule, then watched the agent ignore it. The lesson was simple: keeping a reliable record and building useful context are separate jobs. Here is how the design evolved across file-backed and database-backed implementations.

A few giant markdown files feed their entire contents into every turn.
AI-generated concept illustration of the starting design. File names are illustrative; no file sizes or growth rates are implied.

1. Start with a few giant markdown files

We began with readable files loaded into every turn. As they grew, each request carried more irrelevant context, cost and stale material. A loading limit could cut the tail entirely. Then a concurrent-write test exposed a second problem: both saves succeeded, but one rule disappeared.

  • Recurring inputCaching can reduce price; it does not remove the context footprint.
  • Context rotUseful instructions compete with stale, duplicate and unrelated text.
  • Lost tailA hard cut means some text never reaches the model.
How two successful saves lost one rule
FIG. 01Two successful writes, one missing rule
Time
Writer A
Writer B
1 · read
[R0]
[R0]
2 · add
[R0, A]
[R0, B]
3 · write
Saved A
Saved B last

Final file: [R0, B] · A is lost, although both writers reported success.

Illustrative interleaving of the lost-update failure reproduced on a data copy. Records still need concurrency control; changing the format alone does not fix the race.

2. Separate organization and user context

We moved to one record per claim, with an owner, source and history. Writing chooses where a claim belongs. Reading chooses which records the current actor may see. Sharing useful knowledge and preserving private context need different gates.

Organization and user context on both write and read paths.
AI-generated reference diagram. Org A is illustrative. Write routing and read visibility are separate decisions; limitations are described below.

In the checked implementation, the org-write gate used model-reported evidence and a heuristic affiliation check that allowed unknown cases. It was not independent verification. User rules had a standing block; org memories still relied on search.

Scope and type are independent dimensions: organization, user, tool or agent; instruction, fact, episode or procedure.
AI-generated concept diagram. Scope and type are independent dimensions in the reference design; rows do not imply one-to-one pairings. The typed record combines both dimensions.
Reference record schema and the write/read checks
MemoryRecord {
  id, kind, body
  scope: { organization, user?, project?, tool?, agent? }
  source: { conversation?, message?, evidence? }
  valid_from, valid_until?
  supersedes?, checked_at?
}

The schema is conceptual. The actual organization lesson gate accepted user-stated or verified-in-data labels with an affiliation check; otherwise the lesson stayed with its author. Organization filtering and actor-visible references constrained reads. The inspected 9,380 recalled items had no cross-user or cross-organization leaks; that is evidence for the sample, not a universal guarantee.

3. Give memory types different read policies

A stored “no sales tax” instruction was missed when the next request simply asked for an invoice. We separated applicable instructions from task-relevant search. Facts, episodes and procedures can compete for relevance; standing rules should not depend on the query repeating their words.

FIG. 02Two lanes, one memory block
Standing rules and relevant context converge into the memory for this turn, without searching for the standing instruction.
Explore three requests through the two lanes

A record exists: “Do not charge sales tax on our invoices.”

Request: “Make an invoice.”
Instruction laneScope + applicability

Include the invoice instruction even though the request does not mention tax.

Context laneSearch + rerank

Find invoice context. Similarity search alone can miss the tax rule.

↓ Both lanes feed the budgeted memory block ↓
Pack → harness adapter → model
AI-generated concept image and interactive illustration of the intended read contract, not a live retrieval demo. The one-word case shows the desired separation; the reported rules-block implementation still had an early-return defect.

The reported rules block still overflowed: one user saw 23 of 135 rules. Short replies could skip it too. Separating the lanes is the design requirement; those defects were not established as resolved.

Memory types and the reference read algorithm
KindExampleHow it should be used
Standing instruction“Do not charge sales tax on our invoices.”Consider whenever its scope applies, even if search would miss it.
Fact“The warehouse closes at 5.”Retrieve when relevant; recheck if it can change.
Episode“Last week's invoice reconciliation.”Help locate earlier work and its evidence.
Procedure“How to prepare the daily reconciliation.”Retrieve as a coherent workflow, with its preconditions.
build_memory(actor, task, budget):
  eligible = records_visible_to(actor)
  rules = applicable_instructions(eligible, actor, task)
  candidates = search(eligible, task)
  ranked = rerank(candidates, task)
  block = pack(rules, ranked, budget)
  return block, selection_trace

Authorization precedes relevance. The packer should expose what did not fit; different implementations use different selectors.

4. Optimize reads: rank for the task

A smaller prompt was still noisy. TypeSafe’s Jev introduction gave us a structured scorer to explore. We added task-aware reranking after broad candidate search, with word ordering as a fallback. Later, a separate session-start path kept injecting unrelated tunnel notes. Improving one entrance had not fixed the others.

FIG. 03Separate access, relevance and size
01 · ELIGIBILITYAuthorized records onlyScope is an access boundary.
02 · CANDIDATESBroad, inexpensive searchInclude facts, procedures and earlier work.
03 · RELEVANCETask-aware rerankingFallback when the scorer is unavailable.
04 · BUDGETPack with explicit exclusionsPreserve pointers to the full sources.
Reference read pipeline distilled from the implementations. A scorer cannot recover a needed record that candidate retrieval never found. It also cannot override scope.
READ QUALITYFewer unrelated items reached the slice
Word matching
66%
Task-aware reranking
37%
Noise share; lower is better. Fixed 0–100% scale. Counterfactual slice grading, not task success. Search nDCG also rose from 0.47 to 0.59 on 57 replayed searches.

Candidate snippets went to an external scoring service, adding privacy and latency tradeoffs. A live session’s tunnel lines fell from eight to zero after a scoping fix; fleet-wide task gains were not proven.

5. Optimize reads: delivery and cache placement

Good selection did not guarantee delivery. The harness exposed a preview of a much larger memory block, and the overflow was never opened. We bounded the block and started checking the model-visible prompt rather than trusting the hook output.

FIG. 04Selection is not delivery
Hook emits~15 KB
Selected memory block
HARNESS
PREVIEW CUT
Model sees2 KB
Overflow never opened

45 of 54 inspected injections were cut.

Block sizes are approximate. Both strips use the same 15 KB reference scale; the second shows the preview size. This is one harness integration, not a universal limit.
PROMPT LAYOUTKeep stable and changing context separate
STABLE PREFIXInstructions + tool definitions

Preserve the reusable prefix.

TURN-SPECIFIC TAILSelected rules + retrieved context

Put changing memory in the adapter’s appropriate position.

Reference layout principle. Actual cache boundaries and visibility limits depend on the harness; inspect what it sends.

An 8,000-character cap addressed that integration. It is not a portable budget. A different harness needs its own visibility and cache checks.

6. Maintain records without losing meaning

Revisiting the design through Anthropic’s memory-components article sharpened the maintenance question. A repeat summary silently archived instructions it omitted: 74 user-stated rules were lost in two weeks. We stopped treating omission as withdrawal. Explicit replacements retained the old version, but near-duplicate merging could still remove an obligation.

Version one, version two and the current record remain linked; each version retains a source.
AI-generated concept illustration. Replacement changes which version is active while preserving older versions and their evidence.
FIG. 05Similar wording can remove an obligation
Older active instruction

Use ALL CAPS for category codes and size entries.

↓ marked as “restated” by a similar record
Replacement

Use ALL CAPS for applicable category codes.

The replacement no longer carries the size-entry obligation.

SIMILARITYCandidate pair≠Authority to supersede
A replacement failure from the memory check, with domain-specific labels generalized. The stronger replacement gate discussed below is a design requirement, not an implemented guarantee.
MAINTENANCE LOOPKeep, replace or retire with evidence
  1. Find a candidateDuplicates, contradictions and changing environment facts.
  2. Follow the sourceA similar sentence is not proof of the same obligation.
  3. Change the active setExplicit replacement or retirement; preserve history and leave unresolved claims reviewable.
Reference maintenance policy. Source-check instructions existed in a template; their operation in the live process was not established.
Cleanup results and the remaining merge risk

The restore repair recovered 76 records, a different set from the 74-rule loss sample. A later write replay turned 28 of 316 saves into updates; 10 of 12 sampled merges were judged right. One-time cleanup retired 22 facts across 12 groups. The two questionable merges are why similarity alone cannot authorize superseding a rule.

7. Optimize maintenance with measured feedback

The “used” field only meant selected for a brief. We needed separate evidence at each boundary. A 26-case experiment supported preserving lessons and supplying standing rules, but its improvement interval crossed zero. It was direction for the next iteration, not proof that memory was solved.

FIG. 06Evidence for the design, with uncertainty
Rule-following score0 → 1
No memory
0.51
Existing implementation
0.72
Preserve lessons
0.77
Preserve + standing rules
0.83
Reported paired improvement interval

−0.01 to +0.24 · crosses zero

26 cases · two automated judges · no human validation
All bars share a fixed 0–1 scale. The interval is the reported paired analysis, not inferred by subtracting these rounded scores. Detailed coverage counts follow.
FEEDBACKLocate the failure before choosing the next change
  1. Written?The right claim and scope survived the save.
  2. Selected?The needed memory was picked for the task.
  3. Delivered?The full text reached the model.
  4. Applied?The answer followed the rule correctly.
  5. Outcome improved?Less rework or better success, latency and cost.
Coverage counts and evaluation method
Memory suppliedRule-following scoreNeeded rule reached the model
None0.510 of 26
Existing implementation0.7211 of 26
Preserve lessons across summaries0.7714 of 26
Preserve lessons + standing rules0.8324 of 26

Three answers per case, two blind automated judges, 78% judge agreement and no human validation. The larger planned run had not been performed. Next checks should include simultaneous saves, omitted rules, short replies, overflow, meaning-changing merges and slow search.

The final design: a scoped memory lifecycle

The store remembers what was learned. The read path chooses what belongs in this turn. The maintenance path changes what is still considered true. Each path returns a trace, so a lost instruction can be located at the boundary that failed.

REFERENCE ARCHITECTUREOne durable record, three controlled paths
INPUTS · conversation / tools + actor identity + source evidence
WRITE → STORE

Control the mutation

  1. Classify kind and scope
  2. Check authority and evidence
  3. Compare related claims
  4. Commit a version + event
DURABLE STORE

Keep the claim and its history

Identity
ID + organization + scope owner
Meaning
Kind + full claim
Provenance
Source + evidence + checked date
Lifecycle
Version + active dates + supersedes

Separate search index
Append-only change events

STORE → READ → MODEL

Build this turn’s context

  1. Filter by actor visibility
  2. Load applicable rules
  3. Search + rank context
  4. Pack with exclusion reasons
  5. Adapt to the harness
MAINTENANCE → WRITE PATHSource checks, not silent rewriting

Find drift → verify evidence → keep / replace / retire / review.

FEEDBACK ACROSS EVERY BOUNDARYSelected ≠ delivered ≠ helpful

Join record versions to actual prompts, behavior and task outcomes.

Reference design distilled from the investigations. Files or a database can hold the records; neither replaces the scope, mutation, delivery or maintenance contracts. The paths below specify intended behavior, not a claim that all of it is deployed.

Follow a record through the design

Write a claim without widening its authority

INConversation or tool result + actor identity + evidence
  1. ProposeExtract a durable claim; keep its original source reference.
  2. RouteChoose its kind and destination: organization, user, tool or agent.
  3. GateCheck permission to write that scope and the evidence policy. Similarity never grants broader authority.
  4. CompareFind related active records in the same scope. Decide add, explicit replacement, or review.
  5. CommitSave the new version, supersession link and event together; retire the old version only after the replacement exists.
OUTA scoped record version and an audit event, or an explicit rejection/review result.

Reference requirement: unknown authority must not silently promote a personal claim to organization memory. This is stronger than the permissive affiliation heuristic found in the checked code.

Where the implementation record stops

  • Built or observedScoped records, history, an org/user write gate, standing user rules, bounded recall, reranking and a one-time cleanup.
  • Requirements still exposed by failuresSafe unknown-authority handling, complete rule delivery under overflow, short-reply independence and meaning-preserving replacements.
  • Not establishedOperational nightly source verification and a controlled improvement in real task outcomes.
Readings behind the design

Articles and projects we explored

Related research

The failure cases and numbers in this article are our own retrospective audits and small experiments, not results from those papers. The 26-case comparison used three answers per case and two automated judges, with no human validation. The reported interval crosses zero. We did not isolate a dollar cost or a causal attention effect for the original giant-file design, and no fresh production check is implied by this article.

All generated images are conceptual. The final architecture and walkthroughs describe intended contracts; the implementation coverage panel marks what was observed and what remains unproven.

Get new posts by email

One email when a new post goes up. No other mail, and you can unsubscribe from any email.

Other posts