How the Pasture is populated: Chunker and the RAG sync migrations

SearchRag (docs/AskJooby/ResearchAndMemoryTools.md) searches the AskJoobyRagChunk table. Writing or editing a page under docs/ does not, by itself, make that page’s content searchable: the table is populated by a separate indexing step, described below, with exact file references.

What is, and is not, in this codebase

A generated FluentMigrator migration’s own doc comment names the tool that produces it:

One sync of Jube’s RAG corpus into “AskJoobyRagChunk”, generated by Chunker.

— Jube.Migrations/Rag/20263107510200/Rag20263107510200.cs

Chunker’s own source is not present anywhere in this repository. A case-insensitive search of the whole working tree for chunker turns up exactly one hit: that doc comment, naming the tool that produced the migration sitting next to it. There is no Chunker project, class, script, or CLI entry point checked into this codebase, under any name. Whatever reads docs/ (and the rest of the corpus – rule taxonomy, code, database schema) and turns it into AskJoobyRagChunk rows runs outside this repository.

What Chunker’s output looks like, and how it reaches the database

The consumption side is in the repository and is described precisely here:

1. Chunker computes a delta against the live AskJoobyRagChunk table and the current corpus, and emits that delta as one or more MessagePack payload files, each holding a batch of RagPayloadRow records (hash, an optional ParentHash, CategoryId/KindId as byte-backed enum codes, SourceFile, StartLine/EndLine, QualifiedName, Content, an Embedding vector, CreatedDate, and a Delete flag for a row that should be removed rather than upserted) – see Jube.Migrations/Rag/Models/RagPayloadRow.cs.
2. Those payload files are named rag-<timestamp>-NNNN.msgpack and placed in a “run folder” alongside one generated FluentMigrator migration class with a matching timestamp, for example Jube.Migrations/Rag/20263107510200/, containing Rag20263107510200.cs plus its .msgpack files.
3. Jube.Migrations.csproj embeds every such payload into the compiled assembly as a resource (<EmbeddedResource Include="**\Rag\**\*.msgpack"/>, Jube.Migrations/Jube.Migrations.csproj:39). The doc comment on Rag20263107510200.cs explains why the rows are never emitted as C# literals instead: doing that on a real full-corpus run runs the C# compiler out of memory.
4. The migration’s Up() calls RagPayloadLoader.Apply(dynamicEnvironment, ".rag-20263107510200-") (Jube.Migrations/Rag/RagPayloadLoader.cs), which: finds every embedded resource whose name contains that timestamp token and ends .msgpack; copies each one’s rows into a temporary staging table ("RagSyncStaging", ON COMMIT DROP) via COPY ... FROM STDIN (FORMAT BINARY); then, inside the same transaction, upserts every staged row that is not flagged Delete into AskJoobyRagChunk (INSERT ... ON CONFLICT ("Hash") DO UPDATE SET ..., preserving the existing Embedding when the staged row’s own embedding is null) and deletes every staged row that is flagged Delete. Migration.Down() is a no-op – the doc comment calls reversing it “not worth it for a resync tool,” since doing so would require snapshotting every row the migration touches before it touches it.
5. Because the payload carries only Chunker’s own computed delta against the live table, re-applying the same migration a second time is harmless, and an incremental sync stays small regardless of how large the whole corpus is.

This is the same migration mechanism every other schema or data change in this codebase uses – there is no separate RAG-specific migration runner. Migrations run by letting the application boot normally against the target Postgres instance, which runs every pending FluentMigrator migration (including a new RagNNNN one) in order before the application starts serving requests. There is no dedicated RAG-sync CLI command separate from an ordinary migration run.

What this means for a docs change

Writing or editing a file under docs/ changes the corpus Chunker reads on its next run. It does not, by itself, change anything in the live AskJoobyRagChunk table, and therefore does not change what SearchRag can find. For a docs change to reach the Pasture, two things have to happen:

1. Chunker runs again, externally, against the current corpus, to compute a new delta and emit a new run folder (a new rag-<timestamp>-NNNN.msgpack set plus a new RagNNNN.cs migration) the same shape as Jube.Migrations/Rag/20263107510200/.
2. That new migration then runs against the target database, the same way every migration runs: by booting the application against that database, or by whatever this deployment’s own migration-run step already is.

Chunker’s source, the job that invokes it, and any CI/CD step that runs it on a docs change all live outside this repository, if they exist at all. A docs change is not searchable by SearchRag until Chunker has run again and the resulting migration has been applied to the target database.

Cross-reference

• docs/AskJooby/ResearchAndMemoryTools.md – SearchRag, the tool that reads AskJoobyRagChunk.
• Jube.Migrations/Rag/RagPayloadLoader.cs, Jube.Migrations/Rag/Models/RagPayloadRow.cs, Jube.Migrations/Rag/20263107510200/Rag20263107510200.cs – the consumption-side source this page describes.