Chunk without losing meaning
Chunk without losing meaning — RAG with local embeddings. Start with complete sections; Save chunks.mjs; Measure before adding overlap. Practical assignment: Lab evidence.
Demo edition. The materials are undergoing editorial review.
Topics covered
- Start with complete sections
- Save chunks.mjs
- Measure before adding overlap
Start with complete sections
For this tiny corpus one document can be one chunk. Larger sources need section-aware boundaries; splitting a condition from its exception changes meaning.
RAG supplies selected source passages to a language model at answer time. It does not retrain that model or make every answer correct. This lab uses fictional workshop policies stored in plain text, so the retrieval mechanics remain visible. Keep document IDs, revisions and passage IDs together. A relevant-looking passage from an obsolete policy can be worse than returning no passage at all.
Save chunks.mjs
Create stable chunk IDs and copy source revision into every row. This simple implementation intentionally keeps each short training document intact.
A chunk ID is a reference to stored text, not a summary invented later. When chunk boundaries change, rebuild those IDs and the index together. Otherwise an old citation may still open a real passage while pointing to different content than the answer originally used.
Save each code example in the named .mjs file and run it from the same lab directory. Use one embedding model for both document and query vectors; vectors from different models or dimensions are not interchangeable. Persist the exact text next to its vector. Rebuild the index when text, chunk boundaries or embedding model changes, rather than combining incompatible old and new rows.
import {docs} from './corpus.mjs';
export const chunks=docs.map(d=>({id:d.id+'-r'+d.revision+'-c1',docId:d.id,revision:d.revision,text:d.text}));
console.log(chunks);
Measure before adding overlap
Overlap repeats context and increases storage and prompt length. Add it only when boundary tests lose needed evidence; deduplicate adjacent repeated passages before generation.
Inspect the retrieved passages before changing the answer prompt. If the needed fact is missing there, a more confident generator cannot repair retrieval. Record passage IDs, scores and expected evidence for each question. Similarity scores depend on the embedding setup; they are not calibrated probabilities that a claim is true. Keep an explicit no-evidence branch and verify citations against stored text.
Practical assignment
Lab evidence
Continue to learning