retrieve.ts
@file The two lexical retrievers Stage 0 measures, over one shared tokenizer.
Overview
@file The two lexical retrievers Stage 0 measures, over one shared tokenizer.
Both baselines take the same term list, so a difference between them is a
difference in the index and not in how the question was read.
The FTS5 index is built twice on purpose. SQLite's unicode61 tokenizer treats
_ as a separator, which shreds CONFIG_UART_CONSOLE into three ordinary
English words and makes the single most common query shape in this repo
un-retrievable. tokenchars '_' fixes it. Measuring both quantifies how much
of "lexical search works here" is really "lexical search configured for
identifiers works here".
depends on corpus.ts golden.ts · used by independent.ts scope.ts stage0.ts stage1-probe.ts
API
Fexport function terms(question: string): string[]
Query terms, exact tokens preserved. Order matters only for readability; scoring is by idf, not position.
diagnose, ftsSearch, ripgrepSearch · calls keepFexport function chunkLocator(chunks: Chunk[]): (file: string, line: number) => Chunk | undefined
Chunk lookup by file, for turning a rg file:line into the chunk holding it.
diagnose, main, main, main, makeScorerUndocumented (5)
keep, buildFtsIndex, ftsSearch, ripgrepSearch, corpusFiles