Module embedding

Module embedding 

Source
Expand description

The text a record should be embedded as.

SemanticFallback is a trait: the engine ranks and fuses, and the caller supplies the vector search. That leaves one decision entirely to the implementor — what text goes into the embedder — and it turns out to matter more than any other choice in the semantic layer, including the model and the number of dimensions.

So the answer lives here rather than in a caller’s head.

§What was measured

Six candidate texts, over 1,199 records and 93 questions phrased as people actually speak rather than as the corpus is written (tests/semantic_fusion_probe.rs):

what was embeddedtop-1
the statement alone41/93
statement + hand-written aliases and tags57/93
statement + the predicate line53/93
statement + the whole frontmatter as prose66/93
statement + six LLM-written questions31/93
all of the above together52/93

The winner is embedding_text, and the thing to notice is that it costs nothing: no model call, no second pass at ingestion, no author. It is the fields the record already carries, written as a few lines of prose.

It also beats the hand-written aliases — which were composed by someone who had seen the question set — by nine questions.

§Why it works

A statement gives the value and only implies the attribute. “The user’s usual coffee order is a cortado” answers a question about coffee orders without ever containing the words “coffee order” in the way a question asks them. Naming the attribute outright is worth 12 of the 25 points (41 → 53); the subject, entities and temporal scope are worth the other 13 (53 → 66).

Every one of those fields carries retrievable signal the statement had left implicit. It is not that structured boilerplate flatters the geometry.

§Why not to enrich further

Two separate experiments say adding model-written text makes this worse, and they agree on the reason. LLM-written questions score 31/93, below the bare statement; LLM-written short alias terms score worse than leaving the alias field empty (tests/alias_terms_probe.rs).

The shape was never the variable. Both were applied to every record, and uniform enrichment is what fails: a synonym added to one record is a rare, discriminating term, while the same synonym added to all of them has no IDF left and discriminates nothing — having lengthened a length-normalised field everywhere on the way. Selective enrichment may well pay; nothing here can target it, and ingestion is the wrong place to try.

Functions§

embedding_text
The text to embed for a record.
frontmatter_prose
The frontmatter alone, as the prose embedding_text appends.
predicate_line
The line naming the attribute this record is about.