Expand description
The text a record should be embedded as.
SemanticFallback is a trait: the engine ranks
and fuses, and the caller supplies the vector search. That leaves one
decision entirely to the implementor — what text goes into the embedder —
and it turns out to matter more than any other choice in the semantic layer,
including the model and the number of dimensions.
So the answer lives here rather than in a caller’s head.
§What was measured
Six candidate texts, over 1,199 records and 93 questions phrased as people
actually speak rather than as the corpus is written
(tests/semantic_fusion_probe.rs):
| what was embedded | top-1 |
|---|---|
| the statement alone | 41/93 |
| statement + hand-written aliases and tags | 57/93 |
| statement + the predicate line | 53/93 |
| statement + the whole frontmatter as prose | 66/93 |
| statement + six LLM-written questions | 31/93 |
| all of the above together | 52/93 |
The winner is embedding_text, and the thing to notice is that it costs
nothing: no model call, no second pass at ingestion, no author. It is the
fields the record already carries, written as a few lines of prose.
It also beats the hand-written aliases — which were composed by someone who had seen the question set — by nine questions.
§Why it works
A statement gives the value and only implies the attribute. “The user’s usual coffee order is a cortado” answers a question about coffee orders without ever containing the words “coffee order” in the way a question asks them. Naming the attribute outright is worth 12 of the 25 points (41 → 53); the subject, entities and temporal scope are worth the other 13 (53 → 66).
Every one of those fields carries retrievable signal the statement had left implicit. It is not that structured boilerplate flatters the geometry.
§Why not to enrich further
Two separate experiments say adding model-written text makes this worse, and
they agree on the reason. LLM-written questions score 31/93, below the bare
statement; LLM-written short alias terms score worse than leaving the alias
field empty (tests/alias_terms_probe.rs).
The shape was never the variable. Both were applied to every record, and uniform enrichment is what fails: a synonym added to one record is a rare, discriminating term, while the same synonym added to all of them has no IDF left and discriminates nothing — having lengthened a length-normalised field everywhere on the way. Selective enrichment may well pay; nothing here can target it, and ingestion is the wrong place to try.
Functions§
- embedding_
text - The text to embed for a record.
- frontmatter_
prose - The frontmatter alone, as the prose
embedding_textappends. - predicate_
line - The line naming the attribute this record is about.