What this shows
- Retrieval over a translation memory: each new string reuses the closest stored segment, so output starts from approved translations instead of being generated from scratch.
- Deterministic QA that catches model mistakes: every output is checked against its source, and on the real model run it caught the one output that broke a rule (shown in the examples below).
- Measured experiments only: 6 runs on the same gold set, including one, hybrid keyword search, that made recall worse and was not adopted.
Headline numbers
97.6%
QA pass with a real model (Qwen)80.5% with the retrieval-only baseline
82.9%
Right match in the top 10 (zh-CN filter)
58.5%
Right match first (zh-CN filter)
0
Bad JSON replies in 41 runs
Sources: eval_qwen.json: qwen2.5-coder:14b served locally, prompt v2.0, 41 gold items, 544 s wall time; eval.json: retrieval-only baseline, mock provider; eval_zhcn.json: zh-CN target-language filter.
Pipeline
Recall@k by configuration
Cross-language check: 10 Chinese queries from gold_zh.jsonl searched against the English sources: recall@10 80.0%, MRR 55.9% (eval_crosslang.json).
QA by check: mock vs real model
Each check has two thin bars: mock (grey bar, mock that copies the closest TM match) and Qwen (accent bar, real model). Percentages are the share of the 41 gold items passing that check; checks where the two runs differ are listed first.
From eval.json, eval_qwen.json.
Real examples
| Source | Closest match | zh-CN | zh-TW | th | Output (Qwen) | QA |
|---|---|---|---|---|---|---|
| New Gamequery: start a brand new playthrough | ui-new-game新游戏 | 新游戏 | 新遊戲 | เกมใหม่ | 新游戏 | pass |
| Continuequery: resume my saved adventure | hud-xp经验值 | 继续游戏 | - | เริ่มต่อ | 继续游戏 | pass |
| Healthquery: how much health do I have left | hud-hp生命值 | 生命值 | - | - | 生命值 | pass |
| %d damage dealtquery: how much damage did I deal | combat-damage造成 %d 点伤害 | 造成 %d 点伤害 | - | - | 造成 %d 点伤害 | pass |
| Health Potionquery: use a red potion in battle | item-health-potion生命药水 | 生命药水 | - | - | 生命药水 | fail: required_term_used |
| Welcome to Emberveil Legends, {name}!query: welcome message shown to a new player | ui-new-game@thเกมใหม่ | 欢迎来到烬堇传说,{name}! | - | - | 欢迎来到烬堇传说，{name}！ | pass |
The zh-CN, zh-TW and th columns are the stored translations of that unit in the translation memory (a dash means the TM holds no variant in that language). The output column is what Qwen produced for the query; QA ran on the source segment.
What we tried
| configuration | recall@1 | recall@3 | recall@5 | recall@10 | MRR | QA pass (provider) | kept |
|---|---|---|---|---|---|---|---|
| baseline | 51.2% | 68.3% | 73.2% | 80.5% | 61.5% | 80.5% (mock) | no, reference |
| target-lang zh-CN | 58.5% | 73.2% | 75.6% | 82.9% | 67.5% | 82.9% (mock) | yes, best |
| hybrid RRF+BM25 | 53.7% | 56.1% | 61.0% | 65.8% | 56.3% | 87.8% (mock) | option only, worse recall |
| zh-CN + hybrid | 53.7% | 58.5% | 61.0% | 65.8% | 56.6% | 87.8% (mock) | no |
| zh queries, cross-language | 50.0% | 50.0% | 70.0% | 80.0% | 55.9% | 70.0% (mock) | standing check |
| baseline retrieval + ollama qwen2.5-coder:14b | 51.2% | 68.3% | 73.2% | 80.5% | 61.5% | 97.6% (ollama) | yes, real model run |
- Plain vector search, nothing changed: the right match was in the top 10 for 80.5% of queries and ranked first for 51.2%; kept as the reference run, not the default.
- Keeping only zh-CN units in the ranking lifted the top-10 share to 82.9% and first place to 58.5%; this is the configuration we kept.
- Adding BM25 keyword fusion to the vector search made recall worse at every k, with the top-10 share falling to 65.8%; it stays an option, not the default.
- Filter plus hybrid together was strictly worse than the filter alone (top-10 65.8% against 82.9% for the filter only); not adopted.
- 10 queries written in Chinese, searched against the English sources, still found the right match in the top 10 for 80.0% of them; kept as a standing cross-language check.
- A real local model (Qwen) passed QA on 97.6% of the 41 items with 0 bad JSON replies, against 80.5% for the retrieval-only mock; kept as the translation path.
One row per run in results/: eval.json, eval_crosslang.json, eval_hybrid.json, eval_qwen.json, eval_zhcn.json, eval_zhcn_hybrid.json.
What a production system adds
- A real LLM provider with retries, fallbacks and cost limits (this repo runs one locally: Qwen via Ollama).
- A versioned TM with metadata (domain, quality scores, last used), per-language collections, and auth on the vector store.
- Batch and async ingestion, a human review workflow, and terminology enforced during decoding (constrained decoding).
- Continuous evaluation against a held-out set with alerts when recall or QA pass rate drops.
Limits
- 133 synthetic segments and a 384-dimensional MiniLM bound retrieval quality; recall@1 stays well below 1.
- The mock provider copies the nearest TM match; it does not translate. Its QA failures are reported, not hidden.
- Hybrid BM25 + RRF made recall worse on this paraphrased gold set (measured, kept as an option, not the default).
- The term base covers zh-CN only; zh-TW and th units exist in the TM but terms are not enforced for them.
- Qdrant runs in local mode, single process, no concurrency guarantees.
- Every number on this page is from one run of the committed code on the committed data; deterministic here, not a general claim.