LOCAL INFERENCE
On-device semantic search
An offline Android search system over 2,155 items, fusing quantised dense retrieval with lexical search in a 22 MB local model.
Personal system on a phone, not packaged for release. Numbers recorded when it was built.
MECHANISM
Local because it has to be
MiniLM quantised to int8 runs through onnxruntime with XNNPACK. The model and the index both stay on the phone, so a query costs no network round trip and no query text leaves the device.
- QUERY Typed on the phone no network; nothing leaves the device
- TWO RANKINGS Dense and lexical MiniLM int8 · onnxruntime · XNNPACKBM25
- FUSE Reciprocal rank fusion on rank, not score: cosine and BM25 live on different scales
- DEDUPE One result per file or the top is three fragments of one note
- RESULT 2,155 items, 3 ms
THE FAILURE THAT SHAPED IT
Dense retrieval alone was not enough
A real query about stopping the system from killing long-running jobs missed its answer, because the chunk that held it was dense with commands and paths that diluted its vector. The finished system fuses dense and BM25 rankings by reciprocal rank fusion rather than a weighted sum, because no fixed normalisation between the two scales survives a growing corpus; sweeping the weights changed nothing, which is what fusing on rank looks like.
MEASURED
Small numbers, stated as small
Queries answer in 3 ms warm across 2,155 items. On seven check queries written to share no keywords with their answers, the right result was first five times and in the top three all seven times. That is a sanity check, not a benchmark, and it is described as one.
IN AN INTERVIEW
The two-minute walkthrough
-
0:00
Name the boundary: a personal system on a phone, unpublished; the numbers were recorded when it was built.
Claims sized to their evidence.
-
0:15
Walk the pipeline.
Local inference on a phone, and why it has to be local.
-
0:45
Tell the failure: dense retrieval alone missed a real query.
The failure-then-fix is the credible part, and it explains fusing on rank.
-
1:30
Give the check with its n.
Seven queries is a sanity check, and saying so is the point.