After reading about Jev, I wanted to know if small AI models running on my own hardware could do useful work on my Git forge. I kept settling questions there, forgetting I had, and asking them again weeks later, so the first test was whether a model could remind me. So far, a fifty-year-old way of counting words does it better than any of them.
The tool I ended up with works like this. I describe a new problem in one line, and it lists the old issues I should read first. Here the problem was that after a reboot, one of my servers sent its whole log history again.
$ issue-related "systemd-journal-upload cursor lost after reboot, journal replays"
#956 0.14 hypervisor: no journal entries reached VictoriaLogs in 15m
#968 0.10 checks: service-state cannot see a StateDirectory from a packaged unit
ranked by term overlap, then by links from the top hits; it misses about
a third of real links, so an absent issue here is not a clearanceBoth were the right things to read, and I’d forgotten both. It finds the issue I actually needed in its top five about two times in three. Showing the five newest issues instead manages about one in four, and picking at random almost never.
That also means it cannot be read as a clearance. Checking it, seeing nothing, and concluding the question is new converts I didn’t look into I looked and it was clear, which is worse than never having looked. So it says so on every run.
Underneath is TF-IDF. It counts the words in every document, weights rare words above common ones, and calls two documents similar when they share unusual vocabulary. Two additions carry most of its strength. It reads the comments as well as the issue text, which are twice the volume and worth about seven points. And it lifts whatever its top three hits already link to.
How I graded it
Grading needed an answer key, and the forge already had one. Whenever I mention an old issue in a new one, I’m recording that the two are related. So the test hides that mention, gives the system only the new issue’s text, and lets it search only the issues that existed when the new one was written. If the issue I actually linked comes back in the first five, that counts as a hit.
The models I tried are all small enough to run on a laptop. Most turn each issue into a position in space, so that similar issues land near each other, and suggest the nearest ones. The chart shows each model on its own, and again as a second pass over what counting words found.
Counting words beat every model. The best one on its own, SPLADE, was seven points behind, and blending any of them into counting words added a point or two at most. What did help was structure rather than language. A decision, its follow-ups and the fix that later undid it tend to link to each other, so one good hit pulls in related issues that share few words with the new one.
The likeliest reason is what my issues are made of. They’re full of exact names, like StateDirectory, checks/service-state.nix and the names of my machines. Matching a rare string exactly is what counting words does best, and what a model trained on ordinary prose does worst.
Where a method had settings to tune, I tuned them on older links and checked them on newer ones, so the results aren’t just fitted to the answers. The answer key only counts links someone bothered to write, so a useful suggestion can still score as a miss, and the real numbers are probably a little higher. Most of those links were written by the AI coding agents I run, which search by keyword and might favour counting words, but the models lost on the links I wrote myself too.
The forge is small and I didn’t try the largest models, so none of this says a big model would fail.
What else I tried
Most of what I tried was aimed at giving the models a fairer chance. Letting them read the whole issue in passages, instead of only its start, lifted gte-small five points. Rewriting checks/service-state.nix into plain words changed nothing. Fine-tuning a model on my own linked pairs moved it three points, inside the noise. Qwen3-Embedding-0.6B, newer and larger, was the best single-vector model, and the rerankers, which read the new issue and each candidate together, made the order worse. None of it closed the gap.
To check the result wasn’t peculiar to this forge, I ran the same comparison on my notes vault, graded against its wikilinks. Counting words won there too, in the same order. I’d expected prose to favour the models, but my notes are short and full of names, much like the forge.
Before any of this I aimed a model at my labels, predicting who should act on an issue and escalating when unsure. Four issues in five are labelled within a minute of being filed, so there was no decision left for a model to take. Five minutes of counting would have told me that before I built anything.
So for this job the models came close but didn’t earn their place, and they cost far more. Counting words runs the whole test in under a minute with nothing to download. The small models needed a few minutes to process the forge, and the largest took about seven and a half hours on my server’s processor, reading each issue in passages. Most of that was the half-precision weights it ships, which my processor has to emulate: over each issue’s first 512 tokens the same model takes 55 minutes, and under eight minutes in single precision. Here the cheapest method is also the best one.
The models aren’t bad at language; my issues just aren’t much language. A model trained on code is the one I didn’t run. It would have to beat counting words by a margin no model here came within ten points of, so I left it there.