Semantic Search over Nextcloud in a Claude Code Harness

One of the frustrations of working with agents is that they forget. Every Claude Code [1] session starts fresh, from a system prompt, a project file, and whatever notes you maintain, and all of them share one property, that someone has to decide what is worth remembering. The most popular answer right now is to give the agent a second brain of markdown notes, an idea that gained wide visibility by Andrej Karpathy’s “LLM Wiki” pattern [3]: let an agent compile knowledge into interlinked pages and keep them current. That works, but the issue there is that the wiki only knows what was compiled into it.

First of all it is important to clear whether do you need this (i.e., a solution like this). And for most people tha answer might be “not really”. If your working knowledge fits in a handful of notes, a curated wiki, or even a single project file, is simpler and enough. It starts to matter a bit more when the pile gets large and outdated: when the thing you need was written months ago and you have forgotten it exists, and when organizing all of it by hand would cost more than the answers are worth.

A lot of people, myself included, already use Nextcloud [5] as the vault where all their files live, so the real question is not where to put the memory, it is how to make what is already there usable by an agent. The tool for that is not exotic, and it is not mine: it is qmd [4], the on-device markdown search engine by Shopify founder Tobi Lütke, which is also the tool Karpathy’s own writeup recommends. What I do differently is where I point it. Instead of searching a curated set of pages, I let qmd index everything in my Nextcloud, roughly a thousand markdown documents of academic materials, from paper drafts and lecture notes to supervision and project documentation, so the memory becomes equal to everything I already keep rather than only what I took the time to write up. Any Claude Code session, on any machine on the network, can then ask “what do we already know about X?” and get the relevant passages back in about a second.

It is retrieval-augmented generation [7] where the corpus is simply everything you kept. The memory grows as a side effect of working. Nextcloud was already the place where most of my files live, so making it searchable removed the need for an organization ritual rather than adding more files (please, keep them organized if possible).

How it works

Four self-hosted pieces, described at a high level:

• Nextcloud holds the files. It is the source of truth and does not change at all: the search service reads it, nothing more.
• A small search service runs on the server next to a spare 4 GB GPU, with three small local AI models for keyword and meaning-based retrieval. Nothing leaves the machine. For a corpus of unpublished manuscripts and other academic materials, that local-only property is not a nicety but the thing that makes indexing them defensible at all, and it is the comfortable position under European data-protection rules.
• Claude Code connects to it over the Model Context Protocol [6], one line of configuration per machine.
• The agent even feeds itself: it can write short summary pages back into Nextcloud, where they get indexed like any other document, so a hand-curated wiki and the raw files answer through the same search. In my case this is not manual: a small scheduled agent re-synthesizes those pages whenever I sync, so the loop runs on its own.

In practice it changes how a session behaves. Before proposing an experiment, the agent checks whether a past project already tried it. Before drafting a related-work paragraph, it retrieves my own prior phrasing. When a design choice feels familiar, it surfaces the earlier note where I settled it. None of this required writing a note in advance.

“Preliminary Evaluation”

The honest performance story is about the GPU. On a warm 4 GB GPU a search comes back in about a second. My worst case, a setup that reloaded the models on every session and ran everything on the CPU, took over five minutes per query.

A disclaimer before you proceed: numbers come from a single execution, don’t do that in a paper.

Keeping the models loaded mattered more than raw compute. The engine dropped its models out of GPU memory after a few idle minutes, so most queries paid to reload them from disk before doing any real work. Watching GPU memory during a query shows the reload as a slow climb before anything is computed. Pinning the models in memory removed that stall entirely.

On my box that card is also shared with Plex’s hardware transcoding, so pinning is a deliberate trade: it costs about 2.6 GB of the 4 GB Plex could otherwise use, and it stays tolerable mainly because I query during work hours while Plex runs in the evenings.

The takeaway for hardware is simple. A short query can be embedded on a CPU, so basic search without a GPU is plausible on a “database” this size. What does not survive without a GPU is the optional precision pass that re-scores results, slow enough on a CPU to be impractical. A spare 4 GB workstation card, like my NVIDIA T1000, is roughly the line between search you use constantly and search you use only when desperate.

Shortcomings

Two honest limits, from living with it. First, it only sees markdown, so PDFs, code, and datasets are invisible unless you convert them, which is the biggest gap. Well, it works for me. Second, retrieval is not the same thing as comprehension: the engine returns passages, but whether the agent draws the right conclusion from them is a separate problem. It is a memory that recalls the files, not one that understands their content.

The takeaway

The interesting part is not the search engine, it is the inversion of responsibility. “Curation-based” memory (a vault, a notes app, a compiled wiki) asks you to predict what will matter. Storage-based memory assumes you cannot, and makes recall a query-time decision over everything you kept. The two add-in together rather than compete: a hand-curated wiki can sit on top of the raw files, and both answer through the same search, which is how Karpathy frames it too, good answers get filed back into the wiki as new pages. For anyone whose output is already text in folders, that fits reality: the drafts and notes accumulate anyway, and the agent should be able to read all of it when a question comes up.

References

[1] Anthropic, “Claude Code.” https://code.claude.com/docs

[3] A. Karpathy, “LLM Wiki,” 2026. https://x.com/karpathy/status/2039805659525644595 and https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f

[4] T. Lütke, “qmd: mini CLI search engine for your docs, knowledge bases, meeting notes.” https://github.com/tobi/qmd

[5] Nextcloud. https://nextcloud.com

[6] Anthropic, “Model Context Protocol.” https://modelcontextprotocol.io

[7] P. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” NeurIPS 2020. https://arxiv.org/abs/2005.11401