Chunking my Way to a Better Search

This all started because I really wanted to save tokens.

It was painful watching Claude use filesystem tools to plow through my Obsidian notes. The filesystem's search_files tool will find files, but that's finding files, not search terms. To find a particular term, the filesystem tool reads the entire file. The token cost is brutal, and sometimes, on longer files, the desktop host just says 'whelp' and stops. This is exactly why I put the Obsidian search tools into an MCP server -- to keep Claude from choking itself on a massive token stream.

The Obsidian search tools work really well for what they do, which is deliver consistent results on specific search terms. But where they falter is when the terms aren't so cut and dried. For example, If I want to find notes I've made around data integrity and ask the LLM to look for it, it's going to do a text search on "data integrity" — which won't give me what I'm really looking for, which is notes about data integrity.

• "the pipeline maintains data integrity" → ✅ This will be found
• "ensuring accuracy and consistency of records" → ❌ This will be missed
• "preventing corruption in the dataset" → ❌ This will is also missed

You can steer the model to look for synonyms, and that helps with finding those missing terms.

• "data integrity"
• "consistency"
• "corruption"
• "accuracy"

But the LLM is grabbing tokens based on the entire context of the conversation up to that point, which can introduce terms that aren't part of the original intent. Moreover, the model will have to rely on internal configurations on when to stop supplying synonyms, which can consume a lot of tokens. Without dedicated guidance and/or skills, the model won't know when to stop. Which makes the completeness of searches hard to control.

All this additional work, also generates behavior I was trying to avoid which was wasting tokens. The wastage from over searching isn't as bad as reading a 4MB file, but it adds up, in a world where many a CEO is waking up from a binge and saying, "Wait, tokens cost money?"

I don't always have the vernacular for what I'm trying to solve, so I use the tokens I get back to improve my vocabulary, rephrase, and cast again — getting a clearer picture of the problem and potential solutions with each round. It's an elegant technique I like to call grousing until I get an answer.

This is how I went to adding semantic search to my toolkit. Semantic search differs from simple, or lexical searching by turning those three examples above - a hit and two misses - into three hits.

Looking at this, you might be wondering why you don't use lexical search for everything.

1. You need semantic storage. (e.g. ChromaDB)
2. You need to populate the storage (e.g. nomic-embed-text)
3. semantic search is not quite as fast
4. Search and replace with semantic search is a disaster (for now)

Adding semantic search to my obsidian search tools involved several steps. First I had to turn all my notes into embeddings and put them into ChromaDB. Embeddings are numeric vectors that you can do neat math on to figure out how close tokens are to each other. The process of creating the embeddings was a batch process that has to be run every time there's new data, but surprisingly, since it's augmenting the keyword search, missing a couple of chunks doesn't make this completely useless. But, the more chunks you miss, the more likely you'll miss something important.

Security Note: The ability to turn these numeric vectors into text is exactly why you have to protect your embeddings the same way you'd protect its resulting text. An attacker can use your embeddings and any reasonable data set to figure out your text. It's called an embedding inversion attack. and its well worth reminding people about in case you run into someone who says, "But, but, but they're just numbers!"

For me, creating the embeddings was one of the fiddliest parts of the whole process. I wound up trying parameters until I got satisfactory answers. My chunker evolved into something like this:

1. Strip YAML front matter
2. Split markdown on ## / ### headings.
3. Merge sections under ~80 characters into the previous chunk.
4. Split anything over ~2000 characters on paragraph boundaries.
5. If a single paragraph is still too long, use a fixed character split.
6. When splitting long bodies, prepend the last 200 characters of the previous chunk as overlap.

At last! An embedding store for semantic search. But anything worth engineering is worth over-engineering and so I thought about what a flex it would be if I gave my assistant an assistant - a subagent that didn't use tokens but was a local model that used GPU instead. For this endeavor, Qwen seemed perfect - It fit my machine, and ran quickly. What I didn't ask contributed heavily to this being a bad idea. This particular bad idea looked like this:

It's kind of embarrassing to reflect on. About halfway through I started doubting the approach, but I didn't stop because it was not a question of if I should build it, but whether I could build it. Which takes me down the path of every science-run-amok movie I've ever seen.

As you can imagine, there were all sorts of issues, but the main one was Qwen really really doesn't like to call tools, and when it does, it isn't great at collating what it gets back. So Qwen worked but wasn't effective.

This led me to an even worse idea:

This...wasn't better. Configuring all the tools and connectors to play nice was a pain for the limited value the solution provided. Qwen was still much harder to work with and didn't have enough context to know when it was pulling junk or how to turn that junk into useful data. Larger version of Qwen were terribly slow, and required more work to figure out how to optimize vs the value it was providing that was worth for this experimental project. It also really wasn't helping with token savings, however, the search tool was. I clearly was using the model incorrectly and the tools were the savings, not the model. It was time for Qwen to be shelved.

All that aside, the usefulness of the solution was becoming clearer. Semantic search used by a frontier model was making it avoid brute filesystem searching and the quality of searches improved greatly. I could ask "Which one of these talks map to the OWASP Top 10?" And get answers that never mentioned OWASP or Top 10. (because they had tokens that landed close to OWASP top 10 tokens.) but were still relevant to the OWASP top 10.

Where it Ended Up

The solution started looking much more like this:

I iterated on this process until I got answers that were satisfying for me, and I have a lot more testing to do. Most of my time went into tweaking the chunks and populating the vector store. This is also probably the area that is the most sensitve to different document styles. Denser text might be more forgiving and the overlap of 200 characters felt arbitrary. The code needs a lot more testing, and for my next iteration I'll consider a real markdown parser like mistletoe or markdown-it.

For embedding I used nomic-embed-text, served by ollama. It just worked, and worked quickly. It has to be quick, because the way its used is that the query terms are also turned into embeddings in order to then be used to search the vector store, which, because they're vectors can be measured as distance, so the search is based on proximity rather than direct matches.

This doesn't guarantee the chunks are relevant, and I found the model did a pretty good job of determining how well a given chunk contributed to the context around the query. I could see scenarios where relevance filtering could be helpful -- again, trying to be a token miser, I had another item for the issues list.

The semantic search mcp server is pretty simple. It's written in Python/FastMCP and provides just a few tools: search_similar, index_status, reload_index. The only 'search' tool was search_similar which takes a set of search terms and a limit. Since the limit is an LLM generated parameter, I did add hard min and max values. Theindex_status and reload_index are so I don't have to restart the mcp server when I update the database.

Despite relying on the frontier model to handle the searches and relevence, this method still is a huge token savings because its not forcing the LLM to read huge files, and the LLM doesn't choke on reading the files -- because it doesn't have to. Occasionally it still tries, which means a skill might be in order, or a better description on the tool. In a more elegant implementation, the model would use data from both searches to cross-search. That feedback loop has a name: hybrid search, where semantic results sharpen keyword queries, and keyword hits seed better semantic passes. But, at that point, I might just want to use something like Elasticsearch or OpenSearch. I'm told its massively overkill for something to search Obsidian files, which only encourages me to try it.

I have a whole list of issues to work through with this project before I jumpt at OpenSearch. The work here is still new and has to be tested more rigorously. The obsidian API uses a key which has to match the currently open vault, but as a user, I have multiple vaults open at once. This problem carries over to the vector store that has to match the obsidian vault or hilarity ensues. So, for every vault switch, there's an update to the API key, and the vector store location. This isn't insurmountable, but I'd want a simpler solution before I go too much farther.

Additionally, re-indexing is still a separate, offline process when I don't think it needs to be, but a re-indexing tool is probably not the right move -- not without guardrails on the LLM. Chunking parameters are still trial and error, and based on the type of text the system is working with. These all seem like interesting areas to explore.

Next time.