By Chuck Spencer, AIQ Solutions Principal Software Engineer
Our first blog in this series laid out what Ziggy is and why we built it. That’s all fine and good, but how we built it is where the real fun is. Ziggy’s core is comprised of commonly used, readily available open-source tools on unremarkable hardware and some simple code. Anyone with a little coding know-how, a machine with a GPU and a free day should be able to set up something equivalent (especially with Claude’s help).
This post is a description of our system, along with some sample code illustrating how the main pieces work. It’s a simple, concrete example that we hope people will find helpful.
Ziggy’s Heart
Most of what Ziggy is about is generating and serving up the “corpus”. The corpus is the curated, formatted body of knowledge collected across all the information sources Ziggy is pointed at (Notion, Jira, GitHub, etc). While there’s some AI used to make it searchable, the corpus itself is not AI. It’s just a fancy filesystem database that LLMs can use to access the knowledge within.
Our Machine
OS: Ubuntu 22.04 LTS
Processor: Intel i7-4790 CPU
Memory: 32GB
GPU: NVIDIA Titan-RTX with 24GB VRAM
When we wanted to start experimenting and prototyping, we set stuff up on an old server with an old GPU. Why? Mostly because it was available. We also wanted to get a sense of what the hardware bottlenecks and constraints would look like. The plan was to migrate Ziggy to a “real” server once we started hitting limits, but that hasn’t happened yet. More VRAM would let us run bigger models, which would of course be nice.
The point here being that you don’t necessarily need to shell out for some massive dedicated AI server.
The Models
Ollama to run the models. Takes 5 minutes to install.
Embedding Model: nomic-embed-text
Generative Model: Gemma4 – General
Ziggy runs two LLM models. One, the “embedding” model, is used to digest and index our source data into the vector index that Ziggy uses. It’s pretty small. The other is a general-purpose generative model for chatting and coding. If we had more VRAM, we’d probably add a coding specific model. These models and many more are freely available on Hugging Face and Ollama’s libraries.
The User Interface
Open WebUI to provide a front-end for interacting with the models. This tool provides a rich interface with real user accounts and authentication over an encrypted connection. Ollama’s API port itself doesn’t have any authentication, so it’s important to provide access through a tool like Open WebUI rather than exposing Ollama itself, even on the local network.
The Code
The code that glues it all together is simple and straightforward. Mostly vanilla python. We’ve put up a repo with example code that captures the broad strokes. The main moving parts are as follows:
Retrieve the Content
scripts/ingest_jira.py
Pull it Down
Run a script on a schedule to pull down content from your information sources via API tokens. The example project describes connecting to Jira, but the same pattern can apply to other sources such as Notion or GitHub. These platforms all provide documentation on how to set up these sorts of API tokens.
Take the time to set the API tokens up carefully. They usually just need read access, and that access should be restricted to the content you want the AI to see. Take the usual measures to make sure the tokens stay secret. Don’t check them into a repo, don’t write them on a sticky-note on your monitor, etc.
Keep It Clean
Before saving anything retrieved, the ingest script runs code from scripts/guard.py to try to preemptively block sensitive content that shouldn’t end up in the corpus. Credentials, PHI, personal employee info, that sort of thing. It’s best to keep this content out by restricting the scope of the AI’s access to the source. This should be a failsafe to catch stuff that leaked through.
Format It and Save It
The ingest script creates markdown documents corresponding to the source information. These markdown docs include a “frontmatter” header tracking things like where the information came from and how recently it’s been updated. These headers can be expanded to hold further information to help the AI categorize and evaluate the content.
Make it AI-searchable
scripts/corpus.py
Now it’s time to make the corpus index, the piece that lets LLM clients search all this information efficiently. The corpus.py script holds code that chunks the content up into passages and embeds the information into the vector index. It then provides an interface to search the content in three different ways:
Keyword: A lexical keyword search of the chunked passages
Vector: Use the embedded vector index to search by meaning
Hybrid: Combines both keyword and semantic search and merges the results
Sanity Check
scripts/eval_retrieval.py
None of this is worth anything if we can’t trust the results. After the corpus is updated, the eval_retrieval.py script tests it by running a set of “golden queries” (eval/golden_queries.yaml) and verifying that the expected results come back. This is critical in ensuring search keeps finding the right answers as the corpus changes. The value of this step is predicated on the quality and accuracy of the golden queries, so plan on investing time to develop and maintain those.
Serve It to the AI
scripts/mcp_server.py
The mcp_server.py script is a simple example of a MCP service. MCP (Model Context Protocol) is the standard open-source protocol for connecting LLM clients to data sources. Serving up our corpus via MCP enables AI assistants to integrate with it. The example runs on the same machine as the client.
You can connect the generative LLM model you’re hosting to this MCP service (our Gemma4 model, in our case), then Bob’s your Uncle. In addition to the chat interface over Open WebUI, users can also leverage this LLM using open-source code assistant plug-ins like Cline.
It is possible to connect commercial LLMs like Claude and ChatGPT to an MCP service like this. It’s important to note, however, that in this scenario information returned by the service goes to that company’s servers. Here, that’s a handful of passages quoted from the corpus, each citing where it came from, rather than whole documents, and only content that got past the guard and the access you granted at the source. While it’s certainly not as big a risk as it would be to point these commercial services directly at your source data, it’s a risk you’d need to evaluate.
Where to Start
Get yourself a Linux machine with a sufficient GPU. Install Ollama, Open WebUI, an embedding model and a generative model. Glom it together with code akin to what’s in the example repo and some shell scripts, configure your information sources, and go from there. By the time you get some minimal set of working pieces up and running, you’ll likely already have tons of ideas about where to go next.
In our next post I’ll describe how we use this system to maintain our self-updating, cross-domain code docs.