Like a lot of companies, we at AIQ have spent the last couple of years trying to navigate the hype and figure out real nuts-and-bolts applications for LLMs in our organization. As a medical device software start-up, we have a lot of reasons to be excited. Our team is small and nimble with a science and engineering focus. Just the right people to put AI tools to great use across everything we do.
Also, as a medical device software start-up, we have a lot of reasons to be cautious. Patient privacy, regulatory compliance, and trade secrets all demand that we protect certain information very carefully.
It’s taken us a while to thread that needle, but we’re excited about what we’ve come up with. We thought we’d share some of what we’ve learned.
The Constraints
We’ve chosen Anthropic’s tools due largely to their comparatively good reputation around data privacy. Still, we have certain categories of information that we decided we simply can’t allow through their servers. Policies and user agreements can change in murky ways. Even the best AI companies are vulnerable to hacks and leaks. The legal and regulatory implications in this space are far from settled. For our most sensitive information we prefer to avoid even these arguably small risks.
The key to getting value out of LLMs is context. The more an AI “knows” about our company, the more helpful it can be in all the various use cases across the business. There are a lot of tools out there for piping business information into LLMs. So, so many. More every day. Given our constraints, we can’t just start hooking things up to Claude willy-nilly. We need a way to leverage our information sources while maintaining control of what information goes where.
Cost is another obvious concern. We’re wary of creating dependence on tools whose costs may change and grow in unpredictable ways.
Meet Ziggy
What we landed on is a hybrid: private on-prem tooling alongside commercial AI. We stood up a set of tools and scripts on one of our secured servers. We named it Ziggy (those of us old enough to remember Quantum Leap may get the reference).
Ziggy has tightly controlled access to our privileged data. It does the slow, expensive, token-intensive grunt work of reading and aggregating from live sources like Jira, Notion, and GitHub. It evaluates and digests all of this information into a carefully structured vector index that’s refreshed nightly.
Local vs. Claude
Ziggy provides access to this information in two main ways: open-source LLMs running directly on Ziggy’s hardware, and connection to other AI tools via MCP (the standard protocol for connecting AI tools to information sources).
Using the LLM directly on Ziggy allows 100% local interaction with our data. In this context we can confidently work with even our most sensitive information without worrying about where it’s going. And there’s no per-token cost.
Ziggy’s MCP service is a gatekeeper that allows us to use Claude’s models and tools with our data without losing control of it. The MCP service running on Ziggy is carefully designed with guards and gates that moderate access to our information. MCP doesn’t just pipe prompts straight in. Ziggy filters requests through a fixed set of queries we define and maintain, and sends back appropriate information with appropriate specificity. Information in the index is marked up with metadata indicating its provenance, nature, and sensitivity. Ziggy uses this metadata to control what information goes out through that pipe, and to whom.
Using Claude Desktop and Claude Code with this MCP connection hits the sweet spot between ease and privacy. Our employees get to use powerful, refined, standard tools and still leverage our company data safely. Pulling this information from a locally hosted vector index is also far more cost-effective than having Claude repeatedly search the raw sources itself. The token savings are significant.
Bumps in the Road
It wasn’t all fun and games. Each time we integrated a new information source, we had a lot to reckon with. Not just how to integrate it into Ziggy, but how we should be managing and using that information. Do we need to write our Jira tickets differently? How do we consistently organize our Notion workspaces? What’s the authoritative source of truth for any given piece of information? Thinking about how an AI would comprehend this information forced us to take a hard look at how we’re using these systems and has already forced some changes. Ultimately that’s been a huge positive, but I wouldn’t call it fun.
It took some iterating and redesigning before we started seeing the results we wanted. Early iterations had trouble resolving related information occurring in (and sometimes conflicting between) multiple sources. Tedious as it was, this exercise garnered a lot of value.
That work led to a system of tiers and rules for information sources that help Ziggy prioritize which ones to trust more. When in conflict, a comment in meeting notes in Notion from three months ago doesn’t get the same weight as engineering documentation verified against last night’s code.
We also built a number of quality checks into Ziggy. We added staleness checks and a set of a few dozen golden queries with a known-correct source for each to validate against. These serve as a continuous backstop to keep Ziggy anchored to reality.
What Surprised Us
For all the thought we put into this, the most striking thing about it is how simple it turned out to be. We didn’t need to invest in expensive hardware or commercial software. We didn’t need to invest huge amounts of engineering effort.
The hardware demands are far less constraining than we’d initially feared. The existing open-source tools and knowledge being shared in the AI communities are robust and hugely useful. By combining these building blocks with some carefully thought-out processes and carefully designed scripts, we’ve been able to create a powerful tool that fits our company’s exact needs.
I plan to get into more technical weeds on this in some future blog posts. We’ve had to build our expertise in this space as we go. It’s new to us like it is to so many, and we can only spare so much bandwidth. Nonetheless, we found it approachable once we dug into it. As we roll it out, we’re excited to see what the team does with it.
It’s the wild west out there right now. Eventually the dust will settle, and more mature commercial and open-source solutions will emerge to solve these specific needs. But we don’t have to wait for that. Real solutions are well within the grasp of any decent engineering team, now. The tools and patterns are out there, they’re more cost-effective than you might expect, and the constraints are manageable with the right approach.