A personal project. Full-stack agentic AI infrastructure, running on bare metal, owned entirely by me.
System Zero (internal codename: SYS/00) is a self-hosted, multi-service agentic RAG platform I've been building since late 2025. It runs on a k3s Kubernetes cluster spanning an OVH bare metal VPS and a home machine, and is secured end-to-end with Cloudflare, Tailscale, and SSH key-only access.
The short version: it's a platform that lets you upload documents, ask questions about them, and have an LLM reason over the answers using a ReAct-style agentic loop. All running on infrastructure I own, control, and have had to defend from attackers.
The longer version is this post.
Most AI engineers use managed APIs for everything. OpenAI here, Pinecone there, a SaaS vector store, a hosted LLM endpoint. That's fine for production products but it means you never really understand what's happening underneath, and you're entirely dependent on someone else's infrastructure and pricing.
I wanted to understand the full stack. The document parsing, the chunking strategy, the embedding model, the hybrid retrieval, the async message bus, the inference server, the streaming gateway, the auth layer, the Kubernetes networking. All of it. System Zero is what that curiosity produced.
The system is a collection of independently deployable FastAPI microservices, orchestrated on k3s, communicating via RabbitMQ. Here's the full service map:
User message
↓
SSE Gateway → streams real-time events to React frontend via RabbitMQ
↓
Reasoning Service → LLM orchestrator, ReAct loop
↓ calls tools via MCP
MCP Gateway → aggregates all service tools into one endpoint
├── Search Service → hybrid BM25 + FAISS semantic search
├── Text2SQL Service → natural language to SQL (scales to zero via KEDA)
├── Database Service → vector store and document CRUD
├── Context Service → conversation history for multi-turn reasoning
├── Schema Builder → LLM-driven OLAP star schema design
└── Upload Service → file ingestion entry point
↓ async via RabbitMQ
Ingestion Service → scales to zero, heavy document processing
Embedding Service → sentence-transformers model server
One LoadBalancer (Traefik) for the entire cluster, with path-based routing to ClusterIP services, which means only one cloud LB is provisioned rather than one per service. KEDA scales the Text2SQL and Ingestion services to zero when idle. They are the heavy RAM consumers and don't need to run continuously. RabbitMQ handles all async communication between Upload and Ingestion, and between services and the SSE Gateway. The Context Service stores conversation history so the Reasoning Service can do genuine multi-turn reasoning across sessions.
When you upload a document, this is what happens:
File upload (PDF, DOCX, XLSX, CSV, TXT)
↓
Upload Service → saves to shared PVC, publishes job to RabbitMQ
↓
Ingestion Service (wakes from zero via KEDA)
↓
Docling → structure-aware document parsing (OCR disabled)
↓
Semantic chunking → similarity-gated, structure-preserving
↓
RabbitMQ "embed" routing key (RPC) → Embedding Service → embeddings returned
↓
RabbitMQ "enrich" routing key (async) → Proposition Service → stored to DB
↓
Database Service → FAISS (3 indices: general, summary, detailed) + text storeThe chunking is structure-aware rather than naive character-count splitting. Docling preserves document hierarchy (headings, tables, lists) and the semantic chunking respects those boundaries. The Proposition Service is a central embedding hub running all-MiniLM-L6-v2, handling both the synchronous embedding requests (RPC pattern, response required) and the asynchronous enrichment storage (fire-and-forget).
When a query comes in, the Search Service runs hybrid retrieval combining two fundamentally different approaches.
FAISS dense vector search: embeds the query, finds semantically similar chunks by cosine similarity. Good at understanding meaning and synonyms. Weak on exact terminology and proper nouns.
BM25 sparse retrieval: keyword frequency scoring. Excellent at exact matches, specific terms, and document names. No understanding of meaning.
Running both and merging the results with score normalisation gives retrieval that handles both semantic intent and precise terminology. The Search Service also supports MMR (Maximal Marginal Relevance) as an alternative to standard top-k. MMR optimises for both relevance and diversity, avoiding cases where the top 5 results are all saying the same thing with slightly different wording.
Vector search is powerful for unstructured documents, but a lot of real-world data lives in databases: tables, rows, schemas. Asking a question about a document is one problem. Asking a question about data that lives in a SQL table is a completely different one, and throwing FAISS at it produces nothing useful.
That's what the Text2SQL Service handles. It takes a natural language query, understands the schema of the target database, and generates the correct SQL to answer it. The result comes back as structured data which the Reasoning Service can then incorporate into its final response alongside anything retrieved from the vector store.
The Reasoning Service decides which tool to reach for depending on the query. If the question is about documents, it calls the Search Service. If it's about structured data, it calls Text2SQL. If the answer requires both (say, cross-referencing a report against live database figures), it calls both and synthesises the results. Neither system knows about the other. The intelligence of knowing what to reach for lives entirely in the ReAct loop.
Text2SQL runs as a KEDA-scaled service, meaning it scales to zero when idle. It's one of the heavier services to cold-start, but for a personal platform that's an acceptable trade-off versus having it consuming RAM around the clock.
This is the part that makes it genuinely agentic rather than just RAG.
The Reasoning Service runs an LLM (currently Qwen 7B, self-hosted) in a ReAct loop: Reason, Act, Observe, repeat. It doesn't just retrieve and generate; it decides what it needs, calls tools to get it, observes the results, and repeats until it can answer the question.
The platform supports two inference modes depending on what hardware is available. In CPU mode, the Reasoning Service runs entirely on the OVH VPS, with no GPU required, just system RAM and the right llama.cpp binary for the CPU architecture. In GPU mode, the Reasoning Service is deployed to my home cluster where the GTX 1080 lives, offloading model layers to VRAM for significantly faster inference. Switching between the two is a deployment target change. Same service, pointed at a different node.
This is particularly useful for cost and availability reasons. The VPS runs 24/7 and handles CPU inference fine for lighter loads. The home machine has the GPU but isn't always on. The architecture doesn't care which one is doing the reasoning. The rest of the cluster on the VPS stays the same either way.
This is something that catches people out when self-hosting: you can't just pick any model and expect it to work properly. You need to check your CPU architecture and supported instruction sets before choosing a GGUF quantisation. Get it wrong and you'll get garbled output, nonsensical responses, or silent failures that look like the model is running fine but producing nothing meaningful.
The key things to verify before deploying a model:
grep avx /proc/cpuinfo to see exactly what your machine supports before pulling anything.Before pulling a model, run lscpu, match the binary to your instruction set, and size the quantisation to your available memory. It's not glamorous but skipping this costs hours debugging what looks like a model quality problem when it's actually a hardware compatibility one.
The tool layer is exposed via MCP (Model Context Protocol). The MCP Gateway is where all tools live. It aggregates every service's capabilities into a single endpoint. The Reasoning Service connects to one place and gets everything.
Securing MCP properly matters. Any client that can reach the MCP Gateway can call tools, which means search your documents, query your database, trigger ingestion jobs. That's not something you want unauthenticated.
Keycloak handles all of this. Every MCP connection, whether from the internal Reasoning Service, Claude Desktop, or Claude.ai, authenticates via Keycloak before a single tool call goes through. The tokens are short-lived, scoped, and managed centrally. Adding a new client means issuing it a Keycloak client credential, not hardcoding a secret somewhere and hoping for the best.
The practical result: both Claude Desktop and Claude.ai connect to the same MCP Gateway and can trigger agentic workflows across the full platform, with auth handled cleanly at the gateway level rather than bolted on per service.
The React frontend (SYS/00) has a design intentionally inspired by the minimal, monospace aesthetic of dev tools and terminal UIs. The branding is built around a 3×3 dot matrix logo with a hollow centre void, animated on load with a clockwise dot pop, void breathing cycle, and radar ping.
The UI layout: collapsible left sidebar with chat history grouped by date, main chat area with soft-rounded message bubbles, collapsible right panel for retrieved sources that shrinks to a 32px strip when closed. The settings page covers Documents, Pipeline, Service Health, System configuration, and Profile in a two-column layout.
The frontend communicates with the backend via SSE (Server-Sent Events). The SSE Gateway streams reasoning tokens and tool call updates in real time as the ReAct loop runs.
Deployment stack:
Security:
In 2024 the VPS was compromised by a cryptominer. I noticed the anomaly in resource metrics, investigated, ran forensics (verified binaries, checked process trees, identified the entry vector) and then did a clean infrastructure rebuild from scratch rather than trying to patch around an untrusted environment. Every service was redeployed from verified source. The compromise is now documented and the entry point closed.
It's the kind of thing you only fully appreciate when you've actually had to deal with it. Managing real infrastructure at your own risk level, with no ops team to call, teaches you things that running on managed services never will.
I ran Locust load tests against the live platform and found a clear split in performance.
Search layer (FAISS + BM25): excellent. Sub-40ms medians, near-zero failure rates at 85+ RPS. The hybrid retrieval pipeline handles load well.
/auth/login showed an 18% failure rate with 3.9s median and p99 of 28 seconds under load, with cascading 401 errors on SSE endpoints. Root cause: connection exhaustion on the token write path. Fix: decouple token refresh from the main DB using a lightweight store (Redis), and treat SSE 401s as derivative of the refresh failures rather than a separate issue.Database endpoints that looked problematic in the first test run (503s at 26–52% failure rates) turned out to be cold-start artifacts. The second run with warm services showed near-zero failures across the board.
One thing worth highlighting: the embedding model isn't hardcoded. The entire embedding configuration lives in a Kubernetes ConfigMap, meaning switching from a general-purpose model to a domain-specific one is a config change with no code changes required.
If you're working in a medical context you might swap in BiomedBERT or PubMedBERT for clinical terminology. Education might call for a model fine-tuned on academic text. Legal, finance, code: all have specialist models that outperform general-purpose embeddings on domain-specific retrieval. The pipeline doesn't care. Swap the ConfigMap, redeploy the Embedding Service, and everything downstream gets better retrieval for your domain automatically.