Ollama: The Reference Architecture for Sovereign, Private Large Language Models (LLMs)

TL;DR

Artificial Intelligence (AI) is the new standard, but using cloud APIs like OpenAI (ChatGPT) or Anthropic comes with a significant catch: data privacy and “data gravity.” Sending sensitive company data, source code, or customer information to US servers is often a GDPR nightmare and a strategic risk. Ollama changes the game. It is an extremely lightweight engine to run powerful open-source models (like Meta’s Llama 3, Mistral, or Gemma) directly in your own cluster. By using Ollama, you get the full power of generative AI—without a single byte leaving your network.

1. The Architecture Principle: In-Cluster AI vs. Cloud API

When developers today incorporate AI features into applications, they usually use external APIs. This means every prompt, every context, and every uploaded PDF file leaves your infrastructure, travels through the internet, and is processed on a third-party server.

Ollama brings the brain to the data, not the data to the brain.

• Air-Gapped Capable: Ollama runs as a Container in your Kubernetes cluster. It requires no internet connection at runtime. The model (the “weights”) is downloaded once and resides on your storage.
• Zero Data Leakage: Since processing occurs locally on your worker nodes (ideally with GPUs), the setup is “by design” GDPR-compliant. You can feed internal contracts or unmasked patient data into the AI without violating Compliance guidelines.

2. Core Feature: The OpenAI-Compatible API (Drop-in Replacement)

The biggest hurdle for switching to local AI has been the code. Many apps are tightly integrated with OpenAI SDKs (e.g., in Python or Node.js).

Ollama elegantly solves this problem.

• API Compatibility: Ollama offers an API that looks and behaves exactly like the OpenAI API.
• No Code Rewrite: You don’t have to reprogram your application. You simply change the BASE_URL in your code from https://api.openai.com/v1 to the internal address of your Ollama service (e.g., http://ollama.ai-namespace.svc.cluster.local:11434/v1). The application “thinks” it’s communicating with ChatGPT, but in reality, it’s interacting with your private Llama 3 model.

3. Model Variety & RAG (Retrieval-Augmented Generation)

The open-source AI world is evolving rapidly. Today Model A is the best, tomorrow Model B. With SaaS providers, you are bound to their model cycles.

• Model Switch in Seconds: With Ollama, you can switch models on-the-fly. One command (ollama run mistral) is enough, and the new model is ready. You can use specialized, smaller models for coding, translations, or text summarizations.
• Perfect for RAG: If you want to build an internal AI that knows your company wiki (RAG), you need “embeddings” (text vectorization) alongside the LLM. Ollama provides specialized embedding models right out of the box. You can build your own vector database, completely sovereign.

4. Operating Models Compared: OpenAI API vs. ayedo Managed Ollama

Here it is decided whether AI becomes an incalculable ongoing expense or a scalable infrastructure asset for you.

Scenario A: OpenAI API (The Token Cost Trap)

Cloud APIs are convenient for prototypes but tricky when scaling.

• Pay-per-Token: You pay for every word that goes in and every word that comes out. In RAG systems, where you often send thousands of words of context per request, costs explode exponentially.
• Data Privacy Risk: Even if providers promise not to use API data for training, a residual risk remains, and for highly regulated industries (finance, healthcare, government), that’s often not enough.
• Rate Limits: Under high load, you hit API limits. Your application gets throttled by OpenAI.

Scenario B: Ollama with Managed Kubernetes from ayedo

In the ayedo App Catalog, Ollama is provided as a robust microservice.

• Infrastructure Flat Rate: You don’t pay per token. Whether you generate 100 or 10 million tokens, it costs you exactly the same (the operation of the nodes). With intensive use, your own (GPU) nodes pay off extremely quickly.
• Absolute Control: You decide which model version runs. No surprising “deprecations” of APIs that destroy your application overnight.
• Scalability: Through Kubernetes, Ollama can be horizontally scaled. More traffic simply spins up more Ollama pods.

Technical Comparison of Operating Models

FAQ: Ollama & AI Strategy

Is an open-source model as good as GPT-4?

For general, highly complex logic tasks, GPT-4 (or Claude 3.5) is often still slightly advantageous. BUT: For 90% of business use cases (summarizing texts, classifying support tickets, extracting data from JSON, RAG queries on internal documents), models like Llama 3 (8B or 70B) or Mistral are absolutely equivalent—and much faster and cheaper.

Do I necessarily need expensive GPUs (graphics cards)?

Not necessarily, but it is highly recommended. Ollama can compute smaller models (like Llama 3 8B) purely on the CPU, which is sufficient for simple background jobs (about 5-10 tokens per second). For interactive chat applications where the user expects real-time answers, nodes with NVIDIA GPUs (e.g., T4 or A10) are the standard in the ayedo cluster to guarantee lightning-fast inferences.

How large are the models on the hard drive?

Surprisingly small thanks to quantization (compression). A very capable 8-billion-parameter model often requires only 4 to 5 gigabytes of storage space. An extremely powerful 70B model is about 40 gigabytes.

Is there also a chat interface for employees?

Ollama itself is just the “engine” (the API). In the ayedo stack, we often combine Ollama with frontends like Open WebUI. This gives your employees an interface that looks and feels exactly like ChatGPT—only all data remains securely on your servers.