Local AI: Ollama, self-host and real selection criteria
Local AI vs cloud: privacy, TCO, quality, latency, SLA. Ollama for dev/prototype; multi-user prod is another stack. Decision grid and security checklist for SMEs.
Why local is back
Data sovereignty, predictable cost, latency, air-gap: local returns when generic cloud rubs security policy or scale budget.
Not absolute cloud vs local. Which workload deserves which isolation. An agent reading client contracts is not the same as a marketing rewrite tool.
Local stack in practice: hardware (often 16GB+ RAM minimum for useful models), runtime (Ollama, LM Studio, or multi-user engines like vLLM in prod), model, network isolation.
Real limits vs frontier cloud
Quality gap: small local models often sit clearly below frontier models on multi-step reasoning and complex French. Measure on your 30 business cases, not a leaderboard.
Speed: local inference can be much slower than cloud APIs depending on GPU/CPU. In product iteration, latency changes work rhythm.
SLA: cloud offers availability guarantees; local is your hardware, patches, monitoring. For a multi-user app with a customer SLA, laptop + Ollama is not prod.
Ollama (and peers) excel for dev, prototype, bounded internal load. For multi-user prod, look at throughput-oriented runtimes (e.g. vLLM) plus ops.
Local vs API decision grid
Prefer local: highly sensitive data, high predictable volume, offline/air-gap, contractual hosting, exploding API cost.
Prefer API: max quality, sporadic use, time-to-market, load spikes, team without ML ops, need for frontier models.
Common hybrid: classify / preprocess locally; complex generation via EU API after PII masking. Document the routing matrix.
12-month TCO: hardware + power + ops time + downtime vs API bill. Local free never is.
Security and ops checklist
Data classification, model choice, network isolation (no public endpoint without auth), access logs, least privilege, model updates with business validation, failover if local saturates.
Containers help pin versions and isolate networks. Not a wand if secrets stay messy.
Even locally, prompt injection via a document can trigger tools. Bound tools like in the cloud.
If you want a local vs API roadmap, we can scope in 20-40 minutes.
FAQ
- Is Ollama enough for SME production?
- For bounded internal uses, sometimes. For a multi-user customer SLA: monitoring, HA, suitable runtime, ops ownership.
- Does local AI auto-comply with GDPR?
- No. Location helps, but legal bases, minimization, rights and security remain.
- Which local model?
- The one that scores best on your 30 business cases under VRAM and latency. Ignore hype of the latest release if your tests disagree.
- Mix Claude API and local?
- Yes, often optimal. Route by sensitivity and complexity. Document the matrix.
- When to avoid local?
- Sporadic use, need for frontier quality, no ops, strict SLA without a team to operate it.
Scope your first AI agent
20 minutes to review your tools, data and the first useful case. No jargon, no commitment.