Local AI: Ollama, self-host and real selection criteria

Why local is back

Local AI is back on the table for four reasons: data sovereignty, predictable costs, latency, and sometimes a full air-gap requirement. In short, local becomes interesting again when generic cloud rubs against your security policy or your budget at scale.

The real question comes up workload by workload: which one deserves which isolation. An agent that reads client contracts does not have the same risk profile as a marketing rewrite tool.

In practice, a local stack comes together from four building blocks:

• Hardware sized for the model, quantization and context length
• Runtime: Ollama or LM Studio on a workstation, a load-appropriate engine for multiple users
• The model and its license
• Network isolation and access controls

Real limits vs frontier cloud

The first gap concerns quality. Small local models often sit clearly below frontier models on multi-step reasoning and complex French. Measure the gap on your 30 business cases, not on a leaderboard.

Speed matters too. Local inference can be much slower than cloud APIs depending on your GPU or CPU, and in product iteration, latency changes the pace of work.

Availability also matters. Some cloud contracts offer an SLA; check the service terms. Local operations depend on your hardware and organization. A development workstation alone does not provide production service guarantees.

Ollama and its peers excel for dev, prototyping and bounded internal load. For multi-user production, look instead at throughput-oriented runtimes like vLLM, with real ops behind them.

Local vs API decision grid

Before deciding, run your context through this grid. Each signal points to a reasonable default:

The hybrid setup remains common: classification and preprocessing run locally, and complex generation goes through an EU-hosted API after PII masking. In that case, document the routing matrix.

To settle it, compute the 12-month TCO: hardware, power, ops time and downtime on one side, the API bill on the other. Local "free" never is.

Security and ops checklist

Before putting a local model in front of users, run through this checklist:

• Data classification and model choice
• Network isolation, with no public endpoint without auth
• Access logs and least privilege
• Model updates validated on the business side
• A failover plan if local saturates

Containers help pin versions and isolate the network. They fix nothing, however, if secrets stay poorly managed.

Also keep in mind that a prompt injection slipped into a document can trigger tools, even locally. Bound your tools exactly like in the cloud.

If you want a local vs API roadmap for your context, we can scope it together in 20-40 minutes.

Frequently asked questions