Demo vs deployment
Getting a model to run locally takes minutes. ollama run gpt-oss:20b on a recent laptop gives
you a capable model with no account and no data leaving the machine. That is the demo, and it is genuinely
useful for a developer. A deployment is something else: a model that a team relies on, with an API other
systems call, access control, logging, an upgrade path when a better model ships in six weeks, and someone
on call when it falls over. Most teams that say they want "local AI" want one of these and budget for the
other.
Good reasons to run models yourself
- Data that must not leave. Source code under strict client contracts, health or legal records, anything a regulator or customer says cannot go to a third-party processor. Zero-retention API agreements cover a lot of this now, so check whether yours already does before buying hardware.
- Offline and edge. Field equipment, air-gapped networks, travel, factories. No network, no API.
- Predictable, high-volume inference. Classification, extraction, embedding or summarisation running all day on a model that is good enough. Fixed hardware cost beats per-token pricing when the machine is busy most of the time.
- Control over the model. Fine-tuning, pinning a version for reproducibility, or inspecting behaviour without a provider changing the model under you.
- Developer experimentation. Engineers learn how models behave faster when they can run, break and swap them locally at no marginal cost.
Bad reasons
- "It will be cheaper" at low volume. A workstation that sits idle overnight and at weekends costs the same as one that runs flat out. API providers keep their hardware busier than you will.
- Matching the frontier. The strongest open-weight models are enormous. Kimi K3 is a 2.8-trillion-parameter model; DeepSeek V4 Pro is about 1.6 trillion. What fits on one desk is a smaller model, and the gap shows on the hardest tasks.
- Avoiding a procurement conversation. A GPU box under someone's desk processing customer data is shadow IT with a power cable. It needs the same review as any other system.
The two numbers that decide everything
Memory capacity decides what fits. A model's weights have to sit in memory the accelerator can reach quickly. At 16-bit precision that is about 2 GB per billion parameters; at 4-bit, which is how most local models run, it is roughly 0.5 to 0.6 GB per billion. Then add room for the context (the KV cache), which grows with every token in the conversation. So a 120-billion-parameter model at 4-bit needs something like 65 to 75 GB before context, which is why 128 GB machines are the current sweet spot.
Memory bandwidth decides how fast. Generating each token means reading the active weights from memory. The upper bound on tokens per second is roughly bandwidth divided by the bytes read per token. That is why Mixture-of-Experts models matter so much locally: OpenAI's gpt-oss-120b has 117 billion parameters but only 5.1 billion active per token, so it generates far faster than a dense model of the same size. And it is why a machine with 1.2 TB/s of bandwidth feels very different from one with 273 GB/s, even when both can hold the same model. Real-world speed is lower than the bound, but the ranking usually holds.
Three tiers
1. The developer laptop
32 to 64 GB of memory runs models up to about 30 billion parameters well: gpt-oss-20b, which OpenAI says runs within 16 GB, Google's Gemma 4 family, Qwen 3.8 27B. Use it for code assistance with private repos, prototyping prompts and learning. Runtime: Ollama or LM Studio. Cost: hardware you probably already bought.
2. The team workstation
128 GB to 512 GB of fast memory in one box: a Mac Studio, an NVIDIA DGX Spark, a Ryzen AI Max machine, or a workstation with an RTX PRO 6000. This runs 100-billion-plus parameter models for a handful of concurrent users. Good for a team's private assistant, document search over sensitive material, or batch jobs. The hardware options and their listed prices are compared in best hardware for local LLMs.
3. The on-prem inference server
Multiple data-center GPUs, a serving engine built for concurrency such as vLLM, a gateway for authentication and quotas, and monitoring. This is a production system with an owner. At this tier the honest comparison is with a dedicated endpoint from a neocloud or a hyperscaler, which gives you the same isolation without the hardware.
Hardware is getting more expensive, not less
One thing that changed in 2026: memory shortages pushed local AI hardware prices up. NVIDIA raised the DGX Spark Founders Edition from $3,999 to $4,699 in February, citing memory supply, in a developer forum announcement, and further increases have been reported since. Budget with today's quotes, not last year's reviews.
A decision rule
- If the data can go to an API under a zero-retention agreement and volume is modest, use the API.
- If developers want to experiment, let them run models locally on laptops; it is cheap and builds intuition.
- If a team needs a private model daily, buy one workstation-class machine and measure usage for a quarter before buying more.
- If it becomes a production dependency, run it as a service with vLLM or rent a dedicated endpoint, and give it an owner.
In this cluster
- Best hardware for local LLMs — Mac Studio M5 Ultra vs DGX Spark vs Strix Halo vs RTX PRO 6000, with vendor-listed prices
- Ollama vs LM Studio vs llama.cpp vs vLLM — which runtime for a laptop, a team server and production serving
- Best local LLMs to run — gpt-oss, Qwen 3.8, Gemma 4 and the rest, by hardware tier and licence
- Neoclouds compared — the rent-instead option, with listed GPU-hour prices
- AI infrastructure — the other end of the scale: gigawatt AI data centers
Frequently asked questions
What is a local LLM?
A local LLM is a large language model whose weights you download and run on hardware you control: a laptop, a workstation, or a server in your own data center or private cloud. Requests never go to a model provider's API. Tools such as Ollama, LM Studio, llama.cpp and vLLM load the weights and expose a chat interface or an API, often compatible with OpenAI's.
Is a self-hosted LLM cheaper than an API?
Sometimes, at steady high volume. Hardware has a fixed cost whether you use it or not, while API pricing scales with use, and providers get better utilisation from their hardware than most teams will. Do the arithmetic with your real token volume, include power and the engineer time to run it, and compare against the API price of a model of similar quality, not the most expensive frontier model.
What hardware do I need to run an LLM locally?
For small models (up to about 30 billion parameters at 4-bit) a laptop or desktop with 32 to 64 GB of memory is enough. For 100 to 200 billion parameter models you need around 128 GB of fast memory, which means an Apple Silicon Mac with high unified memory, an NVIDIA DGX Spark, an AMD Ryzen AI Max machine or a 96 GB professional GPU. Anything larger needs 256 to 512 GB or several GPUs.
Are local LLMs as good as ChatGPT or Claude?
The best open-weight models are competitive with closed frontier models on many tasks, but they are also very large. The models that fit on a single workstation are a step below. For code completion, summarisation, extraction, classification and private document search they are often good enough; for the hardest reasoning and long agentic tasks, the gap is still noticeable.
What is the difference between local AI and on-prem LLM?
In practice, scale. "Local AI" usually means a model on an individual's machine for development or private use. "On-prem LLM" means a shared inference service on servers the company controls, serving many users with proper authentication, logging and capacity planning. The tools differ: desktop apps and Ollama for the first, vLLM or SGLang behind a gateway for the second.
Sources: model cards on Hugging Face (OpenAI gpt-oss, Moonshot Kimi K3, DeepSeek); NVIDIA developer forums; vendor specifications as linked from the spoke pages. Memory and speed figures are rules of thumb, not benchmarks.