Self-hosted AI: run your own LLM at home
A chatbot that runs on your own hardware never sends your questions to anyone.
Self-hosted AI means running an open-weight language model on your own computer instead of sending prompts to a company's servers. The easiest setup is Ollama, which downloads and runs the models, plus Open WebUI, which gives you a ChatGPT-style chat page in your browser. The model has to fit in memory: small models are 1 to 6 GB downloads that run on an ordinary 16 GB machine, and a GPU with enough memory makes them faster.
Why run an LLM locally
Everything you type into a hosted chatbot goes to that company's servers, and what happens to it next is whatever their privacy policy says this year. A local model processes the prompt on your machine. Ollama's FAQ says: "Ollama runs locally. We don't see your prompts or data when you run locally."
Ollama now also offers cloud-hosted models. When you pick one of those, your prompts go to Ollama's servers, which it says it processes but does not store or train on. If you want local only, Ollama has a setting that turns the cloud features off.
A model runs without a graphics card, but Ollama's own docs say that for best performance you should avoid offloading the model to the CPU. On CPU-only hardware, stick to the smallest models.
How much RAM you need to run an LLM
The model has to fit in memory. Ollama loads it into system RAM when it runs on the CPU, or into the graphics card's memory (VRAM) when it runs on a GPU. If a model only partly fits on the GPU, Ollama splits it, and the CPU half slows everything down. The download size is a good first estimate of the memory a model needs, before the conversation itself.
These are download sizes from Ollama's model library as of November 2026.
| Model (Ollama tag) | Download size | Where it fits |
|---|---|---|
| llama3.2:1b | 1.3 GB | Almost anything, including a CPU-only box |
| llama3.2 (3B) | 2.0 GB | An 8 GB laptop or a home server |
| qwen3 (8B) | 5.2 GB | A 16 GB machine |
| gemma4 (default tag) | 6.6 to 9.5 GB | A 16 GB machine |
| qwen3:14b | 9.3 GB | A 16 GB GPU or a 32 GB machine |
| gpt-oss:20b | 14 GB | Ollama says as little as 16 GB of memory |
| qwen3:32b | 20 GB | A 24 GB GPU or a 32 GB Mac |
| gpt-oss:120b | 65 GB | Built to fit on a single 80 GB GPU |
Leave room for the conversation. The context window is the model's working memory, and Ollama's docs warn that a larger context needs more memory. By default Ollama picks a context length based on how much VRAM your GPU has, and smaller cards get a shorter one.
On GPUs, Ollama supports NVIDIA cards with compute capability 5.0 or newer, many AMD Radeon cards through ROCm and Vulkan, and Apple silicon Macs through Metal.
Setting up Ollama and Open WebUI
- 1Install Ollama on the computer or server that will run the models. Ollama's download page has the current installer for macOS, Linux, and Windows.
- 2Pull and test a small model such as llama3.2, about 2 GB, and chat with it in the terminal.
- 3If you want local only, turn off Ollama's cloud features as described above.
- 4Install Open WebUI by following its quick start. When Open WebUI runs in Docker, Ollama has to listen beyond the machine itself, which opens port 11434 to your local network, so never forward it on your router.
- 5Open Open WebUI in your browser and create an account. The first account becomes the administrator and sign-ups close after it. Open WebUI's docs say to finish this step over a private network, a VPN, or a local port before you expose it anywhere.
- 6Pick your model in a new chat and start asking questions.
On a home server OS such as StartOS you can skip the commands and install both as apps. A home server with 16 GB or more of RAM is the place I'd put them.
Using it from your phone without exposing it
Ollama listens only on the machine itself by default. You can change that, but do not open it, or Open WebUI, to the internet, because neither is meant to face strangers.
Put the server on a private VPN instead. Tailscale connects your phone and laptop to the server without opening any ports on the router. Open the Open WebUI address in your phone's browser and it works like any chat app.
Which model to start with
Pick the model by how much memory you have. On 8 GB of RAM, start with llama3.2. On 16 GB, try qwen3 or gemma4, and gpt-oss:20b if you can spare nearly all of it. With a 24 GB GPU or a 32 GB Apple silicon Mac, qwen3:32b fits.
Expect a local model that fits on home hardware to know less than the biggest hosted models. Use it for drafting, summarizing your own documents, and questions you would rather not hand to a company. Check anything that matters, the same way you would check a hosted chatbot. Open WebUI can also connect to OpenAI-compatible cloud APIs, but the moment you pick one of those, your prompts leave the house again.
Licenses vary by model. OpenAI's gpt-oss models, for example, are Apache 2.0. Read the license on each model's library page if you plan to build anything on top of one.
AI and your bitcoin
A local model is a good place to ask how a PSBT works or what a wallet error means, because the question never leaves your house. Never type your seed phrase into any chatbot, hosted or local, any more than you would type it into a website. Keep it on paper or metal, and see my seed storage picks for the options.
The box that runs your AI can also run your Bitcoin node. If you are already buying RAM for models, the main extra cost of the node is a large disk. My node guide covers the rest.
Frequently asked questions
Can I run an LLM without a GPU?
Yes. Ollama runs models on the CPU from system RAM when there is no supported GPU. It is slower, and Ollama's docs say to keep the model on the GPU for best performance, so stick to small models such as llama3.2 on CPU-only hardware.
How much RAM do I need to run an LLM locally?
You need at least the model's download size, plus room for the conversation and everything else on the machine. A 2 GB model like llama3.2 runs on 8 GB of RAM. Ollama says gpt-oss:20b, a 14 GB download, runs on systems with as little as 16 GB of memory.
What is the best self-hosted LLM?
The best one is the largest model that fits fully in your memory. For a 16 GB machine that means models of roughly 5 to 10 GB, such as qwen3 or gemma4. Bigger models know more but slow to a crawl once they spill out of GPU memory.
Is Ollama private?
For local models, yes: Ollama says it does not see your prompts when you run locally. Its optional cloud models do send prompts to Ollama's servers, so turn its cloud features off if you want to rule that out.
Is self-hosted AI free?
The software is free: Ollama is MIT-licensed, and Open WebUI is free to run under its own license, which bars removing its branding except on deployments with 50 or fewer users in any 30-day period, with written permission, or under an enterprise license. The models download free from Ollama's library. You pay for the hardware and the electricity.