Setting up local AI means installing an open language model on your own hardware and making it reachable through a local interface. The fastest way in is Ollama on a Mac, done in a few minutes. If you want more control, LM Studio is the next step, and if you want a model running continuously under parallel load, you end up on your own server with vLLM or SGLang.
I now run my own model on a server with SGLang, after going through the same Ollama and LM Studio path this post describes. More on my setup, and when the effort is even worth it, is on the AI page and in Local AI: What Actually Works in 2026.
The three stages build on each other: each one costs more setup time, but buys you more control over speed, parallel requests, and resource use. You don't have to reach stage three. For plenty of tasks, Ollama alone is enough, and moving to your own server only pays off once a model really needs to run continuously and serve several requests at once.
None of the three stages requires prior systems administration knowledge. All four tools ship with a documented command line, and the commands below appear exactly like that in the respective official documentation. What actually differs between the stages isn't the difficulty of individual commands but the effort of operating, monitoring, and securing a model once it needs to stay reachable continuously instead of serving a single conversation.
Setting up local AI, stage 1: Ollama on a Mac
Per the official documentation, you install Ollama with an install script, then pull a model and start it right in the terminal:
curl -fsSL https://ollama.com/install.sh | sh
ollama pull gemma4
ollama run gemma4
One note on these commands: gemma4 is only an example model here, small enough for a first try. My own setup does not run on Ollama but on SGLang with Qwen 3.8, which comes later in this guide. The Ollama commands are good for getting started and for quick tests, not as a picture of my production environment.
ollama pull downloads the model once, ollama run starts a chat in the terminal and pulls the model automatically if it's still missing. If you're running things in the background, ollama ps shows what's currently loaded, ollama stop gemma4 frees it again, and ollama rm gemma4 removes it from disk entirely. If you want to save a model with your own default settings, ollama create -f Modelfile builds a custom variant.
For getting started, that's all you need: small to mid-sized open models run fine on a current Mac with an Apple Silicon chip even without a dedicated graphics card, since the system memory is shared directly with the model. Ollama is deliberately kept lean, a single command takes you from download to your first chat, which makes it the obvious first step when setting up local AI without first learning about quantization formats or serving frameworks. Full command details are in the Ollama documentation.
Setting up local AI, stage 2: LM Studio for more control
LM Studio ships with a graphical interface plus its own command-line tool called lms. With it you load a model with specific settings and, if needed, start a local server for other programs to connect to:
lms get
lms load --gpu=max --context-length=32768
lms server start
lms get opens model search, lms load loads an already downloaded model with the GPU and context settings you choose, lms ps shows what's currently sitting in memory, and lms server start opens a local API that other tools can connect to. The main difference from Ollama is control: the interface shows you directly how much memory a model uses, you can set context length and GPU usage per model, and you can keep several variants of the same model loaded side by side.
I personally use LM Studio on a MacBook Pro M3 Max to run a 27-billion-parameter model with 8-bit quantization and about 27.5 GiB of memory, where I get 11 tokens per second. That's plenty for occasional tasks and for testing whether a model even suits a given purpose; for continuous operation with several requests at once, it gets tight. The full command list is in the LM Studio CLI documentation.
Setting up local AI, stage 3: from Mac to server
Once several requests need to be served at the same time, or a larger model needs to run continuously, a dedicated server with a proper graphics card beats keeping a Mac running around the clock. The reason is architectural: Ollama and LM Studio are built for a single user, one request at a time. vLLM and SGLang are serving frameworks, built to run many requests efficiently on the same graphics card at once, with their own memory management for each request's context.
You install and start vLLM per its quickstart guide like this:
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen2.5-1.5B-Instruct
The server then runs at http://localhost:8000 by default and can be adjusted with --host and --port. You install and start SGLang like this:
pip install uv
uv pip install --prerelease=allow sglang
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
I run my own model under SGLang because it was the official cookbook recipe for the checkpoint I use, including speculative decoding with a dedicated draft model. I tested vLLM as an alternative and got 24 to 26 tokens per second. Both frameworks are built for continuous operation, and both need considerably more setup work than Ollama or LM Studio, but in return you get an API that several programs can use at once, instead of a single local chat window. Full measurements are in Qwen 3.8 Locally: DGX Spark Experience. Official sources: the vLLM quickstart and the SGLang installation guide.
Ollama, LM Studio, vLLM, and SGLang at a glance
Behind the four names sit four different layers: Ollama and LM Studio are wrappers around an inference engine, built for a single machine with as little setup effort as possible. vLLM and SGLang are serving frameworks, built to make efficient use of the same graphics card for many simultaneous requests, with their own memory management for context and intermediate results per request. LM Studio differs from Ollama mainly through its graphical interface and finer control over GPU and memory settings per model, but technically both remain tools for one user at a time.
| Tool |
Layer |
Best for |
Setup time |
| Ollama |
CLI wrapper |
Fast start, single user |
Minutes |
| LM Studio |
GUI plus CLI |
Testing and comparing models |
Minutes |
| vLLM |
Serving framework |
Parallel requests on a server GPU |
Hours |
| SGLang |
Serving framework |
Continuous operation with speculative decoding |
Hours |
The rule of thumb I use myself: as long as I'm the only one querying a model, Ollama or LM Studio is enough. Once an agent, a script, and I all want to reach the same model at once, or once a model needs to be reachable around the clock, I move to a serving framework on a dedicated server.
How you know the setup actually works
Before putting a model to real use, test three things: a short summary of one of your own texts, a simple coding task with a known correct answer, and the same question once with high and once with low thinking depth. Many models give noticeably better answers at maximum thinking depth, but on normal hardware that can take many times longer, sometimes several minutes for a single answer. I turn thinking depth down by default and only raise it for tasks that genuinely need it. A second test is worth running for the context window: feed the model a longer text of your own and check whether it still reliably references details from the beginning near the end, many open models get noticeably less reliable as they approach their stated context limit. A third test is worth running for tool use, if you plan to connect the model to an agent later: have it solve a simple task with a single tool, such as reading and summarizing a file, before you hand it something more complex with several tools.
Mistakes I made along the way
The mistake that cost me the most time concerns the memory share for the model: on hardware where CPU and GPU share the same memory, SGLang's --mem-fraction-static should never go above 0.50, or the whole host can freeze because nothing is left for the operating system. Why that is particularly touchy with unified memory is in DGX Spark Tested: Worth Buying? Because I don't have passwordless admin rights on my server, the service runs as a systemd user unit with linger enabled, so it keeps running even without an open session. And the internet connection at my location only manages around 2 MB per second, so downloading a 70 GB model takes several hours, which you should factor into your schedule before deciding to pull a new model in the evening. The remaining pitfalls from my own setup, from start-up time to Docker's storage cap, are in Qwen 3.8 Locally: DGX Spark Experience.
Connecting a terminal agent instead of just chatting
Ollama, LM Studio, vLLM, and SGLang all expose an API that behaves like the OpenAI or Anthropic one. That means you can point a terminal coding agent like Pi at your local model instead of only chatting with it: the agent sends its requests to the local address instead of a cloud endpoint, everything else stays the same. I do exactly that to compare my local setup against Claude Code, which runs my AI workforce. It's meant for clients whose contracts rule out the cloud, not as a replacement for my workforce. In practice that means the same agent, the same command-line tool, the same tasks, just a different address in the configuration and no per-request API cost.
Frequently asked questions
Ollama or SGLang: which fits for getting started?
For getting started and occasional use, Ollama is plenty and is set up within minutes without any server experience. SGLang only pays off once a model needs to run continuously and serve several requests at once, and it takes more time to set up in return.
Do you need a dedicated graphics card?
Not for smaller models; a current Mac with plenty of memory works fine through Ollama or LM Studio. For larger models running continuously, you need a dedicated graphics card on a server, or a comparable unified-memory architecture like the one I use myself.
Does vLLM run on a Mac?
Per the official documentation, vLLM is built mainly for servers with Nvidia graphics cards. For a Mac, Ollama and LM Studio are the more practical route; vLLM and SGLang belong on a dedicated server with the right hardware.
How do you turn off cloud features in Ollama?
Ollama runs locally by default and only sends requests to your own model on your own hardware. Extra integrations that reach out to external services are optional and can simply be left unused if you want to work purely locally.
How do you connect a terminal agent to the local model?
You point the agent at the local server address, for example http://localhost:11434 for Ollama or http://localhost:30000 for SGLang, instead of a cloud endpoint. Most terminal agents that speak the OpenAI or Anthropic interface can be redirected this way without major changes.
How much disk space do the models need?
That depends on size and quantization: a model with roughly 27 billion parameters runs anywhere from about 17 to nearly 56 GB depending on compression. Budget extra space for additional models you download to compare, and for container images if you run vLLM or SGLang through Docker.
Where to go from here
If you first want to check whether the effort is worth it for you, start with Local AI: What Actually Works in 2026 and its cloud-versus-local decision aid. For the cost side of hardware and power under continuous operation, see Your Own AI Server: Hardware and Cost.
My advice from setting this up myself: start with Ollama on a Mac you already own before investing in server hardware. Only once you hit the limits of stage one or two, because a task needs to run continuously and in parallel, does the jump to vLLM or SGLang pay off, with the effort that brings with it.
