Local AI agents work today: a terminal agent that reads files, edits them, and runs commands can drive an open model with local inference on your own hardware. In my case that's Pi talking to Qwen 3.8 on a DGX Spark, and it reads the same rules from CLAUDE.md as my Claude-based workforce does. What doesn't work yet is rebuilding an entire AI workforce locally, one to one, with several roles, handovers, and approval gates. The local harnesses are missing an approval system above all, then reliability across many steps, and, if you configure them carelessly, simply enough context. This post walks through the three building blocks you need, four harnesses that connect to a local model, and the exact spot where the limit currently sits.
I run my business with an AI workforce built on Claude, every role a folder with a CLAUDE.md, a personnel file, skills, and learnings. Since December 2025 a DGX Spark has been sitting next to it, where I try the same way of working with a local model. The goal: a Claude Code alternative without cloud for clients who don't want to hand their data to a provider. Everything on local AI and the workforce is collected on my AI page.
What a local AI agent needs: three building blocks
An AI agent is a model that calls tools: read a file, write a file, run a command, read the result, continue. Locally, you need three things for that, and all three have to fit together.
First, a model that can do tool calling. It has to emit a tool call in a fixed format the harness understands. Qwen 3.8 can, according to its model card, along with thinking and image input. A model without tool calling will write you text but never touch a file.
Second, a server that exposes the model through an API. For me that's SGLang on the Spark with an OpenAI- and Anthropic-compatible interface; Ollama or LM Studio are easier to start with. How to set that up is covered in Setting Up Local AI: Ollama to SGLang, and for Ollama specifically in Installing Qwen 3.8 with Ollama.
Third, the harness: the program that provides the tools, manages the context, reads your rules, and runs the loop of model reply and tool call. Claude Code is a harness. Pi, OpenCode, and Aider are harnesses that can be pointed at a local model. The harness decides, more than the model does, whether a chat turns into an employee.
Four harnesses for running AI agents locally: connection, tools, approvals
The table sums up what each project's documentation says about running with a local model, as of September 16, 2026. Of the four I have only tested Pi; the other three are listed because their docs describe the path to a local model.
Two notes on the table. For Claude Code against Ollama, the docs say nothing about whether Anthropic's terms of use cover the route; check that yourself, and I don't run it. And the "Approvals" column is the one that matters most for anything beyond a test project: only Claude Code asks on its own before it runs a command or overwrites a file.
The context trap: 4,096 tokens is not enough
The most common mistake with a first local agent sits in the context window, not in the model. Ollama starts models with 4,096 tokens of context according to its FAQ; the Aider docs even cite 2,048 as the default, so the two sources disagree. For a chat that's fine. An agent, though, pushes the system prompt, tool definitions, your CLAUDE.md, every file it reads, and every command output into that same context. At 4,096 tokens that budget is gone fast, depending on your prompt, file sizes and tokenization: in that order of magnitude the beginning tends to drop off after the first or second file, and tool calls become unreliable. That's exactly what the OpenCode docs describe: if tool calls aren't working, raise num_ctx to 16k to 32k. Ollama recommends 64k or more for Claude Code, and Aider raises the value automatically and lets you pin it.
For comparison, from my workforce: Conny's learnings file has 1,347 lines (as of September 14, 2026), and it is read at the start of every session before the first task begins. I haven't measured that file's token count, and the line count alone proves nothing; as a rough estimate, at an assumed 10 tokens per line it would be about 13,000 tokens, several times 4,096. Qwen 3.8 brings 262k of context according to its model card; what matters is that the server actually reserves that context, and that costs memory.
Can a whole AI workforce run locally? Three layers
This is where "does a local agent work" separates from "does my workforce work locally". My employees consist of three layers, and they transfer with very different success.
Layer 1: files. Every employee of mine is a folder: CLAUDE.md as the contract, personalakte.md (the personnel file) with the assignment and access rights in three tiers (free, only after approval, never), .claude/skills/ with one skill per task, wissen/learnings.md as memory, postfach/eingang and postfach/ausgang as an inbox and outbox for handovers. That's plain text, and Pi reads CLAUDE.md and AGENTS.md from the project, picks up skills from .pi/skills/ and .agents/skills/, and has subagents. This layer transfers, and it's the reason I build roles as files. How I use CLAUDE.md is in CLAUDE.md: Claude Code's Memory; how the workforce is put together is in My AI Workforce: Eight Employees.
Layer 2: what the model can do. Peter, my bookkeeping employee, processed 63 receipts in September 2026: twelve commands executed, two of them failed, the check net plus tax equals gross done on all 63, and two questions back to me before producing the output, because one sentence in the assignment was cut off and two currencies showed up. I decided: "Everything in euros, at the daily rate." Whether a local model drives that chain just as cleanly is something I can't back with numbers for Qwen 3.8; my comparison with Claude Code has been qualitative so far. The arithmetic behind it is uncomfortable: if every tool step succeeds 95 percent of the time, the hit rate over ten steps is around 60 percent. For long, unattended runs the cloud model stays ahead for me, as I describe in Qwen vs Claude: Local Versus Cloud.
Layer 3: control. With Claude Code the tool asks before every sensitive step; I've wired that through hooks into my employee OS, so I can grant approvals outside the terminal as well (see Claude Code Hooks: Rules That Always Apply). Pi has no permission system, it runs everything without asking. For a workforce where every task ends at an approval gate (send the report to the client, put the website live, spend money), it is currently a deal-breaker: the gate then has to live outside the harness, for example as a review pass by a separate role before anything leaves the house.
My setup: Pi driving Qwen 3.8 on the DGX Spark
I installed Pi locally and pointed it at the Spark, where Qwen 3.8 runs under SGLang. That worked surprisingly well: the agent called its tools cleanly and worked in my folders without much fiddling with the setup. Pi works like Claude Code with Read, Write, Edit, Bash, and Grep, reads AGENTS.md and CLAUDE.md, has sessions, an MCP adapter, web search, subagents, a plan mode, and a todo list. The hands-on report on the tool itself is in Claude Code Alternative Without Cloud, the measurements for the model in Qwen 3.8 Locally: DGX Spark Experience.
What I do with it today: open the same folder structure I use for my Claude workforce and check whether Pi follows the rules from CLAUDE.md and finds the skills. It does. As expected it is slower than Claude Code, and I notice that in every round; I haven't measured it, so it stays a qualitative impression. Until those numbers exist, my verdict stays an assessment: for bounded tasks inside one folder the local setup is enough; for long runs across many files and for anything with an approval gate, Claude Code remains my tool.
What works well locally, and where it tips over
In my runs so far, the tasks that worked best stay inside one folder and finish in a few steps: rewrite a file according to rules, build a protocol out of raw notes, sort receipts, adjust a script and run it. My tests are qualitative, so I can't give you a success rate. Model, server, and harness are enough for that, once the context is large enough.
It tips over on long, unattended runs across many files; on tasks where the agent would need to ask you something halfway through, which with Pi and no approval system means it simply keeps going; and on anything that leaves the house. And on privacy it easily tips into false comfort: local means no model provider receives your input. It does not mean the remaining duties disappear, such as access control on the server, logging, deletion deadlines, and the question of which data the agent is allowed to read at all. This is not legal advice; which duties remain with local operation is laid out with sources in Local AI and Privacy: Cloud or Server?.
Frequently asked questions
Can I use Claude Code completely offline?
According to the Ollama docs, Claude Code can be redirected to a local Ollama server via the ANTHROPIC_BASE_URL environment variable and started with claude --model <model>; the model then runs on your machine. That proves local inference, not that the tool is offline: Claude Code still talks to web search, MCP servers and update sources unless you switch those off. Whether Anthropic's terms of use cover the route isn't stated there. I don't run it.
Which local model can do tool calling?
Look at the model card: if it lists "Tools" or "Function Calling", the model can emit tool calls in the required format. Qwen 3.8 has that according to its model card, along with thinking and image input. Without that note you get a chat, not a local AI agent.
How much context does a local AI agent need?
More than the default. The OpenCode docs suggest 16k to 32k when tool calls stutter, Ollama recommends 64k or more for Claude Code, and the Aider example sets 65,536. My rule: reserve 64k and pick a smaller model if you must, rather than running a big model with 4,096 tokens of context.
Does a local AI agent run fully offline?
The model does, once it's downloaded. The harness not necessarily: Pi has web search and an MCP adapter, and package managers fetch updates. If you mean "offline AI agent" seriously, you switch those paths off deliberately and verify with a network capture instead of asserting it.
What is Pi and how does it differ from Claude Code?
Pi is a terminal coding agent (npm package @earendil-works/pi-coding-agent). Per the Pi docs (retrieved September 18, 2026) it drives a server through the OpenAI chat completions endpoint /v1/chat/completions, with flags for capabilities not every server has. An OpenAI-compatible server is therefore enough to connect; useful agent work needs server and model to return tool calls in the expected format. The tools and rule files are the same as with Claude Code, and you choose the model freely. The differences: no permission system, a smaller system prompt, and no subscription, but your own hardware instead.
Can my AI workforce run entirely locally?
The files, yes: CLAUDE.md, personnel file, skills, learnings, and inbox are text and work with Pi. The control, no: without an approval system in the harness you have to build the gate outside it. My current state: individual roles locally, the workforce with approval gates on Claude.
Is an AI agent without cloud automatically privacy-compliant?
No. Going local removes the model provider that would otherwise receive your input, not your responsibility for access, logging, and deletion. This is not legal advice; the distinction is laid out in my post on local AI and privacy.
Where to go from here
An afternoon is enough to get started: install Ollama, pull Qwen 3.8, raise num_ctx, install Pi, and put a CLAUDE.md with three rules into a test folder. Then give it one bounded task and watch whether the agent reads the rules and calls the tools cleanly. How the tool behaves day to day is in Claude Code Alternative Without Cloud; when to stay with the cloud model is in Qwen vs Claude: Local Versus Cloud; and which duties local operation does not take off your plate is in Local AI and Privacy: Cloud or Server?. The eight AI employee packages are ready to use in my community; they are built as files and therefore run with a local harness too, with the caveat that the approval gates then have to sit outside the tool.
