Run your own AI · Agents on local models

Agents on your own models

thecontinuitylayer.com/open/learn/local-agents

Checked 6 November 2026. US dollars. No affiliate links.

A chat model answers. An agent acts: it reads files, runs commands, searches, sends messages, then looks at the result and decides what to do next, in a loop. That loop works on a local model too, as long as the model can call tools reliably and has room to remember what it has done.

By the end

An agent working on your own machine, on your own model, with limits you set.

What an agent needs from the model

Tool calling. The model must answer in the exact format that triggers a tool. On Ollama's library, look for the “tools” label.
Size. In practice, about 20B parameters and roughly 23GB of graphics memory before agents stop stumbling. Smaller models call the wrong tool or loop.
Context. Every step adds to the conversation. Raise it to 32K–64K tokens; Ollama's default of a few thousand is far too small.

Memory	Model to try	Why
16GB	gpt-oss-20b	Built for tool use; tight, so keep tasks short
24–32GB	Qwen3.8 27B	Ollama lists it for agent harnesses
24–32GB	Devstral Small 2	Made for coding agents
64–128GB	gpt-oss-120b	The closest a home machine gets to a hosted agent

The quickest start

1
Pull a tool-capable model and start Ollama with a bigger context:
```
ollama pull qwen3.8:27b
OLLAMA_CONTEXT_LENGTH=65536 ollama serve
```
(On Windows, set OLLAMA_CONTEXT_LENGTH as an environment variable.)
2
In a second terminal, run ollama launch. It lists the agents Ollama connects to and sets one up for you.
3
Pick a coding agent to work in a project folder, or a personal assistant to chat through your messaging apps.
4
Give it a small, checkable first job: "read this folder and list what each file does."

Ollama's integrations page names Claude Code and OpenCode for coding, OpenClaw and Hermes Agent as personal assistants, and a VS Code extension, read on 6 November 2026.

Without a terminal

AnythingLLM: built-in agents that browse the web and run scheduled tasks, with a no-code builder.
Open WebUI: tools and web search inside chats on your house server.
Jan: MCP tools on top of local models.

Build your own

Ollama's API takes a list of tools with each request; when the model wants one, it returns a tool call, your code runs it, and you send the result back. Keep calling until no tool call comes back. The pattern, with examples, is in Ollama's tool-calling guide. Because the API is OpenAI-compatible, most agent frameworks work by pointing them at http://localhost:11434/v1.

Give it a memory that lasts

An agent forgets everything when the session ends unless something outside the model holds the thread. That is the job of a continuity layer: a record the agent reads first and writes to last. Start with one file, then build your own continuity layer. The memory tools agents can use are graded on the memory-systems shelf.

← A local AI server for the house Back to the learning path →