Run your own AI · Chat with your own documents

Chat with your own documents

thecontinuitylayer.com/open/learn/your-documents

Checked 6 November 2026. US dollars. No affiliate links.

Document chat, often called RAG, works in two steps. The app splits your files into passages and indexes them. When you ask a question, it finds the most relevant passages and hands them to the model with your question. With a local model and a local index, none of it leaves your computer.

By the end

A folder of your own files you can ask questions of, with answers that cite the passage they came from.

Easiest: AnythingLLM

1
Install the desktop app from anythingllm.com. It has a built-in local model engine, or point it at Ollama or LM Studio.
2
Create a workspace, and drag your PDFs, Word files or text files into it.
3
Ask a question. Answers come with citations to the source passage; open them to check.

For a shared setup: Open WebUI

1
With Open WebUI running on top of Ollama (see a local AI server for the house), add your files in the Workspace.
2
In a chat, type # and pick the document or collection to search. Typing # with a web address pulls in that page instead.
3
Raise the model's context to at least 8,192 tokens, as Open WebUI's own docs advise; with the small default, the passages it finds can be cut off before the model reads them.

Steps from Open WebUI's document guide, read on 6 November 2026.

Get better answers

Ask specific questions. "What is the cancellation fee in the 2025 lease?" beats "Summarize my documents."
Clean files work best. Scanned PDFs without text need OCR first.
Split very large collections into separate workspaces by subject.
Use a model of 8B or more; tiny models misread passages.

← System prompts and custom models Next: Updates and new versions →