# Local and private document Q&A

How to choose a way to ask questions over your own documents, on your machine or in your own infrastructure, instead of sending them to a hosted answer engine. The groups separate installable apps, local runtimes, pipeline parts and hosted notebooks, and this index does not rank them.

Canonical URL: https://globalaiproductindex.com/guides/local-document-qa/

## How to choose

**Who this is for.** People who need to ask questions over private documents — contracts, research notes, internal wikis — and want to control where the data and the model run.

**Who this is not for.** Anyone who mainly wants web-wide answers with citations (see the research Q&A guide and the AI answer engines guide), or teams without the ability to run infrastructure locally.

- Decide what must stay local: the model, the document index, or both. Hosted notebooks are simpler but send content to a vendor.
- Estimate your corpus size and update frequency — a few PDFs favours an all-in-one app, a large changing corpus favours a proper retrieval pipeline.
- Check hardware: local models need enough memory or GPU for the model size you want, and embeddings add load.
- Confirm the document formats supported and whether the app extracts text, tables and scanned pages.
- Check whether answers cite the source passages, so you can verify them rather than trusting the summary.
- Check that "offline" holds at every step: the app, the embedding model and the vector store. An app can run locally yet still fetch models or send text to a hosted model unless you point it at a local one.
- Confirm the formats actually indexed — text, PDF, DOCX, spreadsheets and slides are not the same everywhere — and whether scanned pages or audio need OCR or transcription first.

## Self-hosted apps with document chat

Apps you install and run yourself that combine a chat interface with document upload and local models. Unlike the runtimes next, they are a finished app, not a model runner you still have to wire to a chat window.

- [Open WebUI](https://globalaiproductindex.com/products/open-webui/) (Open WebUI) — Platforms & Infrastructure, Open (open weights/source), launched 2023. Extensible, self-hosted AI interface that runs offline and connects to local or remote models such as Ollama and OpenAI-compatible APIs.
- [LM Studio](https://globalaiproductindex.com/products/lm-studio/) (LM Studio) — Platforms & Infrastructure, Free, launched 2023. Desktop application for discovering, downloading, and running open large language models locally.
- [Jan](https://globalaiproductindex.com/products/jan/) (Menlo Research) — Platforms & Infrastructure, Open (open weights/source), launched 2023. Open-source desktop app for running open language models locally as an offline ChatGPT alternative.

## Local runtimes and open models

Runtimes and open model families you can pair with your own interface to keep inference and data on your machine. Unlike the pipeline group below, they serve a model; they do not by themselves index, search and orchestrate a document corpus.

- [Ollama](https://globalaiproductindex.com/products/ollama/) (Ollama) — Platforms & Infrastructure, Open (open weights/source), launched 2023-07. Open-source tool for running open large language models locally on your own machine with a simple command-line interface.
- [Llama](https://globalaiproductindex.com/products/llama/) (Meta) — Open Models, Open (open weights/source), launched 2023-02. Open-weight large language model family from Meta, spanning Llama 3 and Llama 4 generations across multiple sizes.

## Retrieval and pipeline building blocks

Components for teams that want to build their own retrieval pipeline — indexing, vector search and orchestration — over private documents. Unlike the hosted notebooks next, nothing here answers a question until you assemble and run it.

- [LlamaIndex](https://globalaiproductindex.com/products/llamaindex/) (LlamaIndex) — Agents & Automation, Open (open weights/source), launched 2022-11. Open-source data framework for connecting LLMs to private data, powering retrieval-augmented generation and document agents.
- [Dify](https://globalaiproductindex.com/products/dify/) (LangGenius) — Platforms & Infrastructure, Open (open weights/source), launched 2023. Open-source platform for building LLM applications with visual agent workflows, RAG pipelines, and model management.
- [Pinecone](https://globalaiproductindex.com/products/pinecone/) (Pinecone) — Platforms & Infrastructure, Freemium, launched 2021-01. Managed vector database for similarity search and retrieval in AI applications.
- [Weaviate](https://globalaiproductindex.com/products/weaviate/) (Weaviate) — Platforms & Infrastructure, Open (open weights/source), launched 2019. Open-source vector database for semantic search, retrieval-augmented generation, and hybrid search.
- [LangChain](https://globalaiproductindex.com/products/langchain/) (LangChain) — Agents & Automation, Open (open weights/source), launched 2022-10. Open-source framework for building LLM applications and agents, with LangSmith and LangGraph for observability and orchestration.

## Cloud notebooks when offline is not required

Hosted options that do source-grounded question answering when you do not need the data to stay on your own machines. Unlike every group above, these are the choice you make only after deciding that local or self-hosted is not required.

- [NotebookLM](https://globalaiproductindex.com/products/notebooklm/) (Google) — Productivity, Free, launched 2023-07. Google's AI research and note-taking assistant that grounds answers in a user's uploaded sources.
- [Glean](https://globalaiproductindex.com/products/glean/) (Glean) — Productivity, Paid, launched 2021-09. Enterprise AI search and assistant that connects to a company's apps to answer questions over internal knowledge.

## What was actually run

Executed on 2026-10-09 on one machine (macOS, arm64). The raw records are linked in full below.

One document, three small open models, two runtimes, the same five questions and the same settings. Every raw answer was kept exactly as produced. Only Ollama and llama.cpp were executed — no hosted product was tested.

### Method

- Input: one fixed document (input.md, SHA-256 bcd59f7e…) holding four facts the questions ask for, plus one question the document does not answer (the honesty probe).
- Prompt: identical for every model and runtime; answer from the document alone, and reply exactly NOT STATED when the document does not say.
- Settings: temperature 0, seed 42, num_predict 200, streaming off — one request per question, no retries.
- Runner: scripts/run-t17-doc-qa.mjs writes a timestamped JSON record with the raw answer, wall-clock milliseconds, model digests and host platform. Answers were never edited.

### Run records

| Run (UTC) | Runtime | Models | Result | Raw record |
| --- | --- | --- | --- | --- |
| 2026-10-09T11:33Z | Ollama 0.40.2 (darwin/arm64, Node v24.21.0) | qwen2.5-0.5b-q4_k_m · SmolLM2-360M-Instruct-Q8_0 · TinyLlama-1.1B-Chat-Q4_K_M | 3 models × 5 questions. The recorded per-answer times sum to 27.8 s. qwen2.5-0.5b 5/5, SmolLM2-360M 3/5, TinyLlama-1.1B 2.5/5. | [docs/tests/local-document-qa/runs/2026-10-09T11-33-10-310Z.json](https://globalaiproductindex.com/tests/local-document-qa/runs/2026-10-09T11-33-10-310Z.json) |
| 2026-10-09T12:19Z | llama.cpp llama-server (build b11429-d81235049) — The record's runtime_version field is empty. | qwen2.5-0.5b · SmolLM2-360M · TinyLlama-1.1B — The same three models as GGUF files. | The same three models through an independent runtime. 14 of the 15 answers are byte-identical to the Ollama run. The recorded per-answer times sum to 9.1 s. | [docs/tests/local-document-qa/runs-llamacpp/2026-10-09T12-19-04-079Z.json](https://globalaiproductindex.com/tests/local-document-qa/runs-llamacpp/2026-10-09T12-19-04-079Z.json) |

### What the runs show

- The method reproduces: two independent runtimes returned 14 of 15 answers byte for byte identically, and the 15th differs only in where generation stopped.
- Answers can be graded against the fixture, so correct, wrong and evasive are decided by comparison, not by impression.
- The honesty probe separates a model that declines from one that invents: qwen2.5-0.5b replied NOT STATED, SmolLM2-360M produced a number.

### Failures that were kept

- Honesty probe, SmolLM2-360M: asked for the maximum number of participants on an incident call — which the document never states — it answered “The maximum number of participants allowed on an incident call is **2**.” A confident invented value.
- Retention question: SmolLM2-360M answered “30 minutes” (the billing increment) and TinyLlama-1.1B answered “Infinite” (the postmortem rule); the document says 18 months.
- TinyLlama-1.1B also hit the 200-token limit on the escalation question (done_reason “length”) and padded the answer with unrelated facts from other sections.

### What this does not show

- One machine, three small models (0.5B–1.1B), one short document, five questions. It is a check of the method, not a measurement of any product.
- LM Studio, Jan, Open WebUI, NotebookLM and every hosted document chat were not executed, and this page reports no measured result for them.
- Eight failed llama.cpp invocations are kept unedited under runs-llamacpp/attempts/: their empty or banner-only answers came from wrong CLI flags, not from model behaviour, and they are not used as results.
- Wall-clock times are specific to this machine and are not comparable across products.

**Cost.** No account, no API key, no paid plan — open weights run locally. The measured cost was electricity for running the machine; no metered amount was recorded, so none is published.

### Method, inputs and raw outputs

Each label is the file's path in the source repository; the link opens a byte-identical copy of it published on this site (the first entry opens the index of all copies), so the record can be read without repository access.

- [docs/tests/](https://globalaiproductindex.com/tests/index.md)
- [docs/tests/local-document-qa/input.md](https://globalaiproductindex.com/tests/local-document-qa/input.md)
- [docs/tests/local-document-qa.md](https://globalaiproductindex.com/tests/local-document-qa.md)
- [scripts/run-t17-doc-qa.mjs](https://globalaiproductindex.com/tests/scripts/run-t17-doc-qa.mjs)
- [docs/tests/local-document-qa/runs/2026-10-09T11-33-10-310Z.json](https://globalaiproductindex.com/tests/local-document-qa/runs/2026-10-09T11-33-10-310Z.json)
- [docs/tests/local-document-qa/runs-llamacpp/2026-10-09T12-19-04-079Z.json](https://globalaiproductindex.com/tests/local-document-qa/runs-llamacpp/2026-10-09T12-19-04-079Z.json)

### What was not executed

[LM Studio](https://globalaiproductindex.com/products/lm-studio/) · [Jan](https://globalaiproductindex.com/products/jan/) · [Open WebUI](https://globalaiproductindex.com/products/open-webui/) · [NotebookLM](https://globalaiproductindex.com/products/notebooklm/) · [Glean](https://globalaiproductindex.com/products/glean/)

These products were not run: some need an account or a paid plan, and no document-QA run was performed with the self-hosted apps in this round either. This page reports documented behaviour for them, never a measured result.


## FAQ

**Does this guide rank local document tools?**

No. It groups indexed products by the role they play in a private Q&A setup and assigns no scores. It reports no benchmark or quality results.

**Is local document Q&A really private?**

Only if every component stays local. Some so-called local apps still call a cloud model by default, and some runtimes can fetch models from the internet. Check each tool's defaults before assuming privacy.

**Do I need a GPU?**

Not always. Small quantised models can run on CPU, but speed and usable model size improve with more memory or a GPU. Check the tool's hardware documentation for your platform.

**Can a local pipeline match a hosted answer engine?**

It depends on the model, the retrieval quality and the documents. Local setups trade some convenience and model scale for control, privacy and offline operation.

**How do I check an answer in a local document app?**

Open the passage the answer points to and read it, rather than accepting the summary. If the app only names the document and not the passage, treat the answer as a lead. Some local apps also show which chunks were retrieved, which makes the check faster.

## Sources and record dates

Record dates describe index updates, not independent product tests. Check linked documentation for current features and pricing.

### Open WebUI

Record updated: 2026-10-10

- [Official website](https://openwebui.com)
- [Documentation](https://docs.openwebui.com)
- [Source repository](https://github.com/open-webui/open-webui)
- [Documentation](https://docs.openwebui.com)

### LM Studio

Record updated: 2026-10-10

- [Official website](https://lmstudio.ai)
- [Documentation](https://lmstudio.ai/docs)
- [Welcome to LM Studio Docs! | LM Studio](https://lmstudio.ai/docs/app)
- [Official pricing](https://lmstudio.ai/pricing)
- [Pricing](https://lmstudio.ai/pricing)
- [Documentation](https://lmstudio.ai/docs)

### Jan

Record updated: 2026-10-13

- [Official website](https://jan.ai)
- [Source code (GitHub)](https://github.com/janhq/jan)
- [LICENSE — janhq/jan (Apache-2.0)](https://github.com/janhq/jan/blob/main/LICENSE)
- [LICENSE text (raw) — janhq/jan, main branch](https://raw.githubusercontent.com/janhq/jan/main/LICENSE)
- [LICENSE — janhq/jan v0.5.17 (GNU AGPL-3.0)](https://raw.githubusercontent.com/janhq/jan/v0.5.17/LICENSE)
- [LICENSE — janhq/jan v0.5.18-rc1-beta (Apache License 2.0)](https://raw.githubusercontent.com/janhq/jan/v0.5.18-rc1-beta/LICENSE)
- [Documentation](https://www.jan.ai/docs)
- [Documentation](https://www.jan.ai/docs)

### Ollama

Record updated: 2026-10-10

- [Official website](https://ollama.com)
- [Source code (GitHub)](https://github.com/ollama/ollama)
- [Ollama documentation - Ollama](https://docs.ollama.com/)
- [Official pricing](https://ollama.com/pricing)
- [Pricing](https://ollama.com/pricing)
- [Documentation](https://docs.ollama.com/)

### Llama

Record updated: 2026-09-04

- [Official website](https://developer.meta.com/ai/)
- [Llama (Wikipedia)](https://en.wikipedia.org/wiki/Llama_(language_model))

### LlamaIndex

Record updated: 2026-09-10

- [Official website](https://www.llamaindex.ai)
- [Source code (GitHub)](https://github.com/run-llama/llama_index)
- [Official pricing](https://www.llamaindex.ai/pricing)
- [Pricing](https://www.llamaindex.ai/pricing)

### Dify

Record updated: 2026-10-10

- [Official website](https://dify.ai)
- [Source code (GitHub)](https://github.com/langgenius/dify)
- [Documentation](https://docs.dify.ai/en/home)
- [Official pricing](https://dify.ai/pricing)
- [Pricing](https://dify.ai/pricing)
- [Documentation](https://docs.dify.ai/en/home)

### Pinecone

Record updated: 2026-10-09

- [Official website](https://www.pinecone.io)
- [Documentation](https://docs.pinecone.io)
- [The trusted AI knowledge company | Pinecone](https://www.pinecone.io/)
- [Official pricing](https://www.pinecone.io/pricing/)
- [Pricing](https://www.pinecone.io/pricing/)
- [Documentation](https://docs.pinecone.io)

### Weaviate

Record updated: 2026-09-10

- [Official website](https://weaviate.io)
- [Source code (GitHub)](https://github.com/weaviate/weaviate)
- [Official pricing](https://weaviate.io/pricing)
- [Pricing](https://weaviate.io/pricing)

### LangChain

Record updated: 2026-09-10

- [Official website](https://www.langchain.com)
- [Source code (GitHub)](https://github.com/langchain-ai/langchain)
- [Official pricing](https://www.langchain.com/pricing)
- [Pricing](https://www.langchain.com/pricing)

### NotebookLM

Record updated: 2026-10-10

- [Official website](https://notebooklm.google.com)
- [Google blog](https://blog.google)

### Glean

Record updated: 2026-09-10

- [Official website](https://www.glean.com)
- [Glean Technologies (Wikipedia)](https://en.wikipedia.org/wiki/Glean_Technologies)
- [Documentation](https://www.glean.com/platform/api)
- [Documentation](https://www.glean.com/platform/api)

---
Data from the [Global AI Product Index](https://globalaiproductindex.com/), licensed CC BY 4.0.