Beyond Embeddings: How VectorLess RAG Finds Its Way Through Long Documents
A practical guide to how hierarchical document trees and LLM-guided navigation find precise evidence in long documents—without relying on chunk embeddings or vector search.
A tale of two librarians
Imagine you walk into a library and ask: "How much did this company pay in interest on its loans last year?"
The first librarian has done something odd. She has cut every book into thousands of paragraph-sized strips and filed them by how similar their words sound. She hands you the ten strips that sound most like your question. One says "interest rates rose sharply in the region." Another says "we remain interested in expanding." A third has half a table with no headings. The number you need might be in there, or it might not.
The second librarian takes the company's annual report off the shelf, opens the table of contents, and thinks: "Interest paid is a financial figure, so it's in the Financial Statements. The detail will be in the Notes, probably under Borrowings." She flips to page 118, points at the number, and tells you exactly where she found it.
For the past few years, most AI document assistants have worked like the first librarian. Vectorless RAG is the effort to make them work like the second. Here's what that means and when it's worth using. No technical background needed.
First, what is "RAG"?
Large language models like ChatGPT or Claude are trained on huge amounts of public text, but they have never seen your documents: your contracts, your policies, your quarterly reports. Ask about those and the AI will either admit it doesn't know or, worse, make something up.
RAG (Retrieval-Augmented Generation) fixes this by turning the AI's test into an open-book exam. It works in two steps:
Retrieve: find the passages in your documents that are relevant to the question.
Generate: give those passages to the AI and have it write an answer based on them.
The second step is the easy part, because modern AI is very good at reading and summarising. Almost everything depends on the first step. If the AI is handed the wrong pages, it will give a confident answer based on the wrong pages. So the real question is how the right pages get found.
How traditional RAG finds information
The standard approach, often called vector RAG, works like this:
Chop each document into small pieces ("chunks") of a few hundred words.
Convert each chunk into a long list of numbers called an embedding. You can think of it as GPS coordinates for meaning: chunks about similar topics end up with coordinates close to each other.
Store all those coordinates in a special vector database.
When a question comes in, convert it into coordinates too, and fetch the chunks that sit nearest to it.
This approach is fast, scales to millions of documents, and works well for plenty of everyday tasks. But on long, professional documents it runs into four recurring problems.
Problem 1: Chopping destroys context
A 200-page report isn't a pile of loose paragraphs. A table might start on one page and end on the next. A sentence such as "this figure excludes one-time charges" only makes sense next to the figure it refers to. Chunking cuts these connections apart.
Problem 2: "Similar" is not the same as "relevant"
The phrase you'll see most often in discussions of vectorless RAG is "similarity ≠ relevance." A vector search always returns something that sounds close to your question, but sounding close is not the same as containing the answer. A risk-factors section that talks about "rising interest rates" can sound more like your question than the dry accounting note that actually holds the number.
Problem 3: It can't follow references
Real documents point to themselves all the time: "See Note 14," "as defined in Section 2.3," "refer to Appendix B." A human reader follows the pointer. A similarity search has no idea it's there.
Problem 4: It's hard to explain
When a vector system returns a chunk, the only reason it can give is "the numbers were close." In finance, law, or healthcare, people need to know why a source was picked and where exactly it came from.
Enter Vectorless RAG
Vectorless RAG is a family of retrieval methods that find information without embeddings or a vector database. Instead of measuring how similar two pieces of text are, they rely on document structure and the AI's own reasoning to work out where an answer should be.
The best-known example is PageIndex, an open-source framework from VectifyAI. Its creators describe it as simulating "how human experts navigate and extract knowledge from complex documents through tree search." In their words, it replaces "vibe retrieval" with reasoning you can trace.
A useful way to put the shift in one line:
Vector RAG asks: "Which chunk is closest to the question?" Vectorless RAG asks: "Where in this document would the answer live?"
How it works, step by step
Step 1: Build a smart table of contents (the "tree")
Before any questions come in, the AI reads the document once and builds a hierarchical index, much like a detailed table of contents. Each entry, or "node", has:
a title (e.g., "4.3 Notes to Accounts"),
a short summary of what that section covers, and
the page range where it sits.
Sections contain sub-sections, which contain sub-sub-sections, so the whole thing forms a tree. Nothing is cut into arbitrary chunks. The document keeps its natural shape.
Step 2: Reason your way down the tree
When a question arrives, the AI doesn't search every word. It looks at the top of the tree and asks itself which branch is most likely to hold the answer. It picks a branch, looks at the sub-sections, and decides again. This continues until it reaches the right pages, the same way you would use a table of contents.
Step 3: Read, answer and cite
The AI opens only the chosen sections, reads them in full with their surrounding context, and writes the answer, with page and section references attached. Anyone can check the answer by going to page 118.
Because it is reasoning, the AI can also follow cross-references ("see Note 14"), remember the conversation (you asked about 2025 earlier, so you probably still mean 2025), and apply domain knowledge (in financial filings, the detail usually sits in the notes).
Does it actually work?
The headline result comes from FinanceBench, a demanding benchmark of questions about real company filings such as annual reports. These documents are long, dense and full of tables and cross-references, which is exactly where vector RAG struggles.
When FinanceBench was first published in 2023, its authors found that GPT-4-Turbo combined with a standard retrieval system got 81% of questions wrong or declined to answer. PageIndex's makers report that their system, Mafin 2.5, reached 98.7% accuracy on the same benchmark.
The grey bars tell their own story. With no documents, the AI got 9% right. Vector RAG managed 19%, rising to 50% when the search was limited to the correct document. Simply pasting the whole document into the prompt reached 79%. When the AI can see the full structure and context, it reasons well. Chopping the document up is what hurts.
A fair caveat: the 98.7% figure is reported by the vendor, and it came later than the 2023 tests, using different (and likely newer) AI models. It is not a strict like-for-like comparison. Financial filings are also unusually well structured, and results on messier material such as internal wikis or chat logs are likely to be lower. Even so, the size of the gap shows how much retrieval quality matters.
What is PageIndex?
PageIndex is an open-source framework built by Vectify AI (the team includes Mingtian Zhang and Yu Tang) for asking questions of long, professional documents: financial reports, legal filings, regulatory papers and technical manuals. It is currently the best-known implementation of vectorless RAG.
The makers describe it as simulating "how human experts navigate and extract knowledge from complex documents through tree search." Their pitch is that it replaces what they call "vibe retrieval" (grabbing whatever text feels close) with retrieval you can trace step by step.
Three design choices set it apart:
No vector database and no chunking. The document keeps its natural chapters and sections.
The index is readable by the AI. Instead of a database of numbers that only a computer understands, PageIndex produces a structured table of contents that the AI reads directly, and so can you.
Retrieval is a reasoning task. The AI decides where to look, explains why, and can change its mind.
You can use it in several ways: as an open-source Python package you run yourself, through a chat app where you upload a document and ask questions, or through an API and MCP connector (MCP is a standard way to plug tools into AI assistants such as Claude). A hosted cloud version adds better text recognition (OCR) for scanned or complex PDFs.
How PageIndex works
PageIndex works in two phases. The first happens once per document, and the second happens every time someone asks a question.
Phase 1: Build the tree (once per document)
When you give PageIndex a PDF, an AI reads through it and does three things:
Looks for an existing table of contents in the first few pages (the first 20 by default) and uses it as a starting point.
Organises the document into its natural sections and sub-sections, recording the page range of each one. Sections are kept to a manageable size (by default, no more than about 10 pages each), and larger ones are split into sub-sections.
Writes a short summary of what each section contains.
The result is saved as a tree: a nested table of contents with notes. Each entry ("node") looks roughly like this:
{
"title": "Note 14: Borrowings",
"node_id": "0014",
"start_index": 118,
"end_index": 121,
"summary": "Details of bank loans and bonds, interest rates, repayment schedule and total interest paid during the year.",
"nodes": []
}
In plain English, this says: "This section is called Note 14: Borrowings, runs from page 118 to 121, and covers loans, interest rates and interest paid. It has no smaller sub-sections." The nodes field is where child sections would go, which is what turns a flat list into a tree. PageIndex can also build trees from Markdown files, where it uses the headings to work out the structure.
Phase 2: Search the tree by reasoning (every question)
When a question comes in, PageIndex does not show the AI the whole document. It shows only the tree: titles, summaries and page ranges. That is a few pages of "map" instead of hundreds of pages of text.
The AI is given an instruction along these lines: "You are given a query and the tree structure of a document. You need to find all nodes that are likely to contain the answer." It replies with two things:
its thinking, a short written explanation of why certain sections look promising, and
a list of node IDs to open.
PageIndex then pulls the full text of only those pages and gives it to the AI. The process is a loop: if the AI reads the pages and finds something missing, such as a note saying "see Appendix G", it goes back to the map, picks another section, and reads again. Once it has enough, it writes the answer with the sections and page numbers it used.
Why this design is useful
The reasoning can be checked. Because the AI writes down why it chose each section, you can audit every answer: which sections it looked at, why, and which pages it relied on.
You can add expert know-how without retraining anything. A finance team can add a note such as "for questions about EBITDA, check the management discussion and the footnotes first" to the instructions, and the AI will follow it. With vector RAG, changing search behaviour often means re-tuning or retraining the embedding model.
It understands the conversation. The map sits in the AI's working memory alongside the chat, so a follow-up such as "and the year before?" is interpreted in context, not searched in isolation.
It's a movement, not just one product
PageIndex is the flagship, but "reason, don't just match" is spreading. Boris Cherny, the creator of Anthropic's Claude Code, has said: "Early versions of Claude Code used RAG + a local vector db, but we found pretty quickly that agentic search generally works better." The AI now explores code the way a developer does, with keyword search, browsing folders and opening files, and there's no index to go stale. Plain keyword search and "just paste the whole document" (now practical because models can read hundreds of pages) belong to the same family.
They all make the same bet: as AI gets better at reasoning, let it do the finding as well as the answering.
The trade-offs: nothing is free
Vectorless RAG doesn't make vector RAG obsolete. Each approach suits different jobs.
Vector RAG | Vectorless RAG | |
How it finds info | Similar-sounding text | Reasoning over document structure |
Speed per question | Very fast (milliseconds) | Slower, because the AI makes several reasoning calls |
Cost per question | Low | Higher, since each reasoning step uses the AI |
Scale | Millions of documents | Best for a focused set of long documents |
Accuracy on long, complex docs | Often weak | Strong |
Explainability | "The numbers were close" | "Section 4.3, page 118, because…" |
Infrastructure | Embedding model + vector database | No vector database; needs a good tree |
Weak spots | Chunking, cross-references | Poorly structured docs, very large collections |
As one commentator put it, vectorless RAG doesn't remove the hard part of retrieval. It moves it into building a good tree and writing good prompts. A document with no clear structure gives the reasoning nothing solid to work with.
So which one should you use?
Choose vectorless RAG for long, well-structured documents (annual reports, contracts, regulations, policy manuals) where accuracy matters more than speed, questions need multi-step reasoning, and you need an audit trail of where each answer came from.
Stick with vector RAG for huge volumes of short, loosely structured text (support tickets, reviews, chat logs, FAQs) where you need instant answers at scale and "roughly right, very fast" is acceptable.
Combine them when you have many long documents: use fast search to pick the right few, then let the AI navigate inside them by reasoning. Many production systems are heading towards this hybrid pattern.
Comments