Cutting Irrelevant Retrieval by 50% in a Legal RAG Pipeline

A legal client had built a first-pass document assistant on top of LlamaIndex. The idea was sound. Let associates ask questions across a corpus of contracts, case files, and internal memos instead of grepping through PDFs. The execution was struggling, though. Roughly half the chunks it retrieved were irrelevant. The right answer was usually somewhere in the context, buried under three unrelated clauses that confused the model and burned tokens.

I was brought in to fix retrieval quality. By the end, irrelevant retrieved context dropped by about 50%, and the associates started trusting the answers enough to use it in real work.

The Diagnosis

The original pipeline used flat chunking. Every document got split into 512-token windows, embedded, and dumped into one vector index. For legal documents this is close to the worst possible approach.

Legal text is deeply hierarchical. A contract has parties, recitals, definitions, numbered sections, sub-clauses, schedules. A flat chunk rips a sub-clause out of its section and strips the context that makes it meaningful. "The Receiving Party shall not disclose" means nothing without knowing which agreement, which definitions of "Receiving Party," and which carve-outs apply.

The retrieval was also doing pure semantic similarity with no filtering. A question about an NDA signed in 2022 would happily retrieve similar-sounding clauses from a completely different 2019 agreement.

Hierarchical Chunking

The first change was switching to LlamaIndex's hierarchical node parser. Instead of flat windows, documents were parsed into a tree: document, then section, then clause. Each leaf node kept references to its parents.

At retrieval time I used auto-merging. If enough child chunks from the same parent section were retrieved, the system merged them up to the parent node and returned the whole section instead of fragments. This gave the model coherent, self-contained context instead of disconnected slivers.

from llama_index.core.node_parser import HierarchicalNodeParser
from llama_index.core.retrievers import AutoMergingRetriever
 
node_parser = HierarchicalNodeParser.from_defaults(
    chunk_sizes=[2048, 512, 128]
)
 
retriever = AutoMergingRetriever(
    base_retriever,
    storage_context,
    simple_ratio_thresh=0.5,
)

The three chunk sizes matter: 128-token leaves for precise matching, merging up to 512 or 2048 when a query hits multiple leaves in the same parent. You get precision at the leaves and coherence at the parents.

Metadata Filtering

The second change was attaching structured metadata to every node and filtering on it before semantic search ran.

During ingestion I extracted document type (NDA, MSA, employment, etc.), party names, effective date, governing law, and a document ID. Some of this came from the document structure, some from a lightweight extraction pass with an LLM over the first page.

Then queries could be scoped. When an associate asked about a specific agreement, the UI captured which matter they were working on, and the retriever applied a hard metadata filter:

filters = MetadataFilters(filters=[
    ExactMatchFilter(key="matter_id", value=matter_id),
    ExactMatchFilter(key="doc_type", value="nda"),
])

This alone eliminated a huge chunk of the cross-document noise. You can't retrieve an irrelevant 2019 contract if it's filtered out before similarity ranking even starts.

Reranking

Semantic similarity gets you a candidate set, but the top-5 by cosine distance aren't always the top-5 by actual relevance. I added a reranking stage: retrieve 20 candidates, then run them through a cross-encoder reranker that scores each chunk against the query directly, and keep the top 5.

Cross-encoders are slower than bi-encoder similarity because they process the query and each candidate together rather than comparing pre-computed vectors. But over 20 candidates the latency was acceptable (~200ms), and the relevance improvement was the single biggest lever after metadata filtering. The reranker consistently pushed the genuinely-relevant clause into the top 3, even when it ranked 8th or 9th by raw similarity.

Measuring "Irrelevant Context"

The 50% number needs a definition. I built an eval set of ~80 real questions from the associates, each with the human-labelled set of clauses that should be retrieved to answer it. For each question I measured precision: of the chunks the system retrieved, what fraction were in the relevant set.

Baseline precision was around 0.48, so slightly less than half the retrieved context was relevant. After hierarchical chunking, metadata filtering, and reranking, precision rose to about 0.74. The "50% reduction in irrelevant context" is that gap: the irrelevant fraction went from ~0.52 to ~0.26.

I ran the eval after each change so I could attribute the gains:

• Hierarchical chunking + auto-merge: precision 0.48 → 0.59
• Metadata filtering: 0.59 → 0.68
• Reranking: 0.68 → 0.74

What I'd Change

Extraction quality gates. The metadata extraction pass was good but not perfect. Maybe 5% of documents got a wrong effective date or missed a party. For a legal tool that's a real problem, because a wrong filter silently hides relevant context. I'd add a confidence score to extracted metadata and flag low-confidence documents for human review.

Hybrid search. I stuck with pure dense retrieval. Legal queries often hinge on exact terms: a specific defined term, a section number, a party name. A hybrid dense plus sparse (BM25) approach would have caught the cases where the exact-term match mattered more than semantic similarity. I left it on the roadmap.

Citations down to the clause. The system cited source documents, but associates wanted clause-level citations they could click straight to. The hierarchical node structure already had the data to support this. I just didn't get to wiring it into the UI before the engagement ended.

One thing here generalises beyond legal work. Flat chunking is a default that quietly fails on any structured corpus. The moment your documents have meaningful hierarchy, whether legal, medical, technical manuals, or financial filings, the structure is the retrieval signal. Throwing it away to fit a uniform chunk size throws away the thing that makes retrieval work.