How RAG Works Under the Hood

Study Guide · Chapter 4: RAG Grounding · 7 min read

A RAG pipeline has two halves: an offline ingestion pipeline that builds a searchable library, and an online retrieval loop that serves answers. Both must be healthy for answers to be trustworthy.

Ingestion: building the library

Documents are cleaned, split into chunks small enough to be specific but large enough to carry context, and converted into embeddings — long number vectors that capture each chunk's meaning.

Chunks land in a vector index alongside metadata: source, date, owner, access level. That metadata is not decoration; it powers freshness ranking, filtering, and permission checks at query time.

Retrieval: finding the right passages

At question time, the query itself becomes an embedding, and the index returns the closest chunks. Hybrid search — combining vector similarity with classic keyword matching — catches both 'means the same thing' and 'contains that exact product code'.

A reranking step then scores candidates for true relevance to the question, so only the strongest few passages enter the model's context.

Generation with citations

The prompt instructs the model to answer only from provided passages — and to say when they do not contain the answer. This instruction discipline is what separates grounded systems from confident hallucinators.

Finally, each claim maps back to its source document, producing citations users can click. If retrieval found nothing solid, the correct answer is 'I don't know' — delivered honestly.

Key Points

Chunking quality silently caps everything downstream — bad chunks, bad answers.
Hybrid search (vectors + keywords) outperforms either technique alone.
Rerankers sharpen precision before context is spent on weak passages.
Answers must cite sources and admit ignorance when evidence is missing.

All study guides for this chapter: RAG, Explained Simply · How RAG Works Under the Hood · RAG in the Real World