Retrieval-augmented generation (RAG)

Retrieval-augmented generation (RAG) lets a large language model (LLM) answer questions using information it was never trained on, such as your company’s policies, product documentation or codebase. When a question arrives, the system first retrieves the most relevant passages from your content, then asks the model to generate an answer using those passages.

It’s the most common architecture for enterprise AI assistants, because it keeps answers current and traceable to sources without retraining a model.

Diagram

flowchart LR
    subgraph ingest [Indexing, ahead of time]
        docs[Documents] --> chunk[Split into chunks]
        chunk --> embed1[Embedding model]
        embed1 --> vdb[(Vector index)]
    end
    subgraph query [At question time]
        q[User question] --> embed2[Embedding model]
        embed2 --> search[Similarity search]
        vdb --> search
        search --> prompt[Prompt with retrieved passages]
        q --> prompt
        prompt --> llm[LLM]
        llm --> answer[Answer with citations]
    end

How it works

Chunking. Documents are split into passages small enough to be relevant and large enough to carry meaning.
Embeddings. An embedding model turns each chunk into a vector, a list of numbers representing its meaning. Similar meanings produce similar vectors.
Vector search. The question is embedded the same way, and the closest chunks are retrieved. Many systems combine this with keyword search (hybrid search) and a re-ranker that reorders results.
Prompt assembly. Retrieved passages are placed in the prompt with instructions to answer only from them and cite sources.
Generation. The LLM writes the answer.

Pros

Uses private, current knowledge without retraining the model.
Traceable answers: responses can cite the passages they came from.
Fewer hallucinations than asking the model from memory, when retrieval is good.
Access control: retrieval can respect who is allowed to see which documents.

Cons

Retrieval quality decides answer quality: if the right passage isn’t retrieved, the answer will be wrong.
Many tuning knobs: chunk size, embedding model, ranking and prompt all interact.
Not good at whole-corpus questions such as “summarize every contract”, which need different approaches.
Data freshness and security: indexes must be kept in sync with sources and their permissions.

When to use it

Question answering over internal documentation, policies, tickets or code.
Customer support assistants grounded in product knowledge.
Any use where answers must be current and cite sources.

When to avoid it

Teaching the model a style, format or skill, where fine-tuning or better prompting fits better.
Precise numerical or transactional questions, which should query a database or API directly through tool use.

Common pitfalls

No evaluation set. Without a list of real questions and expected answers, changes can’t be measured, only guessed at.
Bad chunking. Splitting mid-table or mid-section separates facts from their context.
Ignoring permissions. Retrieval that ignores document access rights leaks information.
Stuffing too much context, which raises cost and can bury the relevant passage.

Reference implementation

A production-shaped blueprint. Components are described generically first, then with one concrete example stack; the cloud mapping shows equivalents on each major provider.

Scenario

An internal policy assistant for a company of 5,000 employees. Staff ask questions about HR, security and expense policies. Answers must cite the source document, respect document permissions (managers see manager-only policies; others don’t), and say “I don’t know” rather than guess.

Component blueprint

flowchart LR
    subgraph ingest [Ingestion pipeline]
        src[Document sources:<br/>wiki, SharePoint, PDFs] -- change events --> q[[Queue]]
        q --> worker[Ingestion worker:<br/>parse, chunk, embed]
        worker --> emb1[Embedding model]
    end
    worker --> store[(Postgres + pgvector<br/>chunks, vectors, keywords,<br/>permissions)]
    subgraph serve [Query service]
        api[Assistant API] --> emb2[Embedding model]
        api --> store
        api --> rr[Re-ranker]
        api --> gw[Model gateway] --> llm[LLM]
    end
    user[Employee via chat UI] -- SSO identity --> api
    api --> trace[LLM tracing and evaluation]

Component	Responsibility	Vendor-neutral choice	Example stack
Ingestion worker	Parse, clean, chunk and embed documents	Queue-driven worker	Python, a document parser, a queue
Embedding model	Turn text into vectors	Any embedding model	An open-weight or hosted embedding model
Vector and keyword store	Similarity search, keyword search, permission filters	Database with vector support	PostgreSQL (opens in a new tab) 16 + pgvector (opens in a new tab)
Re-ranker	Reorder candidates by true relevance	Cross-encoder model	A hosted or open-weight re-ranker
Model gateway	One API over model providers, with keys, limits and logging	LLM gateway	LiteLLM (opens in a new tab)
Assistant API	Retrieval, prompt assembly, answer and citations	Stateless web service	Python 3.12, FastAPI (opens in a new tab)
Tracing and evaluation	Record every prompt, retrieval and answer; score quality	LLM observability tool	Langfuse (opens in a new tab), OpenTelemetry (opens in a new tab)

Request flow: answering a question

sequenceDiagram
    participant U as Employee
    participant A as Assistant API
    participant E as Embedding model
    participant S as Postgres + pgvector
    participant R as Re-ranker
    participant L as LLM (via gateway)
    U->>A: "How many days of parental leave do I get?"
    A->>E: embed question
    A->>S: hybrid search, filtered to user's groups
    S-->>A: 20 candidate chunks
    A->>R: re-rank candidates
    R-->>A: top 6 chunks
    alt nothing relevant enough
        A-->>U: "I couldn't find this in the policy documents."
    else
        A->>L: prompt with question + 6 chunks
        L-->>A: answer with [1], [2] markers
        A-->>U: answer + links to cited documents
    end

Key implementation details

1. Store chunks with vectors, keywords and permissions together. Keeping permissions on every chunk means retrieval can never return text the user isn’t allowed to see.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
    id              BIGSERIAL PRIMARY KEY,
    document_id     TEXT        NOT NULL,
    source_url      TEXT        NOT NULL,
    title           TEXT        NOT NULL,
    section         TEXT,
    content         TEXT        NOT NULL,
    allowed_groups  TEXT[]      NOT NULL,          -- copied from the source document's ACL
    content_hash    TEXT        NOT NULL,          -- skip re-embedding unchanged text
    embedding_model TEXT        NOT NULL,          -- re-embed when the model changes
    embedding       vector(1024) NOT NULL,          -- match your model's dimensions
    content_tsv     tsvector GENERATED ALWAYS AS (to_tsvector('english', content)) STORED,
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunks USING gin (content_tsv);
CREATE INDEX ON chunks USING gin (allowed_groups);

2. Chunk along the document’s structure. Split on headings first, then by size, and prefix each chunk with its document title and section so it still makes sense on its own.

def chunk_document(doc, max_tokens=500, overlap_tokens=60):
    for section in doc.sections():                  # split on headings first
        for text in split_by_tokens(section.text, max_tokens, overlap_tokens):
            yield Chunk(
                document_id=doc.id,
                title=doc.title,
                section=section.heading,
                content=f"{doc.title} > {section.heading}\n\n{text}",
                allowed_groups=doc.allowed_groups,
            )

3. Retrieve with hybrid search. Vector search finds paraphrases; keyword search finds exact terms like policy codes. Reciprocal rank fusion merges the two lists without needing comparable scores.

-- $1 = question embedding, $2 = question text, $3 = the user's groups
WITH semantic AS (
    SELECT id, row_number() OVER (ORDER BY embedding <=> $1) AS rnk
    FROM chunks
    WHERE allowed_groups && $3
    ORDER BY embedding <=> $1
    LIMIT 40
),
keyword AS (
    SELECT id, row_number() OVER (ORDER BY ts_rank_cd(content_tsv, q) DESC) AS rnk
    FROM chunks, websearch_to_tsquery('english', $2) AS q
    WHERE content_tsv @@ q AND allowed_groups && $3
    ORDER BY ts_rank_cd(content_tsv, q) DESC
    LIMIT 40
)
SELECT c.id, c.title, c.source_url, c.content,
       COALESCE(1.0 / (60 + s.rnk), 0) + COALESCE(1.0 / (60 + k.rnk), 0) AS score
FROM chunks c
LEFT JOIN semantic s ON s.id = c.id
LEFT JOIN keyword  k ON k.id = c.id
WHERE s.id IS NOT NULL OR k.id IS NOT NULL
ORDER BY score DESC
LIMIT 20;

With very selective permission filters, an approximate vector index can return too few rows. Recent pgvector versions support iterative index scans for this; otherwise raise the inner limit.

4. Orchestrate retrieval and generation, and refuse when unsure.

def answer(question: str, user: User) -> Answer:
    query_vector = embedder.embed(question)
    candidates = store.hybrid_search(query_vector, question, user.groups, limit=20)
    top = reranker.rerank(question, candidates)[:6]

    if not top or top[0].relevance < MIN_RELEVANCE:
        return Answer.not_found("I couldn't find this in the policy documents.")

    reply = llm.generate(
        system=SYSTEM_PROMPT,
        user=render_context(question, top),   # numbered passages [1]..[6]
        temperature=0,
        max_tokens=600,
    )
    return Answer(text=reply.text, citations=cited_sources(reply.text, top))

SYSTEM_PROMPT
You answer employee questions using only the numbered policy passages provided.
- Cite every claim with the passage number, like [2].
- If the passages don't contain the answer, say you couldn't find it.
- Don't give legal or medical advice; point to the HR contact in the passage instead.

5. Evaluate before every change. Keep a set of real questions with known answers, and run it in CI whenever the chunking, embedding model, prompt or model changes.

- question: How many days of parental leave do full-time employees get?
  must_cite: hr/parental-leave-policy
  expected_facts: ["16 weeks", "full-time employees"]
- question: Can I expense a home office chair?
  must_cite: finance/expense-policy
  expected_facts: ["up to $300", "manager approval"]
- question: What is the CEO's home address?
  expect: not_found

Track retrieval recall (did the right document appear in the top 6?), faithfulness (is every statement supported by a cited passage?) and the not-found rate. A drop in any of them blocks the release.

Cloud service mapping

Component	AWS (opens in a new tab)	Azure (opens in a new tab)	Google Cloud (opens in a new tab)
Embedding model and LLM	Amazon Bedrock (opens in a new tab)	Azure OpenAI (opens in a new tab)	Gemini Enterprise Agent Platform (opens in a new tab) (formerly Vertex AI)
Vector store	Aurora PostgreSQL (opens in a new tab) (pgvector) or OpenSearch Service (opens in a new tab)	Azure Database for PostgreSQL (opens in a new tab) (pgvector) or Azure AI Search (opens in a new tab)	AlloyDB (opens in a new tab) / Cloud SQL (opens in a new tab) (pgvector) or Vector Search (opens in a new tab) on Agent Platform
Document storage	Amazon S3 (opens in a new tab)	Blob Storage (opens in a new tab)	Cloud Storage (opens in a new tab)
Ingestion queue	Amazon SQS (opens in a new tab)	Service Bus (opens in a new tab)	Pub/Sub (opens in a new tab)
Re-ranking	Bedrock re-rank models	Azure AI Search semantic ranker	Google Cloud ranking API

Service names change over time; check each provider’s current catalog.

Run it yourself

The RAG reference implementation (opens in a new tab) builds the policy assistant six times against one shared contract (opens in a new tab) and the same retrieval query (opens in a new tab). Models run locally in Ollama (opens in a new tab) (nomic-embed-text for embeddings, qwen2.5:3b for answers), so no document leaves your machine. To stay small it calls Ollama and Postgres directly, with no framework, model gateway, re-ranker or tracing.

Language	Stack	Source
Java 21	Spring Boot (opens in a new tab) 4, JdbcTemplate, RestClient	java (opens in a new tab)
Python 3.12	FastAPI, psycopg 3 (opens in a new tab), httpx (opens in a new tab)	python (opens in a new tab)
Go 1.25	net/http, pgx (opens in a new tab)	go (opens in a new tab)
Rust	axum (opens in a new tab), sqlx (opens in a new tab), reqwest (opens in a new tab)	rust (opens in a new tab)
C# (.NET (opens in a new tab) 10)	ASP.NET Core (opens in a new tab) minimal APIs, Npgsql (opens in a new tab)	dotnet (opens in a new tab)
TypeScript (Node (opens in a new tab) 22)	Fastify (opens in a new tab), node-postgres (opens in a new tab)	node (opens in a new tab)

git clone https://github.com/architecture-compass/reference-implementations
cd reference-implementations/rag
docker compose --profile python up -d --build    # or: java, go, rust, dotnet, node
./demo.sh

The demo ingests five policy documents, re-ingests them to show that unchanged sections aren’t re-embedded, asks a question and checks the cited source, asks a managers-only question as a regular employee and as a manager, and asks something the documents don’t cover to confirm the assistant refuses. Every language passes all seven checks. The first start downloads about 2.2 GB of models.

Want to go further? The README’s Break it yourself (opens in a new tab) section has hands-on experiments that show each pattern at work, including what happens when things fail.

Production readiness checklist

Permissions copied from source systems onto every chunk, and filtered at query time
Deleted or moved documents removed from the index within an agreed time
Golden question set runs in CI; quality drops block releases
Every answer cites sources; “not found” is a valid answer
Prompts, retrieved chunks and answers traced, with personal data redacted
Rate limits and per-user cost tracking at the model gateway
Documents treated as untrusted input: retrieved text can’t change system instructions
Re-embedding plan for when the embedding model changes

AI agents
Hybrid search and re-ranking
Model gateway
Human-in-the-loop review