RAG over 10+ databases: what production taught us

TL;DR: In 2026 the consensus is clear: vector-only RAG doesn’t scale in production. At Darwin we built an agentic compliance system with hybrid retrieval across 10+ databases. The best decisions we made had nothing to do with the model, they were about the data layer.

The problem

Our agentic compliance system has to answer questions like:

“Which mango lots from producer X, processed between May and July, met the CTEs required by FSMA 204, and which ones have evidence gaps?”

A single question that combines:

• Regulatory knowledge (FSMA 204 CTE/KDE definitions, structured text)
• Traceability events (CTEs recorded with timestamp, geolocation, supplier, lot)
• Internal business rules (gap analysis, risk scoring, dynamic logic)
• Relationships between entities (producer → plant → shipment → retailer)

A single vector store with embeddings of everything mixed together can’t answer that well. The right answer requires joining structured data + semantic retrieval + aggregate computation.

The architecture

Our retrieval stack:

The agent doesn’t know where each thing lives, the orchestrator resolves it.

LangGraph as orchestrator

We use LangGraph to route queries in multiple steps:

1. Classify: what kind of question is it (regulatory / operational / mixed)
2. Plan: which retrievals are needed (vector + SQL + graph traversal)
3. Fan-out: execute retrievals in parallel
4. Synthesize: pass results to the LLM with structured context
5. Validate: guardrails to prevent hallucinations on numeric data

The key step was #2: giving the LLM a query planner that decides the retrieval strategy before going to fetch. Without that, the model hallucinates data or pulls in irrelevant context.

What didn’t work

Vector-only with aggressive chunking: our first attempt. It failed on two fronts:

• Counting / aggregation queries (how many lots? weekly average?), the LLM made up numbers when they weren’t explicitly in context
• Relational joins (producer X + time window Y + certification Z), impossible without a structured query

The fix wasn’t “better chunking”, it was separating semantic retrieval from structured queries.

What did work

Explicit query routing → the planner decides whether a question requires vector search, SQL, graph traversal, or a mix.

Numeric guardrails → if the LLM’s answer contains numbers, we verify they match what the structured query returned. If not, fail fast instead of returning wrong data.

Semantic caching at the similar-questions level → cuts LLM costs by ~40% without impact on quality.

Full-trace observability with OpenTelemetry → every query is tracked end-to-end (planner → retrieval → LLM → guardrails). Critical for debugging.

Lessons learned

1. The bottleneck of RAG in production isn’t retrieval, it’s deciding which retrieval to use
2. Numeric guardrails save lives when the correctness of an answer drives regulatory decisions
3. LangGraph beats linear chains for orchestrating conditional retrievals
4. Multi-store + planner > single vector store with better chunking
5. LLMs will hallucinate on structured aggregations: no matter how good the model is

What’s next?

The next iteration is to replace some planner rules with router fine-tuning using real production examples. The planner as an LLM is flexible but expensive, caching its decisions into a smaller model is a logical step.

If you’re building RAG for regulated domains, my advice is: start with the query planner, not the vector store.

Are you building something similar? Let’s talk, we’re open to sharing architecture and learning from other cases.