Evaluating RAG Before You Scale: Golden Sets and Metrics That Matter

The product lead pastes a good answer into Slack. Someone replies with a thumbs-up. The team decides to “roll it out more widely.” Two weeks later support is forwarding screenshots of confident wrong answers, and nobody can say whether retrieval or the model is at fault.

If you cannot measure RAG quality, you cannot improve it. Teams that scale first and evaluate later spend months arguing about anecdotes.

Separate retrieval from generation

A bad answer can come from:

• Wrong chunks retrieved (retrieval failure)
• Right chunks, wrong synthesis (generation failure)

If you only score “did the user like the answer,” you will fix the wrong layer. I have watched teams burn cycles rewriting prompts when the index was the problem.

Useful directions:

• Context recall / precision: did we retrieve what we needed?
• Faithfulness: is the answer supported by retrieved context?
• Answer relevance: does it address the question?
• Citation accuracy: do cited sources actually support the claim?

Build a golden set first

Start small and real: 50–100 questions from actual users or support tickets. Not invented “demo questions.” Real ones, including the awkward ones.

For each item, store:

• Question
• Expected answer or key facts
• Gold documents / passages when known
• Tags (policy, product, edge case, cross-source)

Refresh the set as products and policies change. A stale golden set is theatre. It looks rigorous until the product ships a new pricing page and your suite still grades against last quarter’s truth.

Offline vs online

Offline (pre-deploy): run the suite in CI. Block releases on regression.

Online (production): sample live traffic, capture thumbs-down, and feed failures back into the golden set.

You need both. Offline alone misses drift. Online alone is too slow for safe iteration. The offline suite is your seatbelt. The online loop is how you keep the seatbelt honest.

Practical workflow

When I work with teams on this, the rhythm looks like:

1. Freeze a baseline prompt + retrieval config
2. Score the golden set; record numbers
3. Change one variable (chunking, rerank, prompt)
4. Re-score; keep only improvements
5. Promote behind a gate

This is how you avoid “prompt roulette.” One change at a time. Evidence before promotion.

What “good enough to scale” looks like

Agree thresholds with stakeholders before the pilot:

• Minimum faithfulness / groundedness
• Maximum unsupported-answer rate
• Citation coverage for high-risk domains
• Latency and cost budgets

Document the thresholds. Otherwise every release becomes a negotiation, and the person with the strongest opinion wins.

Tie-in to Knowledge AI

Private Knowledge AI is not “chat over PDFs.” It is retrieval, permissions, citations, and evaluation as a system. See Private Knowledge AI and permissions-aware retrieval. For the operational side of keeping a suite alive, see evaluation suites.

Checklist

Use this as a conversation starter with your eng lead, not as wallpaper:

• Golden set exists and is versioned
• Retrieval and generation metrics are separated
• CI gate on regression
• Failure cases flow back into the set
• Risk tiers have different quality bars

If you are deciding whether your knowledge system is ready to scale, discuss your use case.