RAG Evaluation Services

The problem

RAG is sold as the cure for hallucination. It is only a cure if retrieval works and the model stays inside the retrieved text. Teams often ship a vector store, watch a few good answers, and never measure hit rate, ranking, or whether citations match the passage.

RAG evaluation is QA for that pipeline: query → retrieve → generate → cite. It is not generic LLM evaluation (no corpus) and not hallucination testing alone (that page can apply without retrieval).

What’s at risk

• Answers grounded in the wrong document — still fluent
• Citations that point to a file the model did not use
• “I don’t know” never triggering when the corpus is empty
• Chunking that splits the only sentence that mattered
• A re-index that silently drops a product line

What we evaluate

• Retrieval: recall of the passages a human would need
• Ranking: is the useful chunk near the top?
• Grounding: does the answer stay inside retrieved text?
• Citations: do links and titles match what was used?
• Abstention: refusal or clarification when nothing relevant is found
• Freshness and filters: access control, stale docs, wrong tenant

How a RAG evaluation runs

1. Build a question set from real user queries, not marketing FAQs only
2. Label what a good retrieve would include (even if the current system misses it)
3. Score retrieve and generate separately — a good answer from a lucky retrieve still hides a weak index
4. Probe missing-context and conflicting-document cases
5. Re-run after chunking, embedding, or corpus changes

Parent: AI testing services. If the product is a support bot on top of docs, also see chatbot testing.

Deliverables

• Labeled query set and retrieval notes
• Grounding and citation findings
• A short list of index or prompt changes worth trying — not a magic “RAG score”

Suitable for

Internal search, support assistants, and any product that answers from your documents.

FAQs

Do you tune the embeddings for us?

We report where retrieval fails. Changing chunking or embeddings is engineering work we can inform, not a silent rewrite of your index.

What if we have no labeled questions?

We start from logs or a workshop with the people who know the corpus, then grow the set. Evaluation without any labeled queries is just watching demos.