RAG-Safety-Bench: Reliable Evaluation of Retrieval-Augmented LLM Safety

TL;DR - RAG-Safety-Bench evaluates how retrieval-augmented generation affects LLM safety by isolating retrieval context from retriever quality. Results across five open-source LLMs indicate that existing safety guardrails may not reliably carry over to RAG deployments.

• Tests four settings: no RAG, oracle harmful-answer documents, related documents without answers, and random safe documents.
• Finds an inverse relationship between benign capability and unsafe capability.
• Shows that baseline model safeguards do not guarantee safe outputs after retrieval.
• Finds model-specific evidence that even benign retrieved documents can increase unsafe generation.