Benchmarking Text-to-SQL under Role-Based Access Control

TL;DR - A new framework benchmarks text-to-SQL systems under realistic role-based access controls. It reveals that strong unrestricted performance can mask frequent policy violations and substantial degradation in access-controlled environments.

• Augments existing benchmarks with LLM-synthesized user roles and access policies, audited by domain experts.
• Separately evaluates SQL utility and RBAC compliance.
• Identifies failures such as accessing prohibited data or rejecting answerable, policy-compliant queries.
• Open-weight LLMs were especially prone to sharp performance declines under access constraints.