Benchmarking Text-to-SQL under Role-Based Access Control
Merged summary
TL;DR - A new framework benchmarks text-to-SQL systems under realistic role-based access controls. It reveals that strong unrestricted performance can mask frequent policy violations and substantial degradation in access-controlled environments.
- Augments existing benchmarks with LLM-synthesized user roles and access policies, audited by domain experts.
- Separately evaluates SQL utility and RBAC compliance.
- Identifies failures such as accessing prohibited data or rejecting answerable, policy-compliant queries.
- Open-weight LLMs were especially prone to sharp performance declines under access constraints.
Sources (1)
Benchmarking Text-to-SQL under Role-Based Access Control
TL;DR - A new framework benchmarks text-to-SQL systems under realistic role-based access controls. It reveals that strong unrestricted performance can mask frequent policy violations and substantial degradation in access-controlled environments.
- Augments existing benchmarks with LLM-synthesized user roles and access policies, audited by domain experts.
- Separately evaluates SQL utility and RBAC compliance.
- Identifies failures such as accessing prohibited data or rejecting answerable, policy-compliant queries.
- Open-weight LLMs were especially prone to sharp performance declines under access constraints.