A recent paper published on arXiv cs.AI introduces ESQ-Bench, a new benchmark designed to evaluate Natural Language to SQL (NL2SQL) models in enterprise environments. This benchmark addresses limitations of existing academic benchmarks like Spider and BIRD, which often use simplified schemas and open-source SQL dialects. ESQ-Bench focuses on Oracle-first scenarios, incorporating systematic complexity tiers and a method for evaluating silent semantic divergence.
Key Points
- ESQ-Bench includes six populated schemas with identical seed data across Oracle, PostgreSQL, MySQL, and SQL Server.
- The benchmark comprises 465 tables and 164,682 rows, with no empty tables.
- It features 550 gold-validated question-query pairs divided into three complexity tiers: Tier-1 (95), Tier-2 (228), and Tier-3 (227).
- A four-metric evaluation harness (EM, EX, SR, SD) is used for assessment.
- Schema-linked prompting with GPT-4o in June 2026 showed monotonic execution-match degradation across tiers: 79.8% (Tier-1), 60.3% (Tier-2), and 57.2% (Tier-3) for executed queries.
- Exact Match (EM) accuracy remained below 7% across all tiers.
- Operational silent semantic divergence reached 73% to 99% among queries that passed the execution match (EX) metric.
Context
According to the arXiv paper, state-of-the-art NL2SQL models report execution accuracy exceeding 89% on established benchmarks. However, these benchmarks do not reflect the complexity of enterprise database environments, which often involve more intricate schemas and specific SQL dialects. ESQ-Bench was constructed to bridge this gap by providing a more realistic evaluation setting, including a focus on Oracle databases and a tiered approach to schema complexity.
Why It Matters
This benchmark highlights a significant challenge for builders and researchers deploying NL2SQL models in real-world enterprise settings. The observed degradation in execution accuracy and high rates of silent semantic divergence, even in queries that appear to execute correctly, indicate that current models may not reliably translate natural language into semantically equivalent SQL in complex scenarios. This suggests a need for more robust evaluation and development practices for enterprise-grade NL2SQL solutions.
What To Do
- Review the ESQ-Bench paper to understand the methodology for constructing enterprise-grade NL2SQL benchmarks.
- Note the performance degradation of GPT-4o across complexity tiers, particularly the low Exact Match scores and high silent divergence.
- Consider incorporating silent semantic divergence evaluation into internal NL2SQL model testing.
- Compare the ESQ-Bench findings with performance metrics from other NL2SQL benchmarks to assess model generalization capabilities.