Evaluation Framework for SQL Related questions

Evolution is key to agents.

Agents are good at making choices, but not at defining guardrails and rubrics. So we need to tell agents what success looks like. Besides, eval can also tell us whether an agent is able to answer questions robustly (pass^k) and perform A/B testing (whether switching LLM providers or adding new workflows, code, or skills will degrade the accuracy of existing answers). Even more, eval is a prerequisite for RSI. We need a good eval framework to let agents improve themselves.

So this blog will mainly walk you through how to build our own evaluation framework, mainly for evaluating SQL-related questions.