BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click
TL;DR - BekchiAI combines a deterministic benchmark for evaluating tool-using LLM agents with a web platform for monitoring and controlling deployed agents. It matters because it measures behaviors such as tool adherence, grounding, planning, and security judgment that accuracy-only leaderboards miss.
- The benchmark contains 2,057 verifier-checkable tasks spanning seven categories and 13 ReAct agents.
- Gold answers are computed through canonical SQL, exact DAG schedules, or closed-form functions rather than subjective grading.
- Metrics include tool-call adherence, URL hallucination and source matching, accuracy, and per-model token cost.
- The observability platform provides token and latency telemetry plus remote termination of agent runs.