BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click

TL;DR - BekchiAI combines a deterministic benchmark for evaluating tool-using LLM agents with a web platform for monitoring and controlling deployed agents. It matters because it measures behaviors such as tool adherence, grounding, planning, and security judgment that accuracy-only leaderboards miss.

• The benchmark contains 2,057 verifier-checkable tasks spanning seven categories and 13 ReAct agents.
• Gold answers are computed through canonical SQL, exact DAG schedules, or closed-form functions rather than subjective grading.
• Metrics include tool-call adherence, URL hallucination and source matching, accuracy, and per-model token cost.
• The observability platform provides token and latency telemetry plus remote termination of agent runs.