Compositional Online Learning for Semantic Data Processing Systems

TL;DR - This paper introduces compositional online learning for semantic data systems, exploiting LLM-call latency to hide CPU-side model updates. In a Cortex AISQL case study, combined filter ordering and cascade routing reduce estimated per-row LLM cost by a realistic factor near 8×.

• Learner updates, including gradient steps and threshold optimization, run during the next LLM round-trip rather than adding latency.
• The framework organizes components by decision granularity and update cadence while sharing a common online-training pattern.
• The production design combines memoization, per-call filter ordering, and per-batch cascade routing.
• Theoretical savings reach an 11.4× upper bound under independence, reduced by practical effects such as self-selection, smaller sample budgets, and selectivity drift.