SparseDitto: Customizing GPU Kernels for Different Sparsity Patterns with LLM-Based Agentic System

TL;DR - SparseDitto is an LLM-based agentic system that generates a custom GPU sparse-matrix kernel per matrix, operator, and target GPU, reporting ~2.7-2.8x geometric-mean speedups over cuSPARSE. It matters because no single sparse format or execution strategy dominates across sparsity patterns, so per-workload kernel specialization can unlock large, otherwise-stranded performance.

• Motivating observation: for the same SpMM on the same matrix, cuSPARSE shows a 350x gap between CSR and Blocked-ELL; a survey of formats, specialized systems, and sparse compilers found no consistently dominant implementation.
• Pipeline: a lightweight additive model ranks known strategies from structural matrix features, an architecture-aware planner proposes candidate designs, and coding/verification agents implement and refine them using on-device measurements.
• Coverage and results: unified support for SpMV, SpMM, and SpGEMM; 2.68x geomean (max 146.61x) on an NVIDIA RTX PRO 6000 and 2.79x geomean (max 78.5x) on an H200 versus cuSPARSE.
• Downstream impact: generated SpMM kernels speed up full-batch GCN training by up to 3.39x.