Comparative Approaches to Agent Retrieval over Large Skill Libraries
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - An arXiv study compares hybrid lexical+dense retrieval against a typed knowledge graph for selecting skills from a 690-skill agent library, finding the graph adds no retrieval reach over a strong ranker. It matters because it challenges the assumption that structured workflow graphs improve agent skill retrieval, and exposes a benchmarking pitfall.
- On 117 realistic non-echoing queries, the hybrid ranker hit the correct skill in the top 5 in 73.5% ± 8.0 of cases; the typed graph, used as designed (swapping graph neighbours for ranked results at matched token budget), was 11.2 points worse (p = 0.0007).
- Root cause is a "pre-filter topology bound": graph candidate edges come from the same embedding neighbourhood the ranker already searches, so 98.6% of typed edges connect skills already surfaced together, and 73% of ranker misses are unreachable via the graph.
- The LLM-generated edge layer contributed nothing beyond neighbours obtainable free from a local embedding pass — graph structure enriched relation semantics (prerequisites, data flow, ordering) but not retrieval reach.
- Evaluating on author-written queries inflated hit@5 by up to 44 points, which would have masked the negative result entirely; the paper also outlines conditions under which structural interdependence would help.
Sources (1)
Comparative Approaches to Agent Retrieval over Large Skill Libraries
TL;DR - An arXiv study compares hybrid lexical+dense retrieval against a typed knowledge graph for selecting skills from a 690-skill agent library, finding the graph adds no retrieval reach over a strong ranker. It matters because it challenges the assumption that structured workflow graphs improve agent skill retrieval, and exposes a benchmarking pitfall.
- On 117 realistic non-echoing queries, the hybrid ranker hit the correct skill in the top 5 in 73.5% ± 8.0 of cases; the typed graph, used as designed (swapping graph neighbours for ranked results at matched token budget), was 11.2 points worse (p = 0.0007).
- Root cause is a "pre-filter topology bound": graph candidate edges come from the same embedding neighbourhood the ranker already searches, so 98.6% of typed edges connect skills already surfaced together, and 73% of ranker misses are unreachable via the graph.
- The LLM-generated edge layer contributed nothing beyond neighbours obtainable free from a local embedding pass — graph structure enriched relation semantics (prerequisites, data flow, ordering) but not retrieval reach.
- Evaluating on author-written queries inflated hit@5 by up to 44 points, which would have masked the negative result entirely; the paper also outlines conditions under which structural interdependence would help.