Computation Over Geometry: Meaning Identity Is Computed, Not Shipped in the Embeddings
Ranking
Overall
75
Content
95
Popularity
27
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper argues that semantic identity is computed by jointly processing sentence pairs rather than encoded in their independently produced embedding geometry. The result challenges cosine-based retrieval and RAG systems that assume semantically equivalent sentences naturally occupy nearby vector-space locations.
- Frozen bi-encoders and independently encoded causal-LM states achieved only 0.55–0.70 AUC on overlap-matched PAWS-X, while joint forward-pass probes reached 0.90–0.96.
- The joint signal emerged at intermediate layers, saturated near 0.94 by 3B parameters, and collapsed when sentence partners were shuffled.
- Linear readers and late fusion could not recover identity from independent embeddings; nonlinear pair readers partially succeeded only with the full 49,000-pair training set.
- Fine-tuned bi-encoders reached 0.87–0.93 on PAWS but suffered on transfer and STS-B, suggesting a tradeoff between identity detection and general embedding quality.
Sources (1)
Computation Over Geometry: Meaning Identity Is Computed, Not Shipped in the Embeddings
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - This paper argues that semantic identity is computed by jointly processing sentence pairs rather than encoded in their independently produced embedding geometry. The result challenges cosine-based retrieval and RAG systems that assume semantically equivalent sentences naturally occupy nearby vector-space locations.
- Frozen bi-encoders and independently encoded causal-LM states achieved only 0.55–0.70 AUC on overlap-matched PAWS-X, while joint forward-pass probes reached 0.90–0.96.
- The joint signal emerged at intermediate layers, saturated near 0.94 by 3B parameters, and collapsed when sentence partners were shuffled.
- Linear readers and late fusion could not recover identity from independent embeddings; nonlinear pair readers partially succeeded only with the full 49,000-pair training set.
- Fine-tuned bi-encoders reached 0.87–0.93 on PAWS but suffered on transfer and STS-B, suggesting a tradeoff between identity detection and general embedding quality.