Computation Over Geometry: Meaning Identity Is Computed, Not Shipped in the Embeddings

TL;DR - This paper argues that semantic identity is computed by jointly processing sentence pairs rather than encoded in their independently produced embedding geometry. The result challenges cosine-based retrieval and RAG systems that assume semantically equivalent sentences naturally occupy nearby vector-space locations.

• Frozen bi-encoders and independently encoded causal-LM states achieved only 0.55–0.70 AUC on overlap-matched PAWS-X, while joint forward-pass probes reached 0.90–0.96.
• The joint signal emerged at intermediate layers, saturated near 0.94 by 3B parameters, and collapsed when sentence partners were shuffled.
• Linear readers and late fusion could not recover identity from independent embeddings; nonlinear pair readers partially succeeded only with the full 49,000-pair training set.
• Fine-tuned bi-encoders reached 0.87–0.93 on PAWS but suffered on transfer and STS-B, suggesting a tradeoff between identity detection and general embedding quality.