Representation Measurements Under Function-Preserving Reparameterizations

TL;DR - This paper shows that column-permutation parallel analysis can produce representation measurements that change under function-preserving reparameterizations of language models. Consequently, its component counts may reflect arbitrary hidden-coordinate choices rather than intrinsic model properties.

• Across five models, three retrieval domains, and 75 transformations, median disagreement was 0.79 for component counts and 0.26 for fixed-threshold decisions.
• A centering-only control changed 1,141 of 1,200 component counts despite leaving the observed covariance spectrum unchanged.
• Independent parallel-analysis seeds preserved all corresponding control decisions, isolating the instability to the data-internal reference procedure.
• Orthogonally invariant comparator scores remained numerically stable while achieving similar held-out discrimination.