Do Tabular Foundation Models Agree with Themselves?

TL;DR - An arXiv study tests whether Tabular Foundation Models (TFMs) — transformers that approximate a Bayesian posterior predictive and are extended to multivariate prediction by autoregressive sampling — produce predictions consistent with any valid joint distribution. Every evaluated TFM fails, undermining the probabilistic interpretation of their multivariate outputs.

• TFMs are univariate predictors turned multivariate by sampling a target and appending it to the features; the faithfulness of that induced joint had not been examined.
• Direct evaluation against the true posterior is impossible on real data (ground-truth distribution unknown), so the authors instead ask whether predictions could arise from any joint distribution.
• Two necessary conditions are proposed: marginalization consistency (marginalized conditionals must match directly predicted marginals) and factorization consistency (different factorization orders must yield the same joint).
• All evaluated TFMs violate both requirements, for classification and regression, across every dataset tested.