VAIOM: Continuous-Input, Discrete-Output Decoder-Only Financial Sequence Modeling

TL;DR — VAIOM is a decoder-only Transformer that adapts next-token autoregressive modeling to financial time series by feeding continuous multivariate inputs while predicting a discrete (bucketed) next-return distribution, and it beats a LightGBM baseline on out-of-sample FX return forecasting. It matters as a methodological template for applying autoregressive sequence models to continuous, noisy, non-symbolic data.

• Decouples representation from likelihood: continuous financial-event vectors at the input, categorical distribution over the next volatility-normalized return bucket at the output, trained via cross-entropy for likelihood evaluation.
• The ~0.9M-parameter "Hybrid Continuous Input" model adds categorical asset metadata, a Mixture-of-Market-States return head, and Gap/volatility-regime/Ordinal auxiliary objectives with full-sequence supervision.
• On hourly FX bars (trained pre-2024, validated on 2024H2, tested on two 2025 periods without refitting), all three seeds beat a fixed single-bar LightGBM baseline; canonical checkpoint gains 0.029 and 0.043 bits/event.
• Ablations indicate continuous input > discrete tokens, full-sequence > last-position supervision, and the mixture head plus auxiliary shaping improve return likelihood; a capacity study found the smallest complete architecture performed best on validation.