StableVQ: Practical Guidelines for Stable Vector-Quantized Tokenizer Training

TL;DR - StableVQ is a parameter-free training approach for more stable vector-quantized visual tokenizers. It separates encoder-decoder and codebook responsibilities to improve codebook utilization and reconstruction quality across varied ImageNet settings.

• Dynamic STE stabilizes encoder optimization under discrete regularization, including when codebook utilization is low.
• Region VQ Loss enables the codebook to track the encoder output distribution without depending on encoder oscillations.
• Independent learning-rate schedules reflect the different optimization dynamics of the encoder-decoder and codebook.
• Built on shared-projection codebooks, StableVQ adds no learnable parameters.