ProxyFormer: A Dual-Stream Proxy Architecture for Ultra-Long Context and High-Resolution Generation

TL;DR - ProxyFormer uses persistent local features alongside compressed proxy tokens to make global attention and KV caching substantially cheaper. It enables million-token retrieval and much longer training contexts on limited GPU memory while also showing preliminary applicability to high-resolution image generation.

• Each layer compresses local features into proxy states, performs global interactions in proxy space, then injects the contextualized proxies back into the persistent local stream.
• Factorized multi-level compression, dynamic layer-wise compression ratios, asymmetric embeddings, and proxy-only KV caching further reduce resource requirements.
• On a 16GB GPU, a compression ratio of 64 increases the reported trainable context from roughly 20K to 0.7M tokens.
• A 64K-window model retains 92%–95% multi-needle retrieval accuracy at 1,048,576 tokens; preliminary flow-matching experiments cover pixel and latent image generation.