ProxyFormer: A Dual-Stream Proxy Architecture for Ultra-Long Context and High-Resolution Generation
Ranking
Overall
78
Content
95
Popularity
37
Observed public metrics from 1 member.
Merged summary
TL;DR - ProxyFormer uses persistent local features alongside compressed proxy tokens to make global attention and KV caching substantially cheaper. It enables million-token retrieval and much longer training contexts on limited GPU memory while also showing preliminary applicability to high-resolution image generation.
- Each layer compresses local features into proxy states, performs global interactions in proxy space, then injects the contextualized proxies back into the persistent local stream.
- Factorized multi-level compression, dynamic layer-wise compression ratios, asymmetric embeddings, and proxy-only KV caching further reduce resource requirements.
- On a 16GB GPU, a compression ratio of 64 increases the reported trainable context from roughly 20K to 0.7M tokens.
- A 64K-window model retains 92%–95% multi-needle retrieval accuracy at 1,048,576 tokens; preliminary flow-matching experiments cover pixel and latent image generation.
Sources (1)
ProxyFormer: A Dual-Stream Proxy Architecture for Ultra-Long Context and High-Resolution Generation
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - ProxyFormer uses persistent local features alongside compressed proxy tokens to make global attention and KV caching substantially cheaper. It enables million-token retrieval and much longer training contexts on limited GPU memory while also showing preliminary applicability to high-resolution image generation.
- Each layer compresses local features into proxy states, performs global interactions in proxy space, then injects the contextualized proxies back into the persistent local stream.
- Factorized multi-level compression, dynamic layer-wise compression ratios, asymmetric embeddings, and proxy-only KV caching further reduce resource requirements.
- On a 16GB GPU, a compression ratio of 64 increases the reported trainable context from roughly 20K to 0.7M tokens.
- A 64K-window model retains 92%–95% multi-needle retrieval accuracy at 1,048,576 tokens; preliminary flow-matching experiments cover pixel and latent image generation.