Dynamic Compression in Recurrent Networks

TL;DR - Dynamic compression lets recurrent models selectively revisit earlier tokens and update their fixed-size state when past information becomes relevant. This trades additional computation for lower memory requirements and more effective long-context processing.

• Unlike single-pass recurrence, the method can refine compressed representations using retained raw sequence data.
• In few-shot function-reuse tasks, models selectively re-scan the specific previously learned function needed for the current task.
• Dynamic compression substantially reduces the recurrent state required for accurate function reuse.
• Its memory requirements scale more favorably as the number of stored functions increases.