Long-Context Fine-Tuning with Limited VRAM

TL;DR - A method for fine-tuning and running long-context LLMs on a single 16 GB GPU by keeping only the active segment differentiable in VRAM and offloading older KV cache to RAM/NVMe. It matters because it stretches usable training context roughly 8× and eval context far further on modest hardware.

• Combines Hierarchical Global Attention (HGA), segment-wise backpropagation, and tiered KV storage (VRAM → RAM → NVMe); only the active segment stays differentiable, while HGA loads a bounded set of exact historical tokens per query block.
• On Qwen3-8B with 4-bit QLoRA on PG19, dense training on a 16 GB Quadro RTX 5000 handles 2,048 tokens but fails at 4,096; HGA reaches 16,384 tokens at 15.28 GB peak, and evaluation runs to 131,072 tokens on the same card.
• Quality is competitive: at the shared 2K training length, HGA- and dense-trained adapters score 2.7405 vs. 2.7383 nat (stock model 2.9541) under identical dense-attention readout.
• HGA is already marginally faster at 2K (217.75 vs. 207.02 tokens/s) with a widening throughput lead expected as context grows; dense attention was used for headline quality/retrieval comparisons, and a production serving implementation is noted as in development.