Omega-S: A Functional Resilience Index for LLM Fine-Tuning

TL;DR - Omega-S is a lightweight regularization penalty designed to reduce capability loss during LLM fine-tuning without retaining prior data or weights. On Llama-3-8B with LoRA, it improved code-capability retention after prose fine-tuning while adding under 4% per-step cost.

• HumanEval retention improved from 62.9% to 84.1% across ten seeds.
• Omega-S outperformed tuned weight decay on all ten seeds and tuned EWC on eight.
• Analysis showed its effective mechanism is primarily a node-degree variance penalty, rather than the intended composite topological objective.
• Identical runs showed substantial retention variability (0.104 standard deviation), highlighting reproducibility limits in seed-paired comparisons.